<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.03211] How Does Quantization Affect Multilingual LLMs?</title><meta property="og:description" content="Quantization techniques are widely used to improve inference speed and deployment of large language models. While a wide body of work examines the impact of quantized LLMs on English tasks, none have examined the effec…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="How Does Quantization Affect Multilingual LLMs?">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="How Does Quantization Affect Multilingual LLMs?">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.03211">

<!--Generated on Mon Aug  5 16:19:29 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">How Does Quantization Affect Multilingual LLMs?</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
<span id="id1.1.id1" class="ltx_text ltx_font_bold">Kelly Marchisio<sup id="id1.1.id1.1" class="ltx_sup">1</sup></span>,
<span id="id2.2.id2" class="ltx_text ltx_font_bold">Saurabh Dash<sup id="id2.2.id2.1" class="ltx_sup">2</sup></span>,
<span id="id3.3.id3" class="ltx_text ltx_font_bold">Hongyu Chen<sup id="id3.3.id3.1" class="ltx_sup">1</sup></span>,
<span id="id4.4.id4" class="ltx_text ltx_font_bold">Dennis Aumiller<sup id="id4.4.id4.1" class="ltx_sup">1</sup></span>,

<br class="ltx_break"><span id="id5.5.id5" class="ltx_text ltx_font_bold">Ahmet Üstün<sup id="id5.5.id5.1" class="ltx_sup">2</sup></span>,
<span id="id6.6.id6" class="ltx_text ltx_font_bold">Sara Hooker<sup id="id6.6.id6.1" class="ltx_sup">2</sup></span>,
<span id="id7.7.id7" class="ltx_text ltx_font_bold">Sebastian Ruder<sup id="id7.7.id7.1" class="ltx_sup">1</sup></span>

<br class="ltx_break">
<br class="ltx_break"><sup id="id8.8.id8" class="ltx_sup">1</sup>Cohere
<sup id="id9.9.id9" class="ltx_sup">2</sup>Cohere For AI

<br class="ltx_break">
<span id="id10.10.id10" class="ltx_text ltx_font_bold">Correspondence:</span> kelly@cohere.com
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id11.id1" class="ltx_p">Quantization techniques are widely used to improve inference speed and deployment of large language models. While a wide body of work examines the impact of quantized LLMs on English tasks, none have examined the effect of quantization across languages. We conduct a thorough analysis of quantized multilingual LLMs, focusing on their performance across languages and at varying scales. We use automatic benchmarks, LLM-as-a-Judge methods, and human evaluation, finding that (1) harmful effects of quantization are apparent in human evaluation, and automatic metrics severely underestimate the detriment: a 1.7% average drop in Japanese across automatic tasks corresponds to a 16.0% drop reported by human evaluators on realistic prompts; (2) languages are disparately affected by quantization, with non-Latin script languages impacted worst; and (3) challenging tasks such as mathematical reasoning degrade fastest. As the ability to serve low-compute models is critical for wide global adoption of NLP technologies, our results urge consideration of multilingual performance as a key evaluation criterion for efficient models.</p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.1" class="ltx_block ltx_align_bottom">
<p id="p1.1.1" class="ltx_p"><span id="p1.1.1.1" class="ltx_text ltx_font_bold">How Does Quantization Affect Multilingual LLMs?</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.1.2" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.1.2.1" class="ltx_text ltx_inline-block" style="width:0.0pt;">
<span id="p1.1.2.1.1" class="ltx_tabular ltx_align_top">
<span class="ltx_tbody">
<span id="p1.1.2.1.1.1.1" class="ltx_tr">
<span id="p1.1.2.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="p1.1.2.1.1.1.1.1.1" class="ltx_text ltx_font_bold">
Kelly Marchisio<sup id="p1.1.2.1.1.1.1.1.1.1" class="ltx_sup">1</sup>,
Saurabh Dash<sup id="p1.1.2.1.1.1.1.1.1.2" class="ltx_sup">2</sup>,
Hongyu Chen<sup id="p1.1.2.1.1.1.1.1.1.3" class="ltx_sup">1</sup>,
Dennis Aumiller<sup id="p1.1.2.1.1.1.1.1.1.4" class="ltx_sup">1</sup>,</span></span></span>
<span id="p1.1.2.1.1.2.2" class="ltx_tr">
<span id="p1.1.2.1.1.2.2.1" class="ltx_td ltx_align_center"><span id="p1.1.2.1.1.2.2.1.1" class="ltx_text ltx_font_bold">Ahmet Üstün<sup id="p1.1.2.1.1.2.2.1.1.1" class="ltx_sup">2</sup></span>,
<span id="p1.1.2.1.1.2.2.1.2" class="ltx_text ltx_font_bold">Sara Hooker<sup id="p1.1.2.1.1.2.2.1.2.1" class="ltx_sup">2</sup></span>,
<span id="p1.1.2.1.1.2.2.1.3" class="ltx_text ltx_font_bold">Sebastian Ruder<sup id="p1.1.2.1.1.2.2.1.3.1" class="ltx_sup">1</sup></span></span></span>
<span id="p1.1.2.1.1.3.3" class="ltx_tr">
<span id="p1.1.2.1.1.3.3.1" class="ltx_td ltx_align_center"><sup id="p1.1.2.1.1.3.3.1.1" class="ltx_sup">1</sup>Cohere
<sup id="p1.1.2.1.1.3.3.1.2" class="ltx_sup">2</sup>Cohere For AI</span></span>
<span id="p1.1.2.1.1.4.4" class="ltx_tr">
<span id="p1.1.2.1.1.4.4.1" class="ltx_td ltx_align_center">
<span id="p1.1.2.1.1.4.4.1.1" class="ltx_text ltx_font_bold">Correspondence:</span> kelly@cohere.com</span></span>
</span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Multilingual large language models (LLMs) have the power to bring modern language technology to the world, but only if they are cheap and reliable. Known as the <span id="S1.p1.1.1" class="ltx_text ltx_font_italic">low-resource double bind</span>, underserved languages and severe compute constraints often geographically co-occur <cite class="ltx_cite ltx_citemacro_citep">(Ahia et al., <a href="#bib.bib2" title="" class="ltx_ref">2021</a>)</cite>, meaning that for wide adoption, multilingual LLMs must be highly-performant <span id="S1.p1.1.2" class="ltx_text ltx_font_italic">and</span> lightweight.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">With the shift towards large models, quantization is a widely adopted technique to reduce cost, improve inference speed, and enable wider deployment of LLMs. Work on quantization, however, is by-and-large evaluated in English only <cite class="ltx_cite ltx_citemacro_citep">(e.g. Xiao et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>; Ahmadian et al., <a href="#bib.bib4" title="" class="ltx_ref">2024</a>; Frantar et al., <a href="#bib.bib19" title="" class="ltx_ref">2022</a>)</cite>. No works to our knowledge have characterized the impact of quantization on the multilingual generation capabilities expected from modern LLMs. Ubiquitous use of compression techniques in the real world drives urgency to the question <span id="S1.p2.1.1" class="ltx_text ltx_font_italic">how are multilingual models impacted?</span></p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Our question is timely, given recent work showing that compression techniques such as quantization and sparsity amplify disparate treatment of long-tail features, which may have implications for under-represented languages in multilingual LLMs <cite class="ltx_cite ltx_citemacro_citep">(Hooker et al., <a href="#bib.bib24" title="" class="ltx_ref">2019</a>, <a href="#bib.bib25" title="" class="ltx_ref">2020</a>; Ahia et al., <a href="#bib.bib2" title="" class="ltx_ref">2021</a>; Ogueji et al., <a href="#bib.bib44" title="" class="ltx_ref">2022</a>)</cite>.
Indeed, many model designs choices implicitly overfit to a handful of resource rich languages: from tokenizer choice, to weighting of training data, and to widely-used quantization techniques.
Focusing on a small subset of high-resource languages in design degrades model performance for overlooked languages <cite class="ltx_cite ltx_citemacro_citep">(Schwartz et al., <a href="#bib.bib51" title="" class="ltx_ref">2022</a>; Kotek et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>; Khandelwal et al., <a href="#bib.bib27" title="" class="ltx_ref">2023</a>; Vashishtha et al., <a href="#bib.bib54" title="" class="ltx_ref">2023</a>; Khondaker et al., <a href="#bib.bib29" title="" class="ltx_ref">2023</a>; Pozzobon et al., <a href="#bib.bib49" title="" class="ltx_ref">2024</a>)</cite>, introduces security vulnerabilities <cite class="ltx_cite ltx_citemacro_citep">(Yong et al., <a href="#bib.bib57" title="" class="ltx_ref">2023</a>; Nasr et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>; Li et al., <a href="#bib.bib32" title="" class="ltx_ref">2023a</a>; Lukas et al., <a href="#bib.bib38" title="" class="ltx_ref">2023</a>; Deng et al., <a href="#bib.bib14" title="" class="ltx_ref">2023</a>)</cite>, and unfairly passes high costs to non-English users faced with high latency <cite class="ltx_cite ltx_citemacro_citep">(Held et al., <a href="#bib.bib22" title="" class="ltx_ref">2023</a>; Durmus et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>; Nicholas and Bhatia, <a href="#bib.bib43" title="" class="ltx_ref">2023</a>; Ojo et al., <a href="#bib.bib45" title="" class="ltx_ref">2023</a>; Ahia et al., <a href="#bib.bib3" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">We analyze four state-of-the-art multilingual LLMs across 3 different sizes ranging from 8 to 103 billion parameters and covering up to 23 languages, under various quantization techniques. Critically, it is vital that we move beyond automatic evaluation and gather <span id="S1.p4.1.1" class="ltx_text ltx_font_italic">real human feedback on performance cost</span>. We thus perform multilingual human evaluation on challenging real-world prompts in addition to LLM-as-a-Judge and evaluation on standard automatic benchmarks such as multilingual MMLU <cite class="ltx_cite ltx_citemacro_cite">Hendrycks et al. (<a href="#bib.bib23" title="" class="ltx_ref">2020</a>)</cite>, MGSM <cite class="ltx_cite ltx_citemacro_cite">Shi et al. (<a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite>, and FLORES-200 <cite class="ltx_cite ltx_citemacro_citep">(Costa-jussà et al., <a href="#bib.bib11" title="" class="ltx_ref">2022a</a>)</cite>.
Across experimental set-ups we find that:</p>
</div>
<div id="S1.p5" class="ltx_para">
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.4" class="ltx_p"><span id="S1.I1.i1.p1.4.1" class="ltx_text ltx_font_bold">Automatic metrics severely underestimate damage from quantization.</span> Automatic evaluations estimate performance deterioration relative to FP16 across tasks at <math id="S1.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="-0.3\%" display="inline"><semantics id="S1.I1.i1.p1.1.m1.1a"><mrow id="S1.I1.i1.p1.1.m1.1.1" xref="S1.I1.i1.p1.1.m1.1.1.cmml"><mo id="S1.I1.i1.p1.1.m1.1.1a" xref="S1.I1.i1.p1.1.m1.1.1.cmml">−</mo><mrow id="S1.I1.i1.p1.1.m1.1.1.2" xref="S1.I1.i1.p1.1.m1.1.1.2.cmml"><mn id="S1.I1.i1.p1.1.m1.1.1.2.2" xref="S1.I1.i1.p1.1.m1.1.1.2.2.cmml">0.3</mn><mo id="S1.I1.i1.p1.1.m1.1.1.2.1" xref="S1.I1.i1.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i1.p1.1.m1.1b"><apply id="S1.I1.i1.p1.1.m1.1.1.cmml" xref="S1.I1.i1.p1.1.m1.1.1"><minus id="S1.I1.i1.p1.1.m1.1.1.1.cmml" xref="S1.I1.i1.p1.1.m1.1.1"></minus><apply id="S1.I1.i1.p1.1.m1.1.1.2.cmml" xref="S1.I1.i1.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S1.I1.i1.p1.1.m1.1.1.2.1.cmml" xref="S1.I1.i1.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i1.p1.1.m1.1.1.2.2.cmml" xref="S1.I1.i1.p1.1.m1.1.1.2.2">0.3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i1.p1.1.m1.1c">-0.3\%</annotation></semantics></math> (French) and <math id="S1.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="-1.7\%" display="inline"><semantics id="S1.I1.i1.p1.2.m2.1a"><mrow id="S1.I1.i1.p1.2.m2.1.1" xref="S1.I1.i1.p1.2.m2.1.1.cmml"><mo id="S1.I1.i1.p1.2.m2.1.1a" xref="S1.I1.i1.p1.2.m2.1.1.cmml">−</mo><mrow id="S1.I1.i1.p1.2.m2.1.1.2" xref="S1.I1.i1.p1.2.m2.1.1.2.cmml"><mn id="S1.I1.i1.p1.2.m2.1.1.2.2" xref="S1.I1.i1.p1.2.m2.1.1.2.2.cmml">1.7</mn><mo id="S1.I1.i1.p1.2.m2.1.1.2.1" xref="S1.I1.i1.p1.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i1.p1.2.m2.1b"><apply id="S1.I1.i1.p1.2.m2.1.1.cmml" xref="S1.I1.i1.p1.2.m2.1.1"><minus id="S1.I1.i1.p1.2.m2.1.1.1.cmml" xref="S1.I1.i1.p1.2.m2.1.1"></minus><apply id="S1.I1.i1.p1.2.m2.1.1.2.cmml" xref="S1.I1.i1.p1.2.m2.1.1.2"><csymbol cd="latexml" id="S1.I1.i1.p1.2.m2.1.1.2.1.cmml" xref="S1.I1.i1.p1.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i1.p1.2.m2.1.1.2.2.cmml" xref="S1.I1.i1.p1.2.m2.1.1.2.2">1.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i1.p1.2.m2.1c">-1.7\%</annotation></semantics></math> (Japanese) vs <math id="S1.I1.i1.p1.3.m3.1" class="ltx_Math" alttext="-16.6\%" display="inline"><semantics id="S1.I1.i1.p1.3.m3.1a"><mrow id="S1.I1.i1.p1.3.m3.1.1" xref="S1.I1.i1.p1.3.m3.1.1.cmml"><mo id="S1.I1.i1.p1.3.m3.1.1a" xref="S1.I1.i1.p1.3.m3.1.1.cmml">−</mo><mrow id="S1.I1.i1.p1.3.m3.1.1.2" xref="S1.I1.i1.p1.3.m3.1.1.2.cmml"><mn id="S1.I1.i1.p1.3.m3.1.1.2.2" xref="S1.I1.i1.p1.3.m3.1.1.2.2.cmml">16.6</mn><mo id="S1.I1.i1.p1.3.m3.1.1.2.1" xref="S1.I1.i1.p1.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i1.p1.3.m3.1b"><apply id="S1.I1.i1.p1.3.m3.1.1.cmml" xref="S1.I1.i1.p1.3.m3.1.1"><minus id="S1.I1.i1.p1.3.m3.1.1.1.cmml" xref="S1.I1.i1.p1.3.m3.1.1"></minus><apply id="S1.I1.i1.p1.3.m3.1.1.2.cmml" xref="S1.I1.i1.p1.3.m3.1.1.2"><csymbol cd="latexml" id="S1.I1.i1.p1.3.m3.1.1.2.1.cmml" xref="S1.I1.i1.p1.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i1.p1.3.m3.1.1.2.2.cmml" xref="S1.I1.i1.p1.3.m3.1.1.2.2">16.6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i1.p1.3.m3.1c">-16.6\%</annotation></semantics></math> and <math id="S1.I1.i1.p1.4.m4.1" class="ltx_Math" alttext="-16.0\%" display="inline"><semantics id="S1.I1.i1.p1.4.m4.1a"><mrow id="S1.I1.i1.p1.4.m4.1.1" xref="S1.I1.i1.p1.4.m4.1.1.cmml"><mo id="S1.I1.i1.p1.4.m4.1.1a" xref="S1.I1.i1.p1.4.m4.1.1.cmml">−</mo><mrow id="S1.I1.i1.p1.4.m4.1.1.2" xref="S1.I1.i1.p1.4.m4.1.1.2.cmml"><mn id="S1.I1.i1.p1.4.m4.1.1.2.2" xref="S1.I1.i1.p1.4.m4.1.1.2.2.cmml">16.0</mn><mo id="S1.I1.i1.p1.4.m4.1.1.2.1" xref="S1.I1.i1.p1.4.m4.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i1.p1.4.m4.1b"><apply id="S1.I1.i1.p1.4.m4.1.1.cmml" xref="S1.I1.i1.p1.4.m4.1.1"><minus id="S1.I1.i1.p1.4.m4.1.1.1.cmml" xref="S1.I1.i1.p1.4.m4.1.1"></minus><apply id="S1.I1.i1.p1.4.m4.1.1.2.cmml" xref="S1.I1.i1.p1.4.m4.1.1.2"><csymbol cd="latexml" id="S1.I1.i1.p1.4.m4.1.1.2.1.cmml" xref="S1.I1.i1.p1.4.m4.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i1.p1.4.m4.1.1.2.2.cmml" xref="S1.I1.i1.p1.4.m4.1.1.2.2">16.0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i1.p1.4.m4.1c">-16.0\%</annotation></semantics></math> reported by human evaluators.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.4" class="ltx_p"><span id="S1.I1.i2.p1.4.1" class="ltx_text ltx_font_bold">Quantization affects languages differently.</span> Non-Latin script languages are more greatly harmed
on average. Across tasks, Latin-script languages scored <math id="S1.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="-0.7\%" display="inline"><semantics id="S1.I1.i2.p1.1.m1.1a"><mrow id="S1.I1.i2.p1.1.m1.1.1" xref="S1.I1.i2.p1.1.m1.1.1.cmml"><mo id="S1.I1.i2.p1.1.m1.1.1a" xref="S1.I1.i2.p1.1.m1.1.1.cmml">−</mo><mrow id="S1.I1.i2.p1.1.m1.1.1.2" xref="S1.I1.i2.p1.1.m1.1.1.2.cmml"><mn id="S1.I1.i2.p1.1.m1.1.1.2.2" xref="S1.I1.i2.p1.1.m1.1.1.2.2.cmml">0.7</mn><mo id="S1.I1.i2.p1.1.m1.1.1.2.1" xref="S1.I1.i2.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i2.p1.1.m1.1b"><apply id="S1.I1.i2.p1.1.m1.1.1.cmml" xref="S1.I1.i2.p1.1.m1.1.1"><minus id="S1.I1.i2.p1.1.m1.1.1.1.cmml" xref="S1.I1.i2.p1.1.m1.1.1"></minus><apply id="S1.I1.i2.p1.1.m1.1.1.2.cmml" xref="S1.I1.i2.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S1.I1.i2.p1.1.m1.1.1.2.1.cmml" xref="S1.I1.i2.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i2.p1.1.m1.1.1.2.2.cmml" xref="S1.I1.i2.p1.1.m1.1.1.2.2">0.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i2.p1.1.m1.1c">-0.7\%</annotation></semantics></math> relative to FP16 for a 103B parameter model while non-Latin scripts scored <math id="S1.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="-1.9\%" display="inline"><semantics id="S1.I1.i2.p1.2.m2.1a"><mrow id="S1.I1.i2.p1.2.m2.1.1" xref="S1.I1.i2.p1.2.m2.1.1.cmml"><mo id="S1.I1.i2.p1.2.m2.1.1a" xref="S1.I1.i2.p1.2.m2.1.1.cmml">−</mo><mrow id="S1.I1.i2.p1.2.m2.1.1.2" xref="S1.I1.i2.p1.2.m2.1.1.2.cmml"><mn id="S1.I1.i2.p1.2.m2.1.1.2.2" xref="S1.I1.i2.p1.2.m2.1.1.2.2.cmml">1.9</mn><mo id="S1.I1.i2.p1.2.m2.1.1.2.1" xref="S1.I1.i2.p1.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i2.p1.2.m2.1b"><apply id="S1.I1.i2.p1.2.m2.1.1.cmml" xref="S1.I1.i2.p1.2.m2.1.1"><minus id="S1.I1.i2.p1.2.m2.1.1.1.cmml" xref="S1.I1.i2.p1.2.m2.1.1"></minus><apply id="S1.I1.i2.p1.2.m2.1.1.2.cmml" xref="S1.I1.i2.p1.2.m2.1.1.2"><csymbol cd="latexml" id="S1.I1.i2.p1.2.m2.1.1.2.1.cmml" xref="S1.I1.i2.p1.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i2.p1.2.m2.1.1.2.2.cmml" xref="S1.I1.i2.p1.2.m2.1.1.2.2">1.9</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i2.p1.2.m2.1c">-1.9\%</annotation></semantics></math>. For a smaller 8-billion parameter model, scores were <math id="S1.I1.i2.p1.3.m3.1" class="ltx_Math" alttext="-3.0\%" display="inline"><semantics id="S1.I1.i2.p1.3.m3.1a"><mrow id="S1.I1.i2.p1.3.m3.1.1" xref="S1.I1.i2.p1.3.m3.1.1.cmml"><mo id="S1.I1.i2.p1.3.m3.1.1a" xref="S1.I1.i2.p1.3.m3.1.1.cmml">−</mo><mrow id="S1.I1.i2.p1.3.m3.1.1.2" xref="S1.I1.i2.p1.3.m3.1.1.2.cmml"><mn id="S1.I1.i2.p1.3.m3.1.1.2.2" xref="S1.I1.i2.p1.3.m3.1.1.2.2.cmml">3.0</mn><mo id="S1.I1.i2.p1.3.m3.1.1.2.1" xref="S1.I1.i2.p1.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i2.p1.3.m3.1b"><apply id="S1.I1.i2.p1.3.m3.1.1.cmml" xref="S1.I1.i2.p1.3.m3.1.1"><minus id="S1.I1.i2.p1.3.m3.1.1.1.cmml" xref="S1.I1.i2.p1.3.m3.1.1"></minus><apply id="S1.I1.i2.p1.3.m3.1.1.2.cmml" xref="S1.I1.i2.p1.3.m3.1.1.2"><csymbol cd="latexml" id="S1.I1.i2.p1.3.m3.1.1.2.1.cmml" xref="S1.I1.i2.p1.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i2.p1.3.m3.1.1.2.2.cmml" xref="S1.I1.i2.p1.3.m3.1.1.2.2">3.0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i2.p1.3.m3.1c">-3.0\%</annotation></semantics></math> vs. <math id="S1.I1.i2.p1.4.m4.1" class="ltx_Math" alttext="-3.7\%" display="inline"><semantics id="S1.I1.i2.p1.4.m4.1a"><mrow id="S1.I1.i2.p1.4.m4.1.1" xref="S1.I1.i2.p1.4.m4.1.1.cmml"><mo id="S1.I1.i2.p1.4.m4.1.1a" xref="S1.I1.i2.p1.4.m4.1.1.cmml">−</mo><mrow id="S1.I1.i2.p1.4.m4.1.1.2" xref="S1.I1.i2.p1.4.m4.1.1.2.cmml"><mn id="S1.I1.i2.p1.4.m4.1.1.2.2" xref="S1.I1.i2.p1.4.m4.1.1.2.2.cmml">3.7</mn><mo id="S1.I1.i2.p1.4.m4.1.1.2.1" xref="S1.I1.i2.p1.4.m4.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i2.p1.4.m4.1b"><apply id="S1.I1.i2.p1.4.m4.1.1.cmml" xref="S1.I1.i2.p1.4.m4.1.1"><minus id="S1.I1.i2.p1.4.m4.1.1.1.cmml" xref="S1.I1.i2.p1.4.m4.1.1"></minus><apply id="S1.I1.i2.p1.4.m4.1.1.2.cmml" xref="S1.I1.i2.p1.4.m4.1.1.2"><csymbol cd="latexml" id="S1.I1.i2.p1.4.m4.1.1.2.1.cmml" xref="S1.I1.i2.p1.4.m4.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i2.p1.4.m4.1.1.2.2.cmml" xref="S1.I1.i2.p1.4.m4.1.1.2.2">3.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i2.p1.4.m4.1c">-3.7\%</annotation></semantics></math>.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.3" class="ltx_p"><span id="S1.I1.i3.p1.3.1" class="ltx_text ltx_font_bold">Challenging tasks degrade fastest.</span> Mathematical reasoning (<math id="S1.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="-13.1\%" display="inline"><semantics id="S1.I1.i3.p1.1.m1.1a"><mrow id="S1.I1.i3.p1.1.m1.1.1" xref="S1.I1.i3.p1.1.m1.1.1.cmml"><mo id="S1.I1.i3.p1.1.m1.1.1a" xref="S1.I1.i3.p1.1.m1.1.1.cmml">−</mo><mrow id="S1.I1.i3.p1.1.m1.1.1.2" xref="S1.I1.i3.p1.1.m1.1.1.2.cmml"><mn id="S1.I1.i3.p1.1.m1.1.1.2.2" xref="S1.I1.i3.p1.1.m1.1.1.2.2.cmml">13.1</mn><mo id="S1.I1.i3.p1.1.m1.1.1.2.1" xref="S1.I1.i3.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i3.p1.1.m1.1b"><apply id="S1.I1.i3.p1.1.m1.1.1.cmml" xref="S1.I1.i3.p1.1.m1.1.1"><minus id="S1.I1.i3.p1.1.m1.1.1.1.cmml" xref="S1.I1.i3.p1.1.m1.1.1"></minus><apply id="S1.I1.i3.p1.1.m1.1.1.2.cmml" xref="S1.I1.i3.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S1.I1.i3.p1.1.m1.1.1.2.1.cmml" xref="S1.I1.i3.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i3.p1.1.m1.1.1.2.2.cmml" xref="S1.I1.i3.p1.1.m1.1.1.2.2">13.1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i3.p1.1.m1.1c">-13.1\%</annotation></semantics></math>), performance on real-world challenging prompts judged by humans (<math id="S1.I1.i3.p1.2.m2.1" class="ltx_Math" alttext="-10.5\%" display="inline"><semantics id="S1.I1.i3.p1.2.m2.1a"><mrow id="S1.I1.i3.p1.2.m2.1.1" xref="S1.I1.i3.p1.2.m2.1.1.cmml"><mo id="S1.I1.i3.p1.2.m2.1.1a" xref="S1.I1.i3.p1.2.m2.1.1.cmml">−</mo><mrow id="S1.I1.i3.p1.2.m2.1.1.2" xref="S1.I1.i3.p1.2.m2.1.1.2.cmml"><mn id="S1.I1.i3.p1.2.m2.1.1.2.2" xref="S1.I1.i3.p1.2.m2.1.1.2.2.cmml">10.5</mn><mo id="S1.I1.i3.p1.2.m2.1.1.2.1" xref="S1.I1.i3.p1.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i3.p1.2.m2.1b"><apply id="S1.I1.i3.p1.2.m2.1.1.cmml" xref="S1.I1.i3.p1.2.m2.1.1"><minus id="S1.I1.i3.p1.2.m2.1.1.1.cmml" xref="S1.I1.i3.p1.2.m2.1.1"></minus><apply id="S1.I1.i3.p1.2.m2.1.1.2.cmml" xref="S1.I1.i3.p1.2.m2.1.1.2"><csymbol cd="latexml" id="S1.I1.i3.p1.2.m2.1.1.2.1.cmml" xref="S1.I1.i3.p1.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i3.p1.2.m2.1.1.2.2.cmml" xref="S1.I1.i3.p1.2.m2.1.1.2.2">10.5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i3.p1.2.m2.1c">-10.5\%</annotation></semantics></math>), and LLM-as-a-Judge (<math id="S1.I1.i3.p1.3.m3.1" class="ltx_Math" alttext="-25.9\%" display="inline"><semantics id="S1.I1.i3.p1.3.m3.1a"><mrow id="S1.I1.i3.p1.3.m3.1.1" xref="S1.I1.i3.p1.3.m3.1.1.cmml"><mo id="S1.I1.i3.p1.3.m3.1.1a" xref="S1.I1.i3.p1.3.m3.1.1.cmml">−</mo><mrow id="S1.I1.i3.p1.3.m3.1.1.2" xref="S1.I1.i3.p1.3.m3.1.1.2.cmml"><mn id="S1.I1.i3.p1.3.m3.1.1.2.2" xref="S1.I1.i3.p1.3.m3.1.1.2.2.cmml">25.9</mn><mo id="S1.I1.i3.p1.3.m3.1.1.2.1" xref="S1.I1.i3.p1.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.I1.i3.p1.3.m3.1b"><apply id="S1.I1.i3.p1.3.m3.1.1.cmml" xref="S1.I1.i3.p1.3.m3.1.1"><minus id="S1.I1.i3.p1.3.m3.1.1.1.cmml" xref="S1.I1.i3.p1.3.m3.1.1"></minus><apply id="S1.I1.i3.p1.3.m3.1.1.2.cmml" xref="S1.I1.i3.p1.3.m3.1.1.2"><csymbol cd="latexml" id="S1.I1.i3.p1.3.m3.1.1.2.1.cmml" xref="S1.I1.i3.p1.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S1.I1.i3.p1.3.m3.1.1.2.2.cmml" xref="S1.I1.i3.p1.3.m3.1.1.2.2">25.9</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.I1.i3.p1.3.m3.1c">-25.9\%</annotation></semantics></math>) results are severely degraded.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.1" class="ltx_p"><span id="S1.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Occasionally, quantization brings benefits.</span> Similar to <cite class="ltx_cite ltx_citemacro_citet">Ahia et al. (<a href="#bib.bib2" title="" class="ltx_ref">2021</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Ogueji et al. (<a href="#bib.bib44" title="" class="ltx_ref">2022</a>)</cite> on sparsity, we find that quantization <span id="S1.I1.i4.p1.1.2" class="ltx_text ltx_font_italic">benefits</span> model performance in some cases:
e.g., an average 1.3% boost across tasks for a 35B model quantized with W8A8.</p>
</div>
</li>
</ol>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">As the first to broadly study the impact of quantization on multilingual LLMs, our work is part of a wider body of literature that considers the impact of model design choices on downstream performance. Our results urge attention to multilingual performance at all stages of system design.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Background</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">Quantization compresses the weights and potentially activations of a neural network to lower-bit representations. Compression can be done by training the model at lower precision, known as Quantization Aware Training (QAT), or performed on the final model weights, known as Post Training Quantization (PTQ). Given the difficulties in training LLMs especially at precision lower than 16-bits floating point, PTQ methods which perform the quantization single-shot without needing gradient updates are highly desirable.
Training is completed at higher precision, then weights/activations are quantized without further training. In this work, we focus on <span id="S2.p1.1.1" class="ltx_text ltx_font_italic">post-training quantization</span> because of its simplicity and applicability at scale. PTQ of LLMs can be further categorized into:</p>
</div>
<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_font_bold ltx_title_paragraph">Weight-Only Quantization</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p1.2" class="ltx_p">Weight matrices are quantized offline and the compressed matrices are loaded from memory during inference. Quantized weight matrices have a smaller memory footprint compared to FP16 (<math id="S2.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.1.m1.1a"><mrow id="S2.SS0.SSS0.Px1.p1.1.m1.1b"><mn id="S2.SS0.SSS0.Px1.p1.1.m1.1.1">2</mn><mo lspace="0.222em" id="S2.SS0.SSS0.Px1.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.1.m1.1c">2\times</annotation></semantics></math> smaller for 8-bit and almost <math id="S2.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_math_unparsed" alttext="4\times" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.2.m2.1a"><mrow id="S2.SS0.SSS0.Px1.p1.2.m2.1b"><mn id="S2.SS0.SSS0.Px1.p1.2.m2.1.1">4</mn><mo lspace="0.222em" id="S2.SS0.SSS0.Px1.p1.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.2.m2.1c">4\times</annotation></semantics></math> smaller for 4-bit), enabling inference with less compute. In memory-bound scenarios, it also enables faster inference due to fewer bytes transferred from GPU memory to the compute units.</p>
</div>
<div id="S2.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p2.2" class="ltx_p">For a weight matrix <math id="S2.SS0.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="\mathbf{W}\in\mathbb{R}^{d_{in}\times d_{out}}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p2.1.m1.1a"><mrow id="S2.SS0.SSS0.Px1.p2.1.m1.1.1" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.2" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.2.cmml">𝐖</mi><mo id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.1" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.2" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.cmml"><msub id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.cmml"><mi id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.2" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.cmml"><mi id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.2" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.1" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.3" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.3.cmml">n</mi></mrow></msub><mo lspace="0.222em" rspace="0.222em" id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.1" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.1.cmml">×</mo><msub id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.2" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.2" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.1" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.3" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.1a" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.4" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.4.cmml">t</mi></mrow></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p2.1.m1.1b"><apply id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1"><in id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.1"></in><ci id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.2">𝐖</ci><apply id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3"><times id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.1"></times><apply id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.1.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.2.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.2">𝑑</ci><apply id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3"><times id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.1.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.1"></times><ci id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.2.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.2">𝑖</ci><ci id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.3.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.2.3.3">𝑛</ci></apply></apply><apply id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.2">𝑑</ci><apply id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3"><times id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.1"></times><ci id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.2">𝑜</ci><ci id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.3">𝑢</ci><ci id="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.4.cmml" xref="S2.SS0.SSS0.Px1.p2.1.m1.1.1.3.3.3.3.4">𝑡</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p2.1.m1.1c">\mathbf{W}\in\mathbb{R}^{d_{in}\times d_{out}}</annotation></semantics></math> and input <math id="S2.SS0.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="\mathbf{X}\in\mathbb{R}^{seq\times d_{in}}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p2.2.m2.1a"><mrow id="S2.SS0.SSS0.Px1.p2.2.m2.1.1" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.2" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.2.cmml">𝐗</mi><mo id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.1" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.2" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.cmml"><mrow id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.cmml"><mi id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.2" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.1" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.3" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.1a" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.4" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.4.cmml">q</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.1" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.1.cmml">×</mo><msub id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.2" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.2" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.1" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.3" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.3.cmml">n</mi></mrow></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p2.2.m2.1b"><apply id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1"><in id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.1"></in><ci id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.2">𝐗</ci><apply id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3"><times id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.1"></times><apply id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2"><times id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.1.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.1"></times><ci id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.2.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.2">𝑠</ci><ci id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.3.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.3">𝑒</ci><ci id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.4.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.2.4">𝑞</ci></apply><apply id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.2">𝑑</ci><apply id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3"><times id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.1"></times><ci id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.2">𝑖</ci><ci id="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p2.2.m2.1.1.3.3.3.3.3">𝑛</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p2.2.m2.1c">\mathbf{X}\in\mathbb{R}^{seq\times d_{in}}</annotation></semantics></math>, if only a single scaling factor is used for naive quantization (per-tensor), then the quantized weights are given by:</p>
<table id="S2.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E1.m1.3" class="ltx_math_unparsed" alttext="\mathbf{W}_{Q}=\Delta\cdot\left\lfloor\frac{\mathbf{W}}{\Delta}\right\rceil,\quad\Delta=\frac{\max(|\mathbf{W}|)}{2^{N-1}}" display="block"><semantics id="S2.E1.m1.3a"><mrow id="S2.E1.m1.3b"><msub id="S2.E1.m1.3.4"><mi id="S2.E1.m1.3.4.2">𝐖</mi><mi id="S2.E1.m1.3.4.3">Q</mi></msub><mo id="S2.E1.m1.3.5">=</mo><mi mathvariant="normal" id="S2.E1.m1.3.6">Δ</mi><mo lspace="0.222em" rspace="0.222em" id="S2.E1.m1.3.7">⋅</mo><mrow id="S2.E1.m1.3.8"><mo id="S2.E1.m1.3.8.1">⌊</mo><mfrac id="S2.E1.m1.3.8.2"><mi id="S2.E1.m1.3.8.2.2">𝐖</mi><mi mathvariant="normal" id="S2.E1.m1.3.8.2.3">Δ</mi></mfrac><mo id="S2.E1.m1.3.8.3">⌉</mo></mrow><mo rspace="1.167em" id="S2.E1.m1.3.9">,</mo><mi mathvariant="normal" id="S2.E1.m1.3.10">Δ</mi><mo id="S2.E1.m1.3.11">=</mo><mfrac id="S2.E1.m1.3.3"><mrow id="S2.E1.m1.3.3.3.3"><mi id="S2.E1.m1.2.2.2.2">max</mi><mo id="S2.E1.m1.3.3.3.3a">⁡</mo><mrow id="S2.E1.m1.3.3.3.3.1"><mo stretchy="false" id="S2.E1.m1.3.3.3.3.1.2">(</mo><mrow id="S2.E1.m1.3.3.3.3.1.1.2"><mo stretchy="false" id="S2.E1.m1.3.3.3.3.1.1.2.1">|</mo><mi id="S2.E1.m1.1.1.1.1">𝐖</mi><mo stretchy="false" id="S2.E1.m1.3.3.3.3.1.1.2.2">|</mo></mrow><mo stretchy="false" id="S2.E1.m1.3.3.3.3.1.3">)</mo></mrow></mrow><msup id="S2.E1.m1.3.3.5"><mn id="S2.E1.m1.3.3.5.2">2</mn><mrow id="S2.E1.m1.3.3.5.3"><mi id="S2.E1.m1.3.3.5.3.2">N</mi><mo id="S2.E1.m1.3.3.5.3.1">−</mo><mn id="S2.E1.m1.3.3.5.3.3">1</mn></mrow></msup></mfrac></mrow><annotation encoding="application/x-tex" id="S2.E1.m1.3c">\mathbf{W}_{Q}=\Delta\cdot\left\lfloor\frac{\mathbf{W}}{\Delta}\right\rceil,\quad\Delta=\frac{\max(|\mathbf{W}|)}{2^{N-1}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px1.p2.7" class="ltx_p">where <math id="S2.SS0.SSS0.Px1.p2.3.m1.1" class="ltx_Math" alttext="\Delta\in\mathbb{R}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p2.3.m1.1a"><mrow id="S2.SS0.SSS0.Px1.p2.3.m1.1.1" xref="S2.SS0.SSS0.Px1.p2.3.m1.1.1.cmml"><mi mathvariant="normal" id="S2.SS0.SSS0.Px1.p2.3.m1.1.1.2" xref="S2.SS0.SSS0.Px1.p2.3.m1.1.1.2.cmml">Δ</mi><mo id="S2.SS0.SSS0.Px1.p2.3.m1.1.1.1" xref="S2.SS0.SSS0.Px1.p2.3.m1.1.1.1.cmml">∈</mo><mi id="S2.SS0.SSS0.Px1.p2.3.m1.1.1.3" xref="S2.SS0.SSS0.Px1.p2.3.m1.1.1.3.cmml">ℝ</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p2.3.m1.1b"><apply id="S2.SS0.SSS0.Px1.p2.3.m1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p2.3.m1.1.1"><in id="S2.SS0.SSS0.Px1.p2.3.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p2.3.m1.1.1.1"></in><ci id="S2.SS0.SSS0.Px1.p2.3.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p2.3.m1.1.1.2">Δ</ci><ci id="S2.SS0.SSS0.Px1.p2.3.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p2.3.m1.1.1.3">ℝ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p2.3.m1.1c">\Delta\in\mathbb{R}</annotation></semantics></math> denotes the scale, <math id="S2.SS0.SSS0.Px1.p2.4.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS0.SSS0.Px1.p2.4.m2.1a"><mi id="S2.SS0.SSS0.Px1.p2.4.m2.1.1" xref="S2.SS0.SSS0.Px1.p2.4.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p2.4.m2.1b"><ci id="S2.SS0.SSS0.Px1.p2.4.m2.1.1.cmml" xref="S2.SS0.SSS0.Px1.p2.4.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p2.4.m2.1c">N</annotation></semantics></math> the bit precision, <math id="S2.SS0.SSS0.Px1.p2.5.m3.1" class="ltx_math_unparsed" alttext="|.|" display="inline"><semantics id="S2.SS0.SSS0.Px1.p2.5.m3.1a"><mrow id="S2.SS0.SSS0.Px1.p2.5.m3.1b"><mo fence="false" stretchy="false" id="S2.SS0.SSS0.Px1.p2.5.m3.1.1">|</mo><mo lspace="0.167em" rspace="0.167em" id="S2.SS0.SSS0.Px1.p2.5.m3.1.2">.</mo><mo fence="false" stretchy="false" id="S2.SS0.SSS0.Px1.p2.5.m3.1.3">|</mo></mrow><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p2.5.m3.1c">|.|</annotation></semantics></math> the absolute value over each element in <math id="S2.SS0.SSS0.Px1.p2.6.m4.1" class="ltx_Math" alttext="\mathbf{W}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p2.6.m4.1a"><mi id="S2.SS0.SSS0.Px1.p2.6.m4.1.1" xref="S2.SS0.SSS0.Px1.p2.6.m4.1.1.cmml">𝐖</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p2.6.m4.1b"><ci id="S2.SS0.SSS0.Px1.p2.6.m4.1.1.cmml" xref="S2.SS0.SSS0.Px1.p2.6.m4.1.1">𝐖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p2.6.m4.1c">\mathbf{W}</annotation></semantics></math> and <math id="S2.SS0.SSS0.Px1.p2.7.m5.1" class="ltx_math_unparsed" alttext="\left\lfloor.\right\rceil" display="inline"><semantics id="S2.SS0.SSS0.Px1.p2.7.m5.1a"><mrow id="S2.SS0.SSS0.Px1.p2.7.m5.1b"><mo id="S2.SS0.SSS0.Px1.p2.7.m5.1.1">⌊</mo><mo lspace="0em" rspace="0.167em" id="S2.SS0.SSS0.Px1.p2.7.m5.1.2">.</mo><mo id="S2.SS0.SSS0.Px1.p2.7.m5.1.3">⌉</mo></mrow><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p2.7.m5.1c">\left\lfloor.\right\rceil</annotation></semantics></math> rounding to the nearest integer.</p>
</div>
<div id="S2.SS0.SSS0.Px1.p3" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p3.4" class="ltx_p">A single scaling factor might not be enough if the distribution of parameters in the weight matrix has high variance; thus one could increase the granularity of quantization by using a scale for each output dimension (per-column), i.e., <math id="S2.SS0.SSS0.Px1.p3.1.m1.1" class="ltx_Math" alttext="\Delta\in\mathbb{R}^{d_{out}}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p3.1.m1.1a"><mrow id="S2.SS0.SSS0.Px1.p3.1.m1.1.1" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.cmml"><mi mathvariant="normal" id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.2" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.2.cmml">Δ</mi><mo id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.1" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.2" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.2.cmml">ℝ</mi><msub id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.2" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.2" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.1" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.3" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.1a" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.4" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.4.cmml">t</mi></mrow></msub></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p3.1.m1.1b"><apply id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1"><in id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.1"></in><ci id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.2">Δ</ci><apply id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.2">𝑑</ci><apply id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3"><times id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.1"></times><ci id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.2">𝑜</ci><ci id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.3">𝑢</ci><ci id="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.4.cmml" xref="S2.SS0.SSS0.Px1.p3.1.m1.1.1.3.3.3.4">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p3.1.m1.1c">\Delta\in\mathbb{R}^{d_{out}}</annotation></semantics></math>. However, when <math id="S2.SS0.SSS0.Px1.p3.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS0.SSS0.Px1.p3.2.m2.1a"><mi id="S2.SS0.SSS0.Px1.p3.2.m2.1.1" xref="S2.SS0.SSS0.Px1.p3.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p3.2.m2.1b"><ci id="S2.SS0.SSS0.Px1.p3.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px1.p3.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p3.2.m2.1c">N</annotation></semantics></math> is aggressively lowered to 4 bits or lower, even per-column granularity might be insufficient to cover the range of values in a column. The granularity can be further increased by using a shared scale for a subset of input dimensions called groups (<math id="S2.SS0.SSS0.Px1.p3.3.m3.1" class="ltx_Math" alttext="g" display="inline"><semantics id="S2.SS0.SSS0.Px1.p3.3.m3.1a"><mi id="S2.SS0.SSS0.Px1.p3.3.m3.1.1" xref="S2.SS0.SSS0.Px1.p3.3.m3.1.1.cmml">g</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p3.3.m3.1b"><ci id="S2.SS0.SSS0.Px1.p3.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px1.p3.3.m3.1.1">𝑔</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p3.3.m3.1c">g</annotation></semantics></math>), thus the scale <math id="S2.SS0.SSS0.Px1.p3.4.m4.1" class="ltx_Math" alttext="\Delta\in\mathbb{R}^{\frac{d_{in}}{g}\times d_{out}}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p3.4.m4.1a"><mrow id="S2.SS0.SSS0.Px1.p3.4.m4.1.1" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.cmml"><mi mathvariant="normal" id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.2" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.2.cmml">Δ</mi><mo id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.1" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.2" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.cmml"><mfrac id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.cmml"><msub id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.cmml"><mi id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.2" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.cmml"><mi id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.2" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.1" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.3" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.3.cmml">n</mi></mrow></msub><mi id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.3" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.3.cmml">g</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.1" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.1.cmml">×</mo><msub id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.2" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.2" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.1" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.3" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.1a" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.4" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.4.cmml">t</mi></mrow></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p3.4.m4.1b"><apply id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1"><in id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.1"></in><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.2">Δ</ci><apply id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3"><times id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.1"></times><apply id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2"><divide id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.1.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2"></divide><apply id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.1.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.2.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.2">𝑑</ci><apply id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3"><times id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.1.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.1"></times><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.2.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.2">𝑖</ci><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.3.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.2.3.3">𝑛</ci></apply></apply><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.3.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.2.3">𝑔</ci></apply><apply id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.2">𝑑</ci><apply id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3"><times id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.1"></times><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.2">𝑜</ci><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.3">𝑢</ci><ci id="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.4.cmml" xref="S2.SS0.SSS0.Px1.p3.4.m4.1.1.3.3.3.3.4">𝑡</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p3.4.m4.1c">\Delta\in\mathbb{R}^{\frac{d_{in}}{g}\times d_{out}}</annotation></semantics></math>. A commonly used group size is 128.</p>
</div>
<div id="S2.SS0.SSS0.Px1.p4" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p4.1" class="ltx_p">Equation <a href="#S2.E1" title="In Weight-Only Quantization ‣ 2 Background ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> gives the simplest way to quantize the weights. For <math id="S2.SS0.SSS0.Px1.p4.1.m1.1" class="ltx_Math" alttext="N\leq 4" display="inline"><semantics id="S2.SS0.SSS0.Px1.p4.1.m1.1a"><mrow id="S2.SS0.SSS0.Px1.p4.1.m1.1.1" xref="S2.SS0.SSS0.Px1.p4.1.m1.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p4.1.m1.1.1.2" xref="S2.SS0.SSS0.Px1.p4.1.m1.1.1.2.cmml">N</mi><mo id="S2.SS0.SSS0.Px1.p4.1.m1.1.1.1" xref="S2.SS0.SSS0.Px1.p4.1.m1.1.1.1.cmml">≤</mo><mn id="S2.SS0.SSS0.Px1.p4.1.m1.1.1.3" xref="S2.SS0.SSS0.Px1.p4.1.m1.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p4.1.m1.1b"><apply id="S2.SS0.SSS0.Px1.p4.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p4.1.m1.1.1"><leq id="S2.SS0.SSS0.Px1.p4.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p4.1.m1.1.1.1"></leq><ci id="S2.SS0.SSS0.Px1.p4.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p4.1.m1.1.1.2">𝑁</ci><cn type="integer" id="S2.SS0.SSS0.Px1.p4.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p4.1.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p4.1.m1.1c">N\leq 4</annotation></semantics></math> bits, using more advanced Weight-Only Quantization methods like GPTQ <cite class="ltx_cite ltx_citemacro_citep">(Frantar et al., <a href="#bib.bib19" title="" class="ltx_ref">2022</a>)</cite> or AWQ <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib35" title="" class="ltx_ref">2024</a>)</cite> leads to better downstream performance.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_font_bold ltx_title_paragraph">Weight-and-Activation Quantization</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px2.p1.4" class="ltx_p">As the name suggests, Weight-and-Activation Quantization quantizes the model activations alongside the weights. Unlike Weight-Only Quantization where weights can be quantized offline, quantization of activations happens at runtime. One could compute the quantization scales for various activations by using a small slice of training or validation data (static scaling) but this method typically has large degradation <cite class="ltx_cite ltx_citemacro_citep">(Xiao et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>)</cite>. For minimal degradation, it is preferred to calculate the quantization scaling factor dynamically (dynamic scaling) for each input on-the-fly. While quantizing activations is more difficult, reducing the precision of the activations alongside the weights enables the usage of specialized low-precision matrix multiplication hardware in modern GPUs leading to up to <math id="S2.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.1.m1.1a"><mrow id="S2.SS0.SSS0.Px2.p1.1.m1.1b"><mn id="S2.SS0.SSS0.Px2.p1.1.m1.1.1">2</mn><mo lspace="0.222em" id="S2.SS0.SSS0.Px2.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.1.m1.1c">2\times</annotation></semantics></math> improvement in throughput.
For a weight matrix <math id="S2.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{W}\in\mathbb{R}^{d_{in}\times d_{out}}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.2.m2.1a"><mrow id="S2.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.2" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml">𝐖</mi><mo id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.1" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.cmml"><msub id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.cmml"><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.2" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.2" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.1" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.3" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.3.cmml">n</mi></mrow></msub><mo lspace="0.222em" rspace="0.222em" id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.1" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.1.cmml">×</mo><msub id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.2" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.2" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.1" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.3" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.1a" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.4" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.4.cmml">t</mi></mrow></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.2.m2.1b"><apply id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1"><in id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.1"></in><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.2">𝐖</ci><apply id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3"><times id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.1"></times><apply id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.1.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.2.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.2">𝑑</ci><apply id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3"><times id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.1"></times><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.2">𝑖</ci><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.2.3.3">𝑛</ci></apply></apply><apply id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.2">𝑑</ci><apply id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3"><times id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.1"></times><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.2">𝑜</ci><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.3">𝑢</ci><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.4.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.3.3.4">𝑡</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.2.m2.1c">\mathbf{W}\in\mathbb{R}^{d_{in}\times d_{out}}</annotation></semantics></math> and input <math id="S2.SS0.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{X}\in\mathbb{R}^{seq\times d_{in}}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.3.m3.1a"><mrow id="S2.SS0.SSS0.Px2.p1.3.m3.1.1" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.2" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.2.cmml">𝐗</mi><mo id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.1" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.2" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.cmml"><mrow id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.cmml"><mi id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.2" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.1" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.3" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.1a" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.4" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.4.cmml">q</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.1" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.1.cmml">×</mo><msub id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.2" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.2" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.1" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.3" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.3.cmml">n</mi></mrow></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.3.m3.1b"><apply id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1"><in id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.1"></in><ci id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.2">𝐗</ci><apply id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3"><times id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.1"></times><apply id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2"><times id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.1.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.1"></times><ci id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.2.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.2">𝑠</ci><ci id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.3.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.3">𝑒</ci><ci id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.4.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.2.4">𝑞</ci></apply><apply id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.2">𝑑</ci><apply id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3"><times id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.1"></times><ci id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.2">𝑖</ci><ci id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.3">𝑛</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.3.m3.1c">\mathbf{X}\in\mathbb{R}^{seq\times d_{in}}</annotation></semantics></math>, naive Weight-and-Activation Quantization with per-token input granularity and per-column weight granularity generates output <math id="S2.SS0.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{Y}\in\mathbb{R}^{seq\times d_{out}}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.4.m4.1a"><mrow id="S2.SS0.SSS0.Px2.p1.4.m4.1.1" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.2" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml">𝐘</mi><mo id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.1" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.2" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.cmml"><mrow id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.cmml"><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.2" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.1" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.3" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.1a" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.4" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.4.cmml">q</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.1" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.1.cmml">×</mo><msub id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.2" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.2" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.1" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.3" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.1a" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.4" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.4.cmml">t</mi></mrow></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.4.m4.1b"><apply id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1"><in id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.1"></in><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.2">𝐘</ci><apply id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3"><times id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.1"></times><apply id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2"><times id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.1.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.1"></times><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.2.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.2">𝑠</ci><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.3.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.3">𝑒</ci><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.4.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.2.4">𝑞</ci></apply><apply id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.2">𝑑</ci><apply id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3"><times id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.1"></times><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.2">𝑜</ci><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.3">𝑢</ci><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.4.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.3.3.4">𝑡</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.4.m4.1c">\mathbf{Y}\in\mathbb{R}^{seq\times d_{out}}</annotation></semantics></math> by:</p>
<table id="A1.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E2.m1.12" class="ltx_math_unparsed" alttext="\displaystyle{\mathbf{W}_{Q}}_{:,j}=\left\lfloor\frac{\mathbf{W}_{:,j}}{\Delta^{W}_{:,j}}\right\rceil,\Delta^{W}_{:,j}=\frac{\max(|\mathbf{W}_{:,j}|)}{2^{N-1}}" display="inline"><semantics id="S2.E2.m1.12a"><mrow id="S2.E2.m1.12b"><mmultiscripts id="S2.E2.m1.12.13"><mi id="S2.E2.m1.12.13.2.2">𝐖</mi><mi id="S2.E2.m1.12.13.2.3">Q</mi><mrow id="S2.E2.m1.12.13a"></mrow><mrow id="S2.E2.m1.2.2.2.4"><mo rspace="0em" id="S2.E2.m1.1.1.1.1">:</mo><mo id="S2.E2.m1.2.2.2.4.1">,</mo><mi id="S2.E2.m1.2.2.2.2">j</mi></mrow><mrow id="S2.E2.m1.12.13b"></mrow></mmultiscripts><mo id="S2.E2.m1.12.14">=</mo><mrow id="S2.E2.m1.12.15"><mo id="S2.E2.m1.12.15.1">⌊</mo><mstyle displaystyle="true" id="S2.E2.m1.6.6"><mfrac id="S2.E2.m1.6.6a"><msub id="S2.E2.m1.4.4.2"><mi id="S2.E2.m1.4.4.2.4">𝐖</mi><mrow id="S2.E2.m1.4.4.2.2.2.4"><mo rspace="0em" id="S2.E2.m1.3.3.1.1.1.1">:</mo><mo id="S2.E2.m1.4.4.2.2.2.4.1">,</mo><mi id="S2.E2.m1.4.4.2.2.2.2">j</mi></mrow></msub><msubsup id="S2.E2.m1.6.6.4"><mi mathvariant="normal" id="S2.E2.m1.6.6.4.4.2">Δ</mi><mrow id="S2.E2.m1.6.6.4.2.2.4"><mo rspace="0em" id="S2.E2.m1.5.5.3.1.1.1">:</mo><mo id="S2.E2.m1.6.6.4.2.2.4.1">,</mo><mi id="S2.E2.m1.6.6.4.2.2.2">j</mi></mrow><mi id="S2.E2.m1.6.6.4.4.3">W</mi></msubsup></mfrac></mstyle><mo id="S2.E2.m1.12.15.2">⌉</mo></mrow><mo id="S2.E2.m1.12.16">,</mo><msubsup id="S2.E2.m1.12.17"><mi mathvariant="normal" id="S2.E2.m1.12.17.2.2">Δ</mi><mrow id="S2.E2.m1.8.8.2.4"><mo rspace="0em" id="S2.E2.m1.7.7.1.1">:</mo><mo id="S2.E2.m1.8.8.2.4.1">,</mo><mi id="S2.E2.m1.8.8.2.2">j</mi></mrow><mi id="S2.E2.m1.12.17.2.3">W</mi></msubsup><mo id="S2.E2.m1.12.18">=</mo><mstyle displaystyle="true" id="S2.E2.m1.12.12"><mfrac id="S2.E2.m1.12.12a"><mrow id="S2.E2.m1.12.12.4.4"><mi id="S2.E2.m1.11.11.3.3">max</mi><mo id="S2.E2.m1.12.12.4.4a">⁡</mo><mrow id="S2.E2.m1.12.12.4.4.1"><mo stretchy="false" id="S2.E2.m1.12.12.4.4.1.2">(</mo><mrow id="S2.E2.m1.12.12.4.4.1.1.1"><mo stretchy="false" id="S2.E2.m1.12.12.4.4.1.1.1.2">|</mo><msub id="S2.E2.m1.12.12.4.4.1.1.1.1"><mi id="S2.E2.m1.12.12.4.4.1.1.1.1.2">𝐖</mi><mrow id="S2.E2.m1.10.10.2.2.2.4"><mo rspace="0em" id="S2.E2.m1.9.9.1.1.1.1">:</mo><mo id="S2.E2.m1.10.10.2.2.2.4.1">,</mo><mi id="S2.E2.m1.10.10.2.2.2.2">j</mi></mrow></msub><mo stretchy="false" id="S2.E2.m1.12.12.4.4.1.1.1.3">|</mo></mrow><mo stretchy="false" id="S2.E2.m1.12.12.4.4.1.3">)</mo></mrow></mrow><msup id="S2.E2.m1.12.12.6"><mn id="S2.E2.m1.12.12.6.2">2</mn><mrow id="S2.E2.m1.12.12.6.3"><mi id="S2.E2.m1.12.12.6.3.2">N</mi><mo id="S2.E2.m1.12.12.6.3.1">−</mo><mn id="S2.E2.m1.12.12.6.3.3">1</mn></mrow></msup></mfrac></mstyle></mrow><annotation encoding="application/x-tex" id="S2.E2.m1.12c">\displaystyle{\mathbf{W}_{Q}}_{:,j}=\left\lfloor\frac{\mathbf{W}_{:,j}}{\Delta^{W}_{:,j}}\right\rceil,\Delta^{W}_{:,j}=\frac{\max(|\mathbf{W}_{:,j}|)}{2^{N-1}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
<tbody id="S2.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E3.m1.12" class="ltx_math_unparsed" alttext="\displaystyle{\mathbf{X}_{Q}}_{i,:}=\left\lfloor\frac{\mathbf{X}_{i,:}}{\Delta^{X}_{i,:}}\right\rceil,\Delta^{X}_{i,:}=\frac{\max(|\mathbf{X}_{i,:}|)}{2^{N-1}}" display="inline"><semantics id="S2.E3.m1.12a"><mrow id="S2.E3.m1.12b"><mmultiscripts id="S2.E3.m1.12.13"><mi id="S2.E3.m1.12.13.2.2">𝐗</mi><mi id="S2.E3.m1.12.13.2.3">Q</mi><mrow id="S2.E3.m1.12.13a"></mrow><mrow id="S2.E3.m1.2.2.2.4"><mi id="S2.E3.m1.1.1.1.1">i</mi><mo id="S2.E3.m1.2.2.2.4.1">,</mo><mo id="S2.E3.m1.2.2.2.2">:</mo></mrow><mrow id="S2.E3.m1.12.13b"></mrow></mmultiscripts><mo id="S2.E3.m1.12.14">=</mo><mrow id="S2.E3.m1.12.15"><mo id="S2.E3.m1.12.15.1">⌊</mo><mstyle displaystyle="true" id="S2.E3.m1.6.6"><mfrac id="S2.E3.m1.6.6a"><msub id="S2.E3.m1.4.4.2"><mi id="S2.E3.m1.4.4.2.4">𝐗</mi><mrow id="S2.E3.m1.4.4.2.2.2.4"><mi id="S2.E3.m1.3.3.1.1.1.1">i</mi><mo id="S2.E3.m1.4.4.2.2.2.4.1">,</mo><mo id="S2.E3.m1.4.4.2.2.2.2">:</mo></mrow></msub><msubsup id="S2.E3.m1.6.6.4"><mi mathvariant="normal" id="S2.E3.m1.6.6.4.4.2">Δ</mi><mrow id="S2.E3.m1.6.6.4.2.2.4"><mi id="S2.E3.m1.5.5.3.1.1.1">i</mi><mo id="S2.E3.m1.6.6.4.2.2.4.1">,</mo><mo id="S2.E3.m1.6.6.4.2.2.2">:</mo></mrow><mi id="S2.E3.m1.6.6.4.4.3">X</mi></msubsup></mfrac></mstyle><mo id="S2.E3.m1.12.15.2">⌉</mo></mrow><mo id="S2.E3.m1.12.16">,</mo><msubsup id="S2.E3.m1.12.17"><mi mathvariant="normal" id="S2.E3.m1.12.17.2.2">Δ</mi><mrow id="S2.E3.m1.8.8.2.4"><mi id="S2.E3.m1.7.7.1.1">i</mi><mo id="S2.E3.m1.8.8.2.4.1">,</mo><mo id="S2.E3.m1.8.8.2.2">:</mo></mrow><mi id="S2.E3.m1.12.17.2.3">X</mi></msubsup><mo id="S2.E3.m1.12.18">=</mo><mstyle displaystyle="true" id="S2.E3.m1.12.12"><mfrac id="S2.E3.m1.12.12a"><mrow id="S2.E3.m1.12.12.4.4"><mi id="S2.E3.m1.11.11.3.3">max</mi><mo id="S2.E3.m1.12.12.4.4a">⁡</mo><mrow id="S2.E3.m1.12.12.4.4.1"><mo stretchy="false" id="S2.E3.m1.12.12.4.4.1.2">(</mo><mrow id="S2.E3.m1.12.12.4.4.1.1.1"><mo stretchy="false" id="S2.E3.m1.12.12.4.4.1.1.1.2">|</mo><msub id="S2.E3.m1.12.12.4.4.1.1.1.1"><mi id="S2.E3.m1.12.12.4.4.1.1.1.1.2">𝐗</mi><mrow id="S2.E3.m1.10.10.2.2.2.4"><mi id="S2.E3.m1.9.9.1.1.1.1">i</mi><mo id="S2.E3.m1.10.10.2.2.2.4.1">,</mo><mo id="S2.E3.m1.10.10.2.2.2.2">:</mo></mrow></msub><mo stretchy="false" id="S2.E3.m1.12.12.4.4.1.1.1.3">|</mo></mrow><mo stretchy="false" id="S2.E3.m1.12.12.4.4.1.3">)</mo></mrow></mrow><msup id="S2.E3.m1.12.12.6"><mn id="S2.E3.m1.12.12.6.2">2</mn><mrow id="S2.E3.m1.12.12.6.3"><mi id="S2.E3.m1.12.12.6.3.2">N</mi><mo id="S2.E3.m1.12.12.6.3.1">−</mo><mn id="S2.E3.m1.12.12.6.3.3">1</mn></mrow></msup></mfrac></mstyle></mrow><annotation encoding="application/x-tex" id="S2.E3.m1.12c">\displaystyle{\mathbf{X}_{Q}}_{i,:}=\left\lfloor\frac{\mathbf{X}_{i,:}}{\Delta^{X}_{i,:}}\right\rceil,\Delta^{X}_{i,:}=\frac{\max(|\mathbf{X}_{i,:}|)}{2^{N-1}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
<tbody id="S2.E4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E4.m1.1" class="ltx_Math" alttext="\displaystyle Y=\Delta^{X}\odot(\mathbf{X}_{Q}\mathbf{W}_{Q})\odot\Delta^{W}" display="inline"><semantics id="S2.E4.m1.1a"><mrow id="S2.E4.m1.1.1" xref="S2.E4.m1.1.1.cmml"><mi id="S2.E4.m1.1.1.3" xref="S2.E4.m1.1.1.3.cmml">Y</mi><mo id="S2.E4.m1.1.1.2" xref="S2.E4.m1.1.1.2.cmml">=</mo><mrow id="S2.E4.m1.1.1.1" xref="S2.E4.m1.1.1.1.cmml"><msup id="S2.E4.m1.1.1.1.3" xref="S2.E4.m1.1.1.1.3.cmml"><mi mathvariant="normal" id="S2.E4.m1.1.1.1.3.2" xref="S2.E4.m1.1.1.1.3.2.cmml">Δ</mi><mi id="S2.E4.m1.1.1.1.3.3" xref="S2.E4.m1.1.1.1.3.3.cmml">X</mi></msup><mo lspace="0.222em" rspace="0.222em" id="S2.E4.m1.1.1.1.2" xref="S2.E4.m1.1.1.1.2.cmml">⊙</mo><mrow id="S2.E4.m1.1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E4.m1.1.1.1.1.1.2" xref="S2.E4.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E4.m1.1.1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.1.1.cmml"><msub id="S2.E4.m1.1.1.1.1.1.1.2" xref="S2.E4.m1.1.1.1.1.1.1.2.cmml"><mi id="S2.E4.m1.1.1.1.1.1.1.2.2" xref="S2.E4.m1.1.1.1.1.1.1.2.2.cmml">𝐗</mi><mi id="S2.E4.m1.1.1.1.1.1.1.2.3" xref="S2.E4.m1.1.1.1.1.1.1.2.3.cmml">Q</mi></msub><mo lspace="0em" rspace="0em" id="S2.E4.m1.1.1.1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S2.E4.m1.1.1.1.1.1.1.3" xref="S2.E4.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.E4.m1.1.1.1.1.1.1.3.2" xref="S2.E4.m1.1.1.1.1.1.1.3.2.cmml">𝐖</mi><mi id="S2.E4.m1.1.1.1.1.1.1.3.3" xref="S2.E4.m1.1.1.1.1.1.1.3.3.cmml">Q</mi></msub></mrow><mo rspace="0.055em" stretchy="false" id="S2.E4.m1.1.1.1.1.1.3" xref="S2.E4.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="S2.E4.m1.1.1.1.2a" xref="S2.E4.m1.1.1.1.2.cmml">⊙</mo><msup id="S2.E4.m1.1.1.1.4" xref="S2.E4.m1.1.1.1.4.cmml"><mi mathvariant="normal" id="S2.E4.m1.1.1.1.4.2" xref="S2.E4.m1.1.1.1.4.2.cmml">Δ</mi><mi id="S2.E4.m1.1.1.1.4.3" xref="S2.E4.m1.1.1.1.4.3.cmml">W</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E4.m1.1b"><apply id="S2.E4.m1.1.1.cmml" xref="S2.E4.m1.1.1"><eq id="S2.E4.m1.1.1.2.cmml" xref="S2.E4.m1.1.1.2"></eq><ci id="S2.E4.m1.1.1.3.cmml" xref="S2.E4.m1.1.1.3">𝑌</ci><apply id="S2.E4.m1.1.1.1.cmml" xref="S2.E4.m1.1.1.1"><csymbol cd="latexml" id="S2.E4.m1.1.1.1.2.cmml" xref="S2.E4.m1.1.1.1.2">direct-product</csymbol><apply id="S2.E4.m1.1.1.1.3.cmml" xref="S2.E4.m1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.3.1.cmml" xref="S2.E4.m1.1.1.1.3">superscript</csymbol><ci id="S2.E4.m1.1.1.1.3.2.cmml" xref="S2.E4.m1.1.1.1.3.2">Δ</ci><ci id="S2.E4.m1.1.1.1.3.3.cmml" xref="S2.E4.m1.1.1.1.3.3">𝑋</ci></apply><apply id="S2.E4.m1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.1"><times id="S2.E4.m1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1.1"></times><apply id="S2.E4.m1.1.1.1.1.1.1.2.cmml" xref="S2.E4.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.1.1.1.2.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E4.m1.1.1.1.1.1.1.2.2.cmml" xref="S2.E4.m1.1.1.1.1.1.1.2.2">𝐗</ci><ci id="S2.E4.m1.1.1.1.1.1.1.2.3.cmml" xref="S2.E4.m1.1.1.1.1.1.1.2.3">𝑄</ci></apply><apply id="S2.E4.m1.1.1.1.1.1.1.3.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E4.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3.2">𝐖</ci><ci id="S2.E4.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.E4.m1.1.1.1.1.1.1.3.3">𝑄</ci></apply></apply><apply id="S2.E4.m1.1.1.1.4.cmml" xref="S2.E4.m1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.4.1.cmml" xref="S2.E4.m1.1.1.1.4">superscript</csymbol><ci id="S2.E4.m1.1.1.1.4.2.cmml" xref="S2.E4.m1.1.1.1.4.2">Δ</ci><ci id="S2.E4.m1.1.1.1.4.3.cmml" xref="S2.E4.m1.1.1.1.4.3">𝑊</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E4.m1.1c">\displaystyle Y=\Delta^{X}\odot(\mathbf{X}_{Q}\mathbf{W}_{Q})\odot\Delta^{W}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px2.p1.7" class="ltx_p">where, <math id="S2.SS0.SSS0.Px2.p1.5.m1.1" class="ltx_Math" alttext="\Delta^{W}\in\mathbb{R}^{d_{out}}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.5.m1.1a"><mrow id="S2.SS0.SSS0.Px2.p1.5.m1.1.1" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.cmml"><msup id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.cmml"><mi mathvariant="normal" id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.2" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.2.cmml">Δ</mi><mi id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.3" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.3.cmml">W</mi></msup><mo id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.1" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.2" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.2.cmml">ℝ</mi><msub id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.2" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.2.cmml">d</mi><mrow id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.2" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.2.cmml">o</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.1" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.3" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.1a" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.4" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.4.cmml">t</mi></mrow></msub></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.5.m1.1b"><apply id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1"><in id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.1"></in><apply id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.2">Δ</ci><ci id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.2.3">𝑊</ci></apply><apply id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.2">𝑑</ci><apply id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3"><times id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.1"></times><ci id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.2">𝑜</ci><ci id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.3">𝑢</ci><ci id="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.4.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m1.1.1.3.3.3.4">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.5.m1.1c">\Delta^{W}\in\mathbb{R}^{d_{out}}</annotation></semantics></math>, <math id="S2.SS0.SSS0.Px2.p1.6.m2.1" class="ltx_Math" alttext="\Delta^{X}\in\mathbb{R}^{seq}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.6.m2.1a"><mrow id="S2.SS0.SSS0.Px2.p1.6.m2.1.1" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.cmml"><msup id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.cmml"><mi mathvariant="normal" id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.2" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.2.cmml">Δ</mi><mi id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.3" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.3.cmml">X</mi></msup><mo id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.1" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.2" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.2" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.1" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.3" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.1a" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.4" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.4.cmml">q</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.6.m2.1b"><apply id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1"><in id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.1"></in><apply id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.2">Δ</ci><ci id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.2.3">𝑋</ci></apply><apply id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3"><times id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.1"></times><ci id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.2">𝑠</ci><ci id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.3">𝑒</ci><ci id="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.4.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m2.1.1.3.3.4">𝑞</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.6.m2.1c">\Delta^{X}\in\mathbb{R}^{seq}</annotation></semantics></math> and <math id="S2.SS0.SSS0.Px2.p1.7.m3.1" class="ltx_Math" alttext="\odot" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.7.m3.1a"><mo id="S2.SS0.SSS0.Px2.p1.7.m3.1.1" xref="S2.SS0.SSS0.Px2.p1.7.m3.1.1.cmml">⊙</mo><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.7.m3.1b"><csymbol cd="latexml" id="S2.SS0.SSS0.Px2.p1.7.m3.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m3.1.1">direct-product</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.7.m3.1c">\odot</annotation></semantics></math> denotes element-wise multiplication by broadcasting the elements to match the shape of the operands.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Experiment Set-up</h2>

<section id="S3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Models</h4>

<div id="S3.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px1.p1.1" class="ltx_p">We evaluate Command R+<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span><a target="_blank" href="https://docs.cohere.com/docs/command-r-plus" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://docs.cohere.com/docs/command-r-plus</a></span></span></span>,
Command R<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span><a target="_blank" href="https://docs.cohere.com/docs/command-r" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://docs.cohere.com/docs/command-r</a></span></span></span>, and Aya 23 models <cite class="ltx_cite ltx_citemacro_citep">(Aryabumi et al., <a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite> as representatives of state-of-the-art multilingual LLMs. Command models are 103 and 35 billion parameters, and Aya 23 models are 35 and 8 billion parameters. We quantize the models using the weights available on HuggingFace.</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Quantization</h4>

<div id="S3.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p1.1" class="ltx_p">For Command R and R+, we evaluate both <span id="S3.SS0.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_bold">weight-only quantization</span> at 8-bit (<span id="S3.SS0.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_bold">W8</span> with per-column scaling) and 4-bit (<span id="S3.SS0.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_bold">W4-g</span> with group-wise scaling using GPTQ <cite class="ltx_cite ltx_citemacro_citep">(Frantar et al., <a href="#bib.bib19" title="" class="ltx_ref">2022</a>)</cite>), as well as <span id="S3.SS0.SSS0.Px2.p1.1.4" class="ltx_text ltx_font_bold">weight-and-activation quantization</span> at 8-bit (<span id="S3.SS0.SSS0.Px2.p1.1.5" class="ltx_text ltx_font_bold">W8A8</span> with per-column scaling for weights and per-token scaling for activations).</p>
</div>
<div id="S3.SS0.SSS0.Px2.p2" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p2.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Ahmadian et al. (<a href="#bib.bib4" title="" class="ltx_ref">2024</a>)</cite> show that if the model is trained with the right hyper-parameters, naive Weight-and-Activation Quantization has minimal degradation. Otherwise, one may leverage SmoothQuant <cite class="ltx_cite ltx_citemacro_citep">(Xiao et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>)</cite> to smoothen the distribution of activations making them more amenable to quantization. We therefore also explore <span id="S3.SS0.SSS0.Px2.p2.1.1" class="ltx_text ltx_font_bold">W8A8-SmoothQuant</span> (a W8A8 variant with SmoothQuant) for Command R+ as well as a 4-bit weight-only quantized variant with column-wise scaling (<span id="S3.SS0.SSS0.Px2.p2.1.2" class="ltx_text ltx_font_bold">W4</span>) to understand the impact of scaling granularity at extremely low-bit precision. Following <cite class="ltx_cite ltx_citemacro_citep">(Frantar et al., <a href="#bib.bib19" title="" class="ltx_ref">2022</a>; Xiao et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>)</cite>, we use 128 English samples for calibration for SmoothQuant and GPTQ.</p>
</div>
<div id="S3.SS0.SSS0.Px2.p3" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p3.1" class="ltx_p">For Aya 23 8B and 35B, we use bitsandbytes<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span><a target="_blank" href="https://github.com/TimDettmers/bitsandbytes" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/TimDettmers/bitsandbytes</a></span></span></span> to obtain 8-bit and 4-bit quantized models. Bitsandbytes uses LLM.int8() <cite class="ltx_cite ltx_citemacro_citep">(Dettmers et al., <a href="#bib.bib15" title="" class="ltx_ref">2022</a>)</cite>—similar to W8A8 described above except that it performs certain computations in FP16. Bitsandbytes 4-bit uses the NF4 datatype <cite class="ltx_cite ltx_citemacro_citep">(Dettmers et al., <a href="#bib.bib16" title="" class="ltx_ref">2023</a>)</cite> to perform Quantile Quantization which limits degradation at the expense of inference speedups.</p>
</div>
</section>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Automatic Evaluation</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">We evaluate in 10 primary languages: <span id="S3.SS1.p1.1.1" class="ltx_text ltx_font_italic">Arabic</span>, <span id="S3.SS1.p1.1.2" class="ltx_text ltx_font_italic">French</span>, <span id="S3.SS1.p1.1.3" class="ltx_text ltx_font_italic">German</span>, <span id="S3.SS1.p1.1.4" class="ltx_text ltx_font_italic">English</span>, <span id="S3.SS1.p1.1.5" class="ltx_text ltx_font_italic">Spanish</span>, <span id="S3.SS1.p1.1.6" class="ltx_text ltx_font_italic">Italian</span>, <span id="S3.SS1.p1.1.7" class="ltx_text ltx_font_italic">Portuguese</span>, <span id="S3.SS1.p1.1.8" class="ltx_text ltx_font_italic">Korean</span>, <span id="S3.SS1.p1.1.9" class="ltx_text ltx_font_italic">Japanese</span>, and <span id="S3.SS1.p1.1.10" class="ltx_text ltx_font_italic">Chinese</span>. Quantized models are compared to the original <span id="S3.SS1.p1.1.11" class="ltx_text ltx_font_bold">FP16</span> versions, and we primarily report results as <span id="S3.SS1.p1.1.12" class="ltx_text ltx_font_bold">relative degradation</span> compared to this FP16 baseline:</p>
<table id="S3.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E5.m1.1" class="ltx_math_unparsed" alttext="\%\Delta=\frac{\text{score}_{\text{quantized}}-\text{score}_{\text{FP16}}}{\text{score}_{\text{FP16}}}*100" display="block"><semantics id="S3.E5.m1.1a"><mrow id="S3.E5.m1.1b"><mo id="S3.E5.m1.1.1">%</mo><mi mathvariant="normal" id="S3.E5.m1.1.2">Δ</mi><mo id="S3.E5.m1.1.3">=</mo><mfrac id="S3.E5.m1.1.4"><mrow id="S3.E5.m1.1.4.2"><msub id="S3.E5.m1.1.4.2.2"><mtext id="S3.E5.m1.1.4.2.2.2">score</mtext><mtext id="S3.E5.m1.1.4.2.2.3">quantized</mtext></msub><mo id="S3.E5.m1.1.4.2.1">−</mo><msub id="S3.E5.m1.1.4.2.3"><mtext id="S3.E5.m1.1.4.2.3.2">score</mtext><mtext id="S3.E5.m1.1.4.2.3.3">FP16</mtext></msub></mrow><msub id="S3.E5.m1.1.4.3"><mtext id="S3.E5.m1.1.4.3.2">score</mtext><mtext id="S3.E5.m1.1.4.3.3">FP16</mtext></msub></mfrac><mo lspace="0.222em" rspace="0.222em" id="S3.E5.m1.1.5">∗</mo><mn id="S3.E5.m1.1.6">100</mn></mrow><annotation encoding="application/x-tex" id="S3.E5.m1.1c">\%\Delta=\frac{\text{score}_{\text{quantized}}-\text{score}_{\text{FP16}}}{\text{score}_{\text{FP16}}}*100</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p1.2" class="ltx_p">Raw numeric results are in the Appendix. Results are averaged over 5 runs.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>k=0, p=0.75, temp=0.3, except mMMLU, which, as a QA eval, is run deterministically with temp=0.</span></span></span></p>
</div>
<section id="S3.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Multilingual MMLU (mMMLU)</h4>

<div id="S3.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px1.p1.1" class="ltx_p">This multi-domain question answering task consists of 14,000+ multiple-choice questions. We translate MMLU <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib23" title="" class="ltx_ref">2020</a>)</cite> to 9 languages with Google Translate and refer to this version as mMMLU. We measure accuracy in a 5-shot setting. An example is in Table <a href="#A1.T1" title="Table A1 ‣ A.1 Prompts for mMMLU and LLM-as-a-Judge ‣ Appendix A Appendix ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A1</span></a>.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">MGSM <cite class="ltx_cite ltx_citemacro_citep">(Shi et al., <a href="#bib.bib52" title="" class="ltx_ref">2023</a>)</cite>
</h4>

<div id="S3.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px2.p1.1" class="ltx_p">MGSM is a generative mathematics evaluation set manually translated from GSM8K <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite>. Of our target languages, it is available for German, Spanish, French, Japanese, and Chinese. We report accuracy over the 250-item test set for each language.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">FLORES-200 <cite class="ltx_cite ltx_citemacro_citep">(Costa-jussà et al., <a href="#bib.bib12" title="" class="ltx_ref">2022b</a>)</cite>
</h4>

<div id="S3.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px3.p1.1" class="ltx_p">This well-known multi-way parallel test set evaluates translation capabilities. We translate into and out of English, and report SacreBLEU <cite class="ltx_cite ltx_citemacro_cite">Post (<a href="#bib.bib48" title="" class="ltx_ref">2018</a>)</cite>.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Language Confusion <cite class="ltx_cite ltx_citemacro_citep">(Marchisio et al., <a href="#bib.bib39" title="" class="ltx_ref">2024</a>)</cite>
</h4>

<div id="S3.SS1.SSS0.Px4.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px4.p1.3" class="ltx_p">These test sets assess a model’s ability to respond in a user’s desired language. In the monolingual setting, prompts are in language <math id="S3.SS1.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S3.SS1.SSS0.Px4.p1.1.m1.1a"><mi id="S3.SS1.SSS0.Px4.p1.1.m1.1.1" xref="S3.SS1.SSS0.Px4.p1.1.m1.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px4.p1.1.m1.1b"><ci id="S3.SS1.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S3.SS1.SSS0.Px4.p1.1.m1.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px4.p1.1.m1.1c">l</annotation></semantics></math> and the model must respond in language <math id="S3.SS1.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S3.SS1.SSS0.Px4.p1.2.m2.1a"><mi id="S3.SS1.SSS0.Px4.p1.2.m2.1.1" xref="S3.SS1.SSS0.Px4.p1.2.m2.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px4.p1.2.m2.1b"><ci id="S3.SS1.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S3.SS1.SSS0.Px4.p1.2.m2.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px4.p1.2.m2.1c">l</annotation></semantics></math>. For instance, a user prompts in Arabic, so implicitly desires an Arabic response. In the cross-lingual variant, a prompt is provided in English but the user requests output in a different language <math id="S3.SS1.SSS0.Px4.p1.3.m3.1" class="ltx_Math" alttext="l^{\prime}" display="inline"><semantics id="S3.SS1.SSS0.Px4.p1.3.m3.1a"><msup id="S3.SS1.SSS0.Px4.p1.3.m3.1.1" xref="S3.SS1.SSS0.Px4.p1.3.m3.1.1.cmml"><mi id="S3.SS1.SSS0.Px4.p1.3.m3.1.1.2" xref="S3.SS1.SSS0.Px4.p1.3.m3.1.1.2.cmml">l</mi><mo id="S3.SS1.SSS0.Px4.p1.3.m3.1.1.3" xref="S3.SS1.SSS0.Px4.p1.3.m3.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.SSS0.Px4.p1.3.m3.1b"><apply id="S3.SS1.SSS0.Px4.p1.3.m3.1.1.cmml" xref="S3.SS1.SSS0.Px4.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.SSS0.Px4.p1.3.m3.1.1.1.cmml" xref="S3.SS1.SSS0.Px4.p1.3.m3.1.1">superscript</csymbol><ci id="S3.SS1.SSS0.Px4.p1.3.m3.1.1.2.cmml" xref="S3.SS1.SSS0.Px4.p1.3.m3.1.1.2">𝑙</ci><ci id="S3.SS1.SSS0.Px4.p1.3.m3.1.1.3.cmml" xref="S3.SS1.SSS0.Px4.p1.3.m3.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSS0.Px4.p1.3.m3.1c">l^{\prime}</annotation></semantics></math>.<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>An example from the <span id="footnote5.1" class="ltx_text ltx_font_italic">Okapi</span> subsection of the evaluation is: “Reply in Spanish. Explain a common misconception about your topic. Topic: Using AI to Augment Human Capabilities”</span></span></span> <span id="S3.SS1.SSS0.Px4.p1.3.1" class="ltx_text ltx_font_italic">fastText</span> <cite class="ltx_cite ltx_citemacro_cite">Joulin et al. (<a href="#bib.bib26" title="" class="ltx_ref">2016</a>)</cite> language identification is run over the output. We report <span id="S3.SS1.SSS0.Px4.p1.3.2" class="ltx_text ltx_font_italic">line-level pass rate</span> (<span id="S3.SS1.SSS0.Px4.p1.3.3" class="ltx_text ltx_font_bold">LPR)</span>, i.e., the percentage of responses for which all lines in the response are in the user’s desired language.</p>
</div>
</section>
<section id="S3.SS1.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Aya Evaluation</h4>

<div id="S3.SS1.SSS0.Px5.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px5.p1.1" class="ltx_p">Aya 23 models are evaluated using an extended version of the Aya evaluation setup <cite class="ltx_cite ltx_citemacro_citep">(Aryabumi et al., <a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite> using the unseen discriminative tasks (XWinograd <cite class="ltx_cite ltx_citemacro_citep">(Muennighoff et al., <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite>, XCOPA <cite class="ltx_cite ltx_citemacro_citep">(Ponti et al., <a href="#bib.bib47" title="" class="ltx_ref">2020</a>)</cite>, XStoryCloze <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib36" title="" class="ltx_ref">2022</a>)</cite>), mMMLU <cite class="ltx_cite ltx_citemacro_citep">(Okapi; Dac Lai et al., <a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite>, MGSM, and Belebele <cite class="ltx_cite ltx_citemacro_citep">(Bandarkar et al., <a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite> from <span id="S3.SS1.SSS0.Px5.p1.1.1" class="ltx_text ltx_font_typewriter">eval-harness</span> <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib20" title="" class="ltx_ref">2023</a>)</cite>.<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>We follow the setup used by <cite class="ltx_cite ltx_citemacro_citet">Üstün et al. (<a href="#bib.bib60" title="" class="ltx_ref">2024</a>)</cite>: each evaluation is run once, and for FLORES, no sampling is used and metric is spBLEU.</span></span></span> We evaluate models on languages included in the covered 23 languages, except for the unseen tasks where we use all available languages.<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>mMMLU: ar, de, es, fr, hi, id, it, nl, pt, ro, ru, uk, vi, zh. MGSM: de, es, fr, ja, ru, zh. Belebele: {mMMLU} + cs, fa, el, ja, ko, pl, tr. FLORES: {Belebele} + he.</span></span></span> Aya evaluations allow us to add: <span id="S3.SS1.SSS0.Px5.p1.1.2" class="ltx_text ltx_font_italic">Czech, Greek, Hebrew, Hindi, Indonesian, Dutch, Persian, Polish, Romanian, Russian, Turkish, Ukrainian, Vietnamese</span>.</p>
</div>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Human Evaluation</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">We run human evaluation in <span id="S3.SS2.p1.1.1" class="ltx_text ltx_font_italic">Spanish</span>, <span id="S3.SS2.p1.1.2" class="ltx_text ltx_font_italic">French</span>, <span id="S3.SS2.p1.1.3" class="ltx_text ltx_font_italic">Korean</span>, and <span id="S3.SS2.p1.1.4" class="ltx_text ltx_font_italic">Japanese</span>.</p>
</div>
<section id="S3.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Internal Evaluation Suite</h4>

<div id="S3.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px1.p1.1" class="ltx_p">150 diverse prompts designed to be more complex than public evaluation benchmarks. As such, we expect greater degradation with increased quantization given the difficulty of the samples. Prompts for all four languages are translated by humans from an English seed prompt, ensuring that respective language-specific subsets share the same prompts.</p>
</div>
</section>
<section id="S3.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Aya Dolly-200  <cite class="ltx_cite ltx_citemacro_citep">(Singh et al., <a href="#bib.bib53" title="" class="ltx_ref">2024</a>)</cite>
</h4>

<div id="S3.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px2.p1.1" class="ltx_p">We use multilingual data from the Aya Evaluation Suite to assess open-ended generation capabilities. For Korean and Japanese, we use prompts from the Aya Dolly-200 test set (<span id="S3.SS2.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_bold">dolly-machine-translated</span>), which are automatically translated from English Dolly-15k <cite class="ltx_cite ltx_citemacro_citep">(Conover et al., <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite> then human-curated to avoid references requiring specific cultural or geographic knowledge.
For French and Spanish, we use <span id="S3.SS2.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_bold">dolly-human-edited</span>, a human post-edited version of <span id="S3.SS2.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_bold">dolly-machine-translated</span>. For each language, we evaluate using the first 150 prompts.</p>
</div>
</section>
<section id="S3.SS2.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Annotator Statistics</h4>

<div id="S3.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px3.p1.1" class="ltx_p">Annotations and translations were completed by native-level speakers of the respective languages, each of whom is also fluent in English.
Annotators were paid by the hour, with compensation above the federal minimum wage of the country of employment.</p>
</div>
</section>
<section id="S3.SS2.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Annotation Interface</h4>

<div id="S3.SS2.SSS0.Px4.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px4.p1.1" class="ltx_p">We use a pairwise evaluation setup. Annotators see a prompt and two (shuffled) completions of the FP16 model and a quantized variant. They rate each response on a 5-point Likert scale, then express a preference between the two model outputs (tie, weak preference, strong preference). We encourage annotators to avoid tied rankings. Win rates are based on the ranking preferences alone.</p>
</div>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>LLM/RM-as-a-Judge</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">Because human evaluation is costly and time-intensive, it is common to use an “LLM-as-a-Judge” to rate model completions <cite class="ltx_cite ltx_citemacro_citep">(e.g. Li et al., <a href="#bib.bib34" title="" class="ltx_ref">2023b</a>; Zheng et al., <a href="#bib.bib58" title="" class="ltx_ref">2023</a>)</cite>. Reward models (RMs) can also simulate human preference. An RM scores multiple completions given the same prompt, and the prompt-completion pair with the higher score is deemed preferred. We call this <em id="S3.SS3.p1.1.1" class="ltx_emph ltx_font_italic">RM-as-a-Judge</em>.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.4" class="ltx_p">We assess quantized model outputs using LLM- and RM-as-a-Judge. In the former, an LLM selects a preferred response from a
<math id="S3.SS3.p2.1.m1.1" class="ltx_Math" alttext="&lt;" display="inline"><semantics id="S3.SS3.p2.1.m1.1a"><mo id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml">&lt;</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.1b"><lt id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1"></lt></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.1c">&lt;</annotation></semantics></math><span id="S3.SS3.p2.2.1" class="ltx_text ltx_font_typewriter">instruction, modelA_completion, modelB_completion<math id="S3.SS3.p2.2.1.m1.1" class="ltx_Math" alttext="&gt;" display="inline"><semantics id="S3.SS3.p2.2.1.m1.1a"><mo id="S3.SS3.p2.2.1.m1.1.1" xref="S3.SS3.p2.2.1.m1.1.1.cmml">&gt;</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.1.m1.1b"><gt id="S3.SS3.p2.2.1.m1.1.1.cmml" xref="S3.SS3.p2.2.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.1.m1.1c">&gt;</annotation></semantics></math></span> tuple (see Table <a href="#A1.T2" title="Table A2 ‣ A.1 Prompts for mMMLU and LLM-as-a-Judge ‣ Appendix A Appendix ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A2</span></a>). Following <cite class="ltx_cite ltx_citemacro_cite">Üstün et al. (<a href="#bib.bib60" title="" class="ltx_ref">2024</a>); Aryabumi et al. (<a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite>
we use GPT-4<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span><span id="footnote8.1" class="ltx_text ltx_font_typewriter">gpt-4-turbo (gpt-4-1106-preview)</span>: <a target="_blank" href="https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4</a></span></span></span> as an LLM proxy judge. To minimize bias, we randomize the order of model outputs. For RM-as-a-Judge, a multilingual reward model scores <math id="S3.SS3.p2.3.m2.1" class="ltx_Math" alttext="&lt;" display="inline"><semantics id="S3.SS3.p2.3.m2.1a"><mo id="S3.SS3.p2.3.m2.1.1" xref="S3.SS3.p2.3.m2.1.1.cmml">&lt;</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.3.m2.1b"><lt id="S3.SS3.p2.3.m2.1.1.cmml" xref="S3.SS3.p2.3.m2.1.1"></lt></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.3.m2.1c">&lt;</annotation></semantics></math><span id="S3.SS3.p2.4.2" class="ltx_text ltx_font_typewriter">prompt, completion<math id="S3.SS3.p2.4.2.m1.1" class="ltx_Math" alttext="&gt;" display="inline"><semantics id="S3.SS3.p2.4.2.m1.1a"><mo id="S3.SS3.p2.4.2.m1.1.1" xref="S3.SS3.p2.4.2.m1.1.1.cmml">&gt;</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.4.2.m1.1b"><gt id="S3.SS3.p2.4.2.m1.1.1.cmml" xref="S3.SS3.p2.4.2.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.4.2.m1.1c">&gt;</annotation></semantics></math></span>
pairs for each model output, over which we calculate win-rate. We report win-rates of quantized models versus the FP16 baseline.</p>
</div>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.1" class="ltx_p">We assess the outputs of quantized models over the <span id="S3.SS3.p3.1.1" class="ltx_text ltx_font_italic">Internal Evaluation Suite</span> and <span id="S3.SS3.p3.1.2" class="ltx_text ltx_font_italic">Aya Dolly-200</span> described in Section <a href="#S3.SS2" title="3.2 Human Evaluation ‣ 3 Experiment Set-up ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>. We use the same prompt and completion pairs as in human evaluation, which provides the ability to relate LLM/RM-as-a-Judge performance with human evaluation.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Results</h2>

<figure id="S4.T1" class="ltx_table">
<div id="S4.T1.3" class="ltx_inline-block ltx_transformed_outer" style="width:433.6pt;height:173pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-53.9pt,21.5pt) scale(0.800856449702828,0.800856449702828) ;">
<table id="S4.T1.3.3" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T1.3.3.4.1" class="ltx_tr">
<th id="S4.T1.3.3.4.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="S4.T1.3.3.4.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="S4.T1.3.3.4.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt"><span id="S4.T1.3.3.4.1.3.1" class="ltx_text ltx_font_bold">Avg.</span></th>
<th id="S4.T1.3.3.4.1.4" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="S4.T1.3.3.4.1.5" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="S4.T1.3.3.4.1.6" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="S4.T1.3.3.4.1.7" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="S4.T1.3.3.4.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="4"><span id="S4.T1.3.3.4.1.8.1" class="ltx_text ltx_font_bold">FLORES</span></th>
<th id="S4.T1.3.3.4.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="4"><span id="S4.T1.3.3.4.1.9.1" class="ltx_text ltx_font_bold">Language Confusion</span></th>
</tr>
<tr id="S4.T1.3.3.3" class="ltx_tr">
<th id="S4.T1.3.3.3.4" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S4.T1.3.3.3.5" class="ltx_td ltx_th ltx_th_column ltx_border_r"></th>
<th id="S4.T1.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T1.1.1.1.1.1" class="ltx_text ltx_font_bold">Rel. <math id="S4.T1.1.1.1.1.1.m1.1" class="ltx_math_unparsed" alttext="\%\Delta" display="inline"><semantics id="S4.T1.1.1.1.1.1.m1.1a"><mrow id="S4.T1.1.1.1.1.1.m1.1b"><mo id="S4.T1.1.1.1.1.1.m1.1.1">%</mo><mi mathvariant="normal" id="S4.T1.1.1.1.1.1.m1.1.2">Δ</mi></mrow><annotation encoding="application/x-tex" id="S4.T1.1.1.1.1.1.m1.1c">\%\Delta</annotation></semantics></math></span></th>
<th id="S4.T1.3.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T1.3.3.3.6.1" class="ltx_text ltx_font_bold">mMMLU</span></th>
<th id="S4.T1.3.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T1.3.3.3.7.1" class="ltx_text ltx_font_bold">MGSM</span></th>
<th id="S4.T1.2.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T1.2.2.2.2.1" class="ltx_text ltx_font_bold">En<math id="S4.T1.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T1.2.2.2.2.1.m1.1a"><mo stretchy="false" id="S4.T1.2.2.2.2.1.m1.1.1" xref="S4.T1.2.2.2.2.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.2.2.1.m1.1b"><ci id="S4.T1.2.2.2.2.1.m1.1.1.cmml" xref="S4.T1.2.2.2.2.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.2.2.1.m1.1c">\rightarrow</annotation></semantics></math>L2</span></th>
<th id="S4.T1.3.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T1.3.3.3.3.1" class="ltx_text ltx_font_bold">L2<math id="S4.T1.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T1.3.3.3.3.1.m1.1a"><mo stretchy="false" id="S4.T1.3.3.3.3.1.m1.1.1" xref="S4.T1.3.3.3.3.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T1.3.3.3.3.1.m1.1b"><ci id="S4.T1.3.3.3.3.1.m1.1.1.cmml" xref="S4.T1.3.3.3.3.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.3.3.3.1.m1.1c">\rightarrow</annotation></semantics></math>En</span></th>
<th id="S4.T1.3.3.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T1.3.3.3.8.1" class="ltx_text ltx_font_bold">Monolingual</span></th>
<th id="S4.T1.3.3.3.9" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2"><span id="S4.T1.3.3.3.9.1" class="ltx_text ltx_font_bold">Cross-lingual</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T1.3.3.5.1" class="ltx_tr">
<td id="S4.T1.3.3.5.1.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.3.3.5.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="S4.T1.3.3.5.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T1.3.3.5.1.4" class="ltx_td ltx_align_right ltx_border_t">66.7</td>
<td id="S4.T1.3.3.5.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T1.3.3.5.1.6" class="ltx_td ltx_align_right ltx_border_t">70.6</td>
<td id="S4.T1.3.3.5.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T1.3.3.5.1.8" class="ltx_td ltx_align_right ltx_border_t">37.7</td>
<td id="S4.T1.3.3.5.1.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T1.3.3.5.1.10" class="ltx_td ltx_align_right ltx_border_t">39.6</td>
<td id="S4.T1.3.3.5.1.11" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T1.3.3.5.1.12" class="ltx_td ltx_align_right ltx_border_t">99.2</td>
<td id="S4.T1.3.3.5.1.13" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T1.3.3.5.1.14" class="ltx_td ltx_align_right ltx_border_t">91.5</td>
<td id="S4.T1.3.3.5.1.15" class="ltx_td ltx_align_center ltx_border_t">-</td>
</tr>
<tr id="S4.T1.3.3.6.2" class="ltx_tr">
<td id="S4.T1.3.3.6.2.1" class="ltx_td"></td>
<td id="S4.T1.3.3.6.2.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="S4.T1.3.3.6.2.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#FDF9F9;"><span id="S4.T1.3.3.6.2.3.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="S4.T1.3.3.6.2.4" class="ltx_td ltx_align_right">66.7</td>
<td id="S4.T1.3.3.6.2.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FFFFFF;"><span id="S4.T1.3.3.6.2.5.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="S4.T1.3.3.6.2.6" class="ltx_td ltx_align_right">69.9</td>
<td id="S4.T1.3.3.6.2.7" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FAE6E4;"><span id="S4.T1.3.3.6.2.7.1" class="ltx_text" style="background-color:#FAE6E4;">-1.0%</span></td>
<td id="S4.T1.3.3.6.2.8" class="ltx_td ltx_align_right">37.7</td>
<td id="S4.T1.3.3.6.2.9" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FFFFFF;"><span id="S4.T1.3.3.6.2.9.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="S4.T1.3.3.6.2.10" class="ltx_td ltx_align_right">39.6</td>
<td id="S4.T1.3.3.6.2.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFEFE;"><span id="S4.T1.3.3.6.2.11.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="S4.T1.3.3.6.2.12" class="ltx_td ltx_align_right">99.2</td>
<td id="S4.T1.3.3.6.2.13" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FFFFFF;"><span id="S4.T1.3.3.6.2.13.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="S4.T1.3.3.6.2.14" class="ltx_td ltx_align_right">91.2</td>
<td id="S4.T1.3.3.6.2.15" class="ltx_td ltx_align_right" style="background-color:#FDF7F7;"><span id="S4.T1.3.3.6.2.15.1" class="ltx_text" style="background-color:#FDF7F7;">-0.3%</span></td>
</tr>
<tr id="S4.T1.3.3.7.3" class="ltx_tr">
<td id="S4.T1.3.3.7.3.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.3.3.7.3.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8A8-sq</td>
<td id="S4.T1.3.3.7.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#FCF1F0;"><span id="S4.T1.3.3.7.3.3.1" class="ltx_text" style="background-color:#FCF1F0;">-0.5%</span></td>
<td id="S4.T1.3.3.7.3.4" class="ltx_td ltx_align_right ltx_border_t">66.3</td>
<td id="S4.T1.3.3.7.3.5" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FCF1F0;"><span id="S4.T1.3.3.7.3.5.1" class="ltx_text" style="background-color:#FCF1F0;">-0.5%</span></td>
<td id="S4.T1.3.3.7.3.6" class="ltx_td ltx_align_right ltx_border_t">69.5</td>
<td id="S4.T1.3.3.7.3.7" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F6D4D1;"><span id="S4.T1.3.3.7.3.7.1" class="ltx_text" style="background-color:#F6D4D1;">-1.6%</span></td>
<td id="S4.T1.3.3.7.3.8" class="ltx_td ltx_align_right ltx_border_t">37.8</td>
<td id="S4.T1.3.3.7.3.9" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FBFEFC;"><span id="S4.T1.3.3.7.3.9.1" class="ltx_text" style="background-color:#FBFEFC;">0.2%</span></td>
<td id="S4.T1.3.3.7.3.10" class="ltx_td ltx_align_right ltx_border_t">39.1</td>
<td id="S4.T1.3.3.7.3.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F8DDDB;"><span id="S4.T1.3.3.7.3.11.1" class="ltx_text" style="background-color:#F8DDDB;">-1.3%</span></td>
<td id="S4.T1.3.3.7.3.12" class="ltx_td ltx_align_right ltx_border_t">99.2</td>
<td id="S4.T1.3.3.7.3.13" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FFFFFF;"><span id="S4.T1.3.3.7.3.13.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="S4.T1.3.3.7.3.14" class="ltx_td ltx_align_right ltx_border_t">91.5</td>
<td id="S4.T1.3.3.7.3.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFFFF;"><span id="S4.T1.3.3.7.3.15.1" class="ltx_text" style="background-color:#FEFFFF;">0.1%</span></td>
</tr>
<tr id="S4.T1.3.3.8.4" class="ltx_tr">
<td id="S4.T1.3.3.8.4.1" class="ltx_td"></td>
<td id="S4.T1.3.3.8.4.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="S4.T1.3.3.8.4.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#FAE9E7;"><span id="S4.T1.3.3.8.4.3.1" class="ltx_text" style="background-color:#FAE9E7;">-0.8%</span></td>
<td id="S4.T1.3.3.8.4.4" class="ltx_td ltx_align_right">65.6</td>
<td id="S4.T1.3.3.8.4.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F6D3D0;"><span id="S4.T1.3.3.8.4.5.1" class="ltx_text" style="background-color:#F6D3D0;">-1.7%</span></td>
<td id="S4.T1.3.3.8.4.6" class="ltx_td ltx_align_right">69.8</td>
<td id="S4.T1.3.3.8.4.7" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F9E2E0;"><span id="S4.T1.3.3.8.4.7.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="S4.T1.3.3.8.4.8" class="ltx_td ltx_align_right">37.7</td>
<td id="S4.T1.3.3.8.4.9" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FFFFFF;"><span id="S4.T1.3.3.8.4.9.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="S4.T1.3.3.8.4.10" class="ltx_td ltx_align_right">39.1</td>
<td id="S4.T1.3.3.8.4.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F8DEDC;"><span id="S4.T1.3.3.8.4.11.1" class="ltx_text" style="background-color:#F8DEDC;">-1.2%</span></td>
<td id="S4.T1.3.3.8.4.12" class="ltx_td ltx_align_right">99.4</td>
<td id="S4.T1.3.3.8.4.13" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FBFEFC;"><span id="S4.T1.3.3.8.4.13.1" class="ltx_text" style="background-color:#FBFEFC;">0.2%</span></td>
<td id="S4.T1.3.3.8.4.14" class="ltx_td ltx_align_right">90.4</td>
<td id="S4.T1.3.3.8.4.15" class="ltx_td ltx_align_right" style="background-color:#F8DFDD;"><span id="S4.T1.3.3.8.4.15.1" class="ltx_text" style="background-color:#F8DFDD;">-1.2%</span></td>
</tr>
<tr id="S4.T1.3.3.9.5" class="ltx_tr">
<td id="S4.T1.3.3.9.5.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.3.3.9.5.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W4-g</td>
<td id="S4.T1.3.3.9.5.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#FAE6E5;"><span id="S4.T1.3.3.9.5.3.1" class="ltx_text" style="background-color:#FAE6E5;">-0.9%</span></td>
<td id="S4.T1.3.3.9.5.4" class="ltx_td ltx_align_right ltx_border_t">65.7</td>
<td id="S4.T1.3.3.9.5.5" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F7D9D6;"><span id="S4.T1.3.3.9.5.5.1" class="ltx_text" style="background-color:#F7D9D6;">-1.4%</span></td>
<td id="S4.T1.3.3.9.5.6" class="ltx_td ltx_align_right ltx_border_t">68.6</td>
<td id="S4.T1.3.3.9.5.7" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F0B3AD;"><span id="S4.T1.3.3.9.5.7.1" class="ltx_text" style="background-color:#F0B3AD;">-2.9%</span></td>
<td id="S4.T1.3.3.9.5.8" class="ltx_td ltx_align_right ltx_border_t">37.8</td>
<td id="S4.T1.3.3.9.5.9" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F6FCF9;"><span id="S4.T1.3.3.9.5.9.1" class="ltx_text" style="background-color:#F6FCF9;">0.4%</span></td>
<td id="S4.T1.3.3.9.5.10" class="ltx_td ltx_align_right ltx_border_t">39.4</td>
<td id="S4.T1.3.3.9.5.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FCF2F1;"><span id="S4.T1.3.3.9.5.11.1" class="ltx_text" style="background-color:#FCF2F1;">-0.5%</span></td>
<td id="S4.T1.3.3.9.5.12" class="ltx_td ltx_align_right ltx_border_t">99.2</td>
<td id="S4.T1.3.3.9.5.13" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FFFFFF;"><span id="S4.T1.3.3.9.5.13.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="S4.T1.3.3.9.5.14" class="ltx_td ltx_align_right ltx_border_t">90.5</td>
<td id="S4.T1.3.3.9.5.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F9E2E0;"><span id="S4.T1.3.3.9.5.15.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
</tr>
<tr id="S4.T1.3.3.10.6" class="ltx_tr">
<td id="S4.T1.3.3.10.6.1" class="ltx_td ltx_align_left"><span id="S4.T1.3.3.10.6.1.1" class="ltx_text">103B</span></td>
<td id="S4.T1.3.3.10.6.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="S4.T1.3.3.10.6.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#F2BEB9;"><span id="S4.T1.3.3.10.6.3.1" class="ltx_text" style="background-color:#F2BEB9;">-2.5%</span></td>
<td id="S4.T1.3.3.10.6.4" class="ltx_td ltx_align_right">63.8</td>
<td id="S4.T1.3.3.10.6.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E98D85;"><span id="S4.T1.3.3.10.6.5.1" class="ltx_text" style="background-color:#E98D85;">-4.3%</span></td>
<td id="S4.T1.3.3.10.6.6" class="ltx_td ltx_align_right">64.4</td>
<td id="S4.T1.3.3.10.6.7" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E67C73;"><span id="S4.T1.3.3.10.6.7.1" class="ltx_text" style="background-color:#E67C73;">-8.8%</span></td>
<td id="S4.T1.3.3.10.6.8" class="ltx_td ltx_align_right">37.1</td>
<td id="S4.T1.3.3.10.6.9" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F6D3D0;"><span id="S4.T1.3.3.10.6.9.1" class="ltx_text" style="background-color:#F6D3D0;">-1.6%</span></td>
<td id="S4.T1.3.3.10.6.10" class="ltx_td ltx_align_right">39.0</td>
<td id="S4.T1.3.3.10.6.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F7D5D2;"><span id="S4.T1.3.3.10.6.11.1" class="ltx_text" style="background-color:#F7D5D2;">-1.6%</span></td>
<td id="S4.T1.3.3.10.6.12" class="ltx_td ltx_align_right">99.3</td>
<td id="S4.T1.3.3.10.6.13" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FCFEFD;"><span id="S4.T1.3.3.10.6.13.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="S4.T1.3.3.10.6.14" class="ltx_td ltx_align_right">92.8</td>
<td id="S4.T1.3.3.10.6.15" class="ltx_td ltx_align_right" style="background-color:#DBF1E6;"><span id="S4.T1.3.3.10.6.15.1" class="ltx_text" style="background-color:#DBF1E6;">1.4%</span></td>
</tr>
<tr id="S4.T1.3.3.11.7" class="ltx_tr">
<td id="S4.T1.3.3.11.7.1" class="ltx_td ltx_border_tt"></td>
<td id="S4.T1.3.3.11.7.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt">FP16</td>
<td id="S4.T1.3.3.11.7.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T1.3.3.11.7.4" class="ltx_td ltx_align_right ltx_border_tt">59.4</td>
<td id="S4.T1.3.3.11.7.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T1.3.3.11.7.6" class="ltx_td ltx_align_right ltx_border_tt">49.8</td>
<td id="S4.T1.3.3.11.7.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T1.3.3.11.7.8" class="ltx_td ltx_align_right ltx_border_tt">32.4</td>
<td id="S4.T1.3.3.11.7.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T1.3.3.11.7.10" class="ltx_td ltx_align_right ltx_border_tt">35.5</td>
<td id="S4.T1.3.3.11.7.11" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T1.3.3.11.7.12" class="ltx_td ltx_align_right ltx_border_tt">98.7</td>
<td id="S4.T1.3.3.11.7.13" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T1.3.3.11.7.14" class="ltx_td ltx_align_right ltx_border_tt">66.5</td>
<td id="S4.T1.3.3.11.7.15" class="ltx_td ltx_align_center ltx_border_tt">-</td>
</tr>
<tr id="S4.T1.3.3.12.8" class="ltx_tr">
<td id="S4.T1.3.3.12.8.1" class="ltx_td"></td>
<td id="S4.T1.3.3.12.8.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="S4.T1.3.3.12.8.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#FDF9F9;"><span id="S4.T1.3.3.12.8.3.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="S4.T1.3.3.12.8.4" class="ltx_td ltx_align_right">59.3</td>
<td id="S4.T1.3.3.12.8.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFDFD;"><span id="S4.T1.3.3.12.8.5.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="S4.T1.3.3.12.8.6" class="ltx_td ltx_align_right">49.4</td>
<td id="S4.T1.3.3.12.8.7" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FBECEB;"><span id="S4.T1.3.3.12.8.7.1" class="ltx_text" style="background-color:#FBECEB;">-0.7%</span></td>
<td id="S4.T1.3.3.12.8.8" class="ltx_td ltx_align_right">32.3</td>
<td id="S4.T1.3.3.12.8.9" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFBFA;"><span id="S4.T1.3.3.12.8.9.1" class="ltx_text" style="background-color:#FEFBFA;">-0.2%</span></td>
<td id="S4.T1.3.3.12.8.10" class="ltx_td ltx_align_right">35.4</td>
<td id="S4.T1.3.3.12.8.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEF9F9;"><span id="S4.T1.3.3.12.8.11.1" class="ltx_text" style="background-color:#FEF9F9;">-0.2%</span></td>
<td id="S4.T1.3.3.12.8.12" class="ltx_td ltx_align_right">98.8</td>
<td id="S4.T1.3.3.12.8.13" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FDFFFE;"><span id="S4.T1.3.3.12.8.13.1" class="ltx_text" style="background-color:#FDFFFE;">0.1%</span></td>
<td id="S4.T1.3.3.12.8.14" class="ltx_td ltx_align_right">66.3</td>
<td id="S4.T1.3.3.12.8.15" class="ltx_td ltx_align_right" style="background-color:#FDF9F9;"><span id="S4.T1.3.3.12.8.15.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
</tr>
<tr id="S4.T1.3.3.13.9" class="ltx_tr">
<td id="S4.T1.3.3.13.9.1" class="ltx_td"></td>
<td id="S4.T1.3.3.13.9.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="S4.T1.3.3.13.9.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#FCFEFD;"><span id="S4.T1.3.3.13.9.3.1" class="ltx_text" style="background-color:#FCFEFD;">0.2%</span></td>
<td id="S4.T1.3.3.13.9.4" class="ltx_td ltx_align_right">59.3</td>
<td id="S4.T1.3.3.13.9.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFAFA;"><span id="S4.T1.3.3.13.9.5.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="S4.T1.3.3.13.9.6" class="ltx_td ltx_align_right">47.1</td>
<td id="S4.T1.3.3.13.9.7" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E67C73;"><span id="S4.T1.3.3.13.9.7.1" class="ltx_text" style="background-color:#E67C73;">-5.5%</span></td>
<td id="S4.T1.3.3.13.9.8" class="ltx_td ltx_align_right">32.9</td>
<td id="S4.T1.3.3.13.9.9" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#D6EFE3;"><span id="S4.T1.3.3.13.9.9.1" class="ltx_text" style="background-color:#D6EFE3;">1.6%</span></td>
<td id="S4.T1.3.3.13.9.10" class="ltx_td ltx_align_right">35.8</td>
<td id="S4.T1.3.3.13.9.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E8F6EF;"><span id="S4.T1.3.3.13.9.11.1" class="ltx_text" style="background-color:#E8F6EF;">0.9%</span></td>
<td id="S4.T1.3.3.13.9.12" class="ltx_td ltx_align_right">99.0</td>
<td id="S4.T1.3.3.13.9.13" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F7FCF9;"><span id="S4.T1.3.3.13.9.13.1" class="ltx_text" style="background-color:#F7FCF9;">0.3%</span></td>
<td id="S4.T1.3.3.13.9.14" class="ltx_td ltx_align_right">68.9</td>
<td id="S4.T1.3.3.13.9.15" class="ltx_td ltx_align_right" style="background-color:#A0D9BD;"><span id="S4.T1.3.3.13.9.15.1" class="ltx_text" style="background-color:#A0D9BD;">3.7%</span></td>
</tr>
<tr id="S4.T1.3.3.14.10" class="ltx_tr">
<td id="S4.T1.3.3.14.10.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T1.3.3.14.10.1.1" class="ltx_text">35B</span></td>
<td id="S4.T1.3.3.14.10.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4-g</td>
<td id="S4.T1.3.3.14.10.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="background-color:#F0B4AF;"><span id="S4.T1.3.3.14.10.3.1" class="ltx_text" style="background-color:#F0B4AF;">-2.8%</span></td>
<td id="S4.T1.3.3.14.10.4" class="ltx_td ltx_align_right ltx_border_bb">58.2</td>
<td id="S4.T1.3.3.14.10.5" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F5CBC7;"><span id="S4.T1.3.3.14.10.5.1" class="ltx_text" style="background-color:#F5CBC7;">-2.0%</span></td>
<td id="S4.T1.3.3.14.10.6" class="ltx_td ltx_align_right ltx_border_bb">43.3</td>
<td id="S4.T1.3.3.14.10.7" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#E67C73;"><span id="S4.T1.3.3.14.10.7.1" class="ltx_text" style="background-color:#E67C73;">-13.1%</span></td>
<td id="S4.T1.3.3.14.10.8" class="ltx_td ltx_align_right ltx_border_bb">31.7</td>
<td id="S4.T1.3.3.14.10.9" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F5CDC9;"><span id="S4.T1.3.3.14.10.9.1" class="ltx_text" style="background-color:#F5CDC9;">-1.9%</span></td>
<td id="S4.T1.3.3.14.10.10" class="ltx_td ltx_align_right ltx_border_bb">35.3</td>
<td id="S4.T1.3.3.14.10.11" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#FBEDEB;"><span id="S4.T1.3.3.14.10.11.1" class="ltx_text" style="background-color:#FBEDEB;">-0.7%</span></td>
<td id="S4.T1.3.3.14.10.12" class="ltx_td ltx_align_right ltx_border_bb">98.3</td>
<td id="S4.T1.3.3.14.10.13" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#FCF3F3;"><span id="S4.T1.3.3.14.10.13.1" class="ltx_text" style="background-color:#FCF3F3;">-0.4%</span></td>
<td id="S4.T1.3.3.14.10.14" class="ltx_td ltx_align_right ltx_border_bb">67.1</td>
<td id="S4.T1.3.3.14.10.15" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E7F5EE;"><span id="S4.T1.3.3.14.10.15.1" class="ltx_text" style="background-color:#E7F5EE;">1.0%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 1: </span><span id="S4.T1.11.2" class="ltx_text ltx_font_bold">Per-dataset average performance across non-English languages for 103B and 35B Command models at varying levels of quantization.</span> %<math id="S4.T1.7.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S4.T1.7.m1.1b"><mi mathvariant="normal" id="S4.T1.7.m1.1.1" xref="S4.T1.7.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S4.T1.7.m1.1c"><ci id="S4.T1.7.m1.1.1.cmml" xref="S4.T1.7.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.7.m1.1d">\Delta</annotation></semantics></math> the relative performance vs. FP16 [ex., for MGSM at W4-g on the 35B: <math id="S4.T1.8.m2.1" class="ltx_Math" alttext="\frac{43.3-49.8}{49.8}*100=-13.1\%" display="inline"><semantics id="S4.T1.8.m2.1b"><mrow id="S4.T1.8.m2.1.1" xref="S4.T1.8.m2.1.1.cmml"><mrow id="S4.T1.8.m2.1.1.2" xref="S4.T1.8.m2.1.1.2.cmml"><mfrac id="S4.T1.8.m2.1.1.2.2" xref="S4.T1.8.m2.1.1.2.2.cmml"><mrow id="S4.T1.8.m2.1.1.2.2.2" xref="S4.T1.8.m2.1.1.2.2.2.cmml"><mn id="S4.T1.8.m2.1.1.2.2.2.2" xref="S4.T1.8.m2.1.1.2.2.2.2.cmml">43.3</mn><mo id="S4.T1.8.m2.1.1.2.2.2.1" xref="S4.T1.8.m2.1.1.2.2.2.1.cmml">−</mo><mn id="S4.T1.8.m2.1.1.2.2.2.3" xref="S4.T1.8.m2.1.1.2.2.2.3.cmml">49.8</mn></mrow><mn id="S4.T1.8.m2.1.1.2.2.3" xref="S4.T1.8.m2.1.1.2.2.3.cmml">49.8</mn></mfrac><mo lspace="0.222em" rspace="0.222em" id="S4.T1.8.m2.1.1.2.1" xref="S4.T1.8.m2.1.1.2.1.cmml">∗</mo><mn id="S4.T1.8.m2.1.1.2.3" xref="S4.T1.8.m2.1.1.2.3.cmml">100</mn></mrow><mo id="S4.T1.8.m2.1.1.1" xref="S4.T1.8.m2.1.1.1.cmml">=</mo><mrow id="S4.T1.8.m2.1.1.3" xref="S4.T1.8.m2.1.1.3.cmml"><mo id="S4.T1.8.m2.1.1.3b" xref="S4.T1.8.m2.1.1.3.cmml">−</mo><mrow id="S4.T1.8.m2.1.1.3.2" xref="S4.T1.8.m2.1.1.3.2.cmml"><mn id="S4.T1.8.m2.1.1.3.2.2" xref="S4.T1.8.m2.1.1.3.2.2.cmml">13.1</mn><mo id="S4.T1.8.m2.1.1.3.2.1" xref="S4.T1.8.m2.1.1.3.2.1.cmml">%</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.8.m2.1c"><apply id="S4.T1.8.m2.1.1.cmml" xref="S4.T1.8.m2.1.1"><eq id="S4.T1.8.m2.1.1.1.cmml" xref="S4.T1.8.m2.1.1.1"></eq><apply id="S4.T1.8.m2.1.1.2.cmml" xref="S4.T1.8.m2.1.1.2"><times id="S4.T1.8.m2.1.1.2.1.cmml" xref="S4.T1.8.m2.1.1.2.1"></times><apply id="S4.T1.8.m2.1.1.2.2.cmml" xref="S4.T1.8.m2.1.1.2.2"><divide id="S4.T1.8.m2.1.1.2.2.1.cmml" xref="S4.T1.8.m2.1.1.2.2"></divide><apply id="S4.T1.8.m2.1.1.2.2.2.cmml" xref="S4.T1.8.m2.1.1.2.2.2"><minus id="S4.T1.8.m2.1.1.2.2.2.1.cmml" xref="S4.T1.8.m2.1.1.2.2.2.1"></minus><cn type="float" id="S4.T1.8.m2.1.1.2.2.2.2.cmml" xref="S4.T1.8.m2.1.1.2.2.2.2">43.3</cn><cn type="float" id="S4.T1.8.m2.1.1.2.2.2.3.cmml" xref="S4.T1.8.m2.1.1.2.2.2.3">49.8</cn></apply><cn type="float" id="S4.T1.8.m2.1.1.2.2.3.cmml" xref="S4.T1.8.m2.1.1.2.2.3">49.8</cn></apply><cn type="integer" id="S4.T1.8.m2.1.1.2.3.cmml" xref="S4.T1.8.m2.1.1.2.3">100</cn></apply><apply id="S4.T1.8.m2.1.1.3.cmml" xref="S4.T1.8.m2.1.1.3"><minus id="S4.T1.8.m2.1.1.3.1.cmml" xref="S4.T1.8.m2.1.1.3"></minus><apply id="S4.T1.8.m2.1.1.3.2.cmml" xref="S4.T1.8.m2.1.1.3.2"><csymbol cd="latexml" id="S4.T1.8.m2.1.1.3.2.1.cmml" xref="S4.T1.8.m2.1.1.3.2.1">percent</csymbol><cn type="float" id="S4.T1.8.m2.1.1.3.2.2.cmml" xref="S4.T1.8.m2.1.1.3.2.2">13.1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.8.m2.1d">\frac{43.3-49.8}{49.8}*100=-13.1\%</annotation></semantics></math>.] Languages: ar, de, es, fr, it, ja, ko, pt, zh; except MGSM: de, es, fr, ja, zh.
Any discrepancy is due to rounding: raw scores and <span id="S4.T1.9.1" class="ltx_text ltx_font_bold">%<math id="S4.T1.9.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S4.T1.9.1.m1.1b"><mi mathvariant="normal" id="S4.T1.9.1.m1.1.1" xref="S4.T1.9.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S4.T1.9.1.m1.1c"><ci id="S4.T1.9.1.m1.1.1.cmml" xref="S4.T1.9.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.9.1.m1.1d">\Delta</annotation></semantics></math></span> were calculated at full precision.
</figcaption>
</figure>
<figure id="S4.T2" class="ltx_table">
<div id="S4.T2.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:115.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-52.5pt,14.0pt) scale(0.805069295199337,0.805069295199337) ;">
<table id="S4.T2.3.3" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.3.3.4.1" class="ltx_tr">
<th id="S4.T2.3.3.4.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="S4.T2.3.3.4.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="S4.T2.3.3.4.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt"><span id="S4.T2.3.3.4.1.3.1" class="ltx_text ltx_font_bold">Avg.</span></th>
<th id="S4.T2.3.3.4.1.4" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="S4.T2.3.3.4.1.5" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="S4.T2.3.3.4.1.6" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="S4.T2.3.3.4.1.7" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="S4.T2.3.3.4.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="4"><span id="S4.T2.3.3.4.1.8.1" class="ltx_text ltx_font_bold">FLORES</span></th>
<th id="S4.T2.3.3.4.1.9" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="S4.T2.3.3.4.1.10" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="S4.T2.3.3.4.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S4.T2.3.3.4.1.11.1" class="ltx_text ltx_font_bold">Unseen</span></th>
</tr>
<tr id="S4.T2.3.3.3" class="ltx_tr">
<th id="S4.T2.3.3.3.4" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S4.T2.3.3.3.5" class="ltx_td ltx_th ltx_th_column ltx_border_r"></th>
<th id="S4.T2.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T2.1.1.1.1.1" class="ltx_text ltx_font_bold">Rel. <math id="S4.T2.1.1.1.1.1.m1.1" class="ltx_math_unparsed" alttext="\%\Delta" display="inline"><semantics id="S4.T2.1.1.1.1.1.m1.1a"><mrow id="S4.T2.1.1.1.1.1.m1.1b"><mo id="S4.T2.1.1.1.1.1.m1.1.1">%</mo><mi mathvariant="normal" id="S4.T2.1.1.1.1.1.m1.1.2">Δ</mi></mrow><annotation encoding="application/x-tex" id="S4.T2.1.1.1.1.1.m1.1c">\%\Delta</annotation></semantics></math></span></th>
<th id="S4.T2.3.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T2.3.3.3.6.1" class="ltx_text ltx_font_bold">mMMLU</span></th>
<th id="S4.T2.3.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T2.3.3.3.7.1" class="ltx_text ltx_font_bold">MGSM</span></th>
<th id="S4.T2.2.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T2.2.2.2.2.1" class="ltx_text ltx_font_bold">En<math id="S4.T2.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T2.2.2.2.2.1.m1.1a"><mo stretchy="false" id="S4.T2.2.2.2.2.1.m1.1.1" xref="S4.T2.2.2.2.2.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.2.2.1.m1.1b"><ci id="S4.T2.2.2.2.2.1.m1.1.1.cmml" xref="S4.T2.2.2.2.2.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.2.2.1.m1.1c">\rightarrow</annotation></semantics></math>L2</span></th>
<th id="S4.T2.3.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T2.3.3.3.3.1" class="ltx_text ltx_font_bold">L2<math id="S4.T2.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T2.3.3.3.3.1.m1.1a"><mo stretchy="false" id="S4.T2.3.3.3.3.1.m1.1.1" xref="S4.T2.3.3.3.3.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T2.3.3.3.3.1.m1.1b"><ci id="S4.T2.3.3.3.3.1.m1.1.1.cmml" xref="S4.T2.3.3.3.3.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.3.3.3.3.1.m1.1c">\rightarrow</annotation></semantics></math>En</span></th>
<th id="S4.T2.3.3.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r" colspan="2"><span id="S4.T2.3.3.3.8.1" class="ltx_text ltx_font_bold">Belebele</span></th>
<th id="S4.T2.3.3.3.9" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2"><span id="S4.T2.3.3.3.9.1" class="ltx_text ltx_font_bold">Tasks</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.3.3.5.1" class="ltx_tr">
<td id="S4.T2.3.3.5.1.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T2.3.3.5.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="S4.T2.3.3.5.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T2.3.3.5.1.4" class="ltx_td ltx_align_right ltx_border_t">58.2</td>
<td id="S4.T2.3.3.5.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T2.3.3.5.1.6" class="ltx_td ltx_align_right ltx_border_t">51.2</td>
<td id="S4.T2.3.3.5.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T2.3.3.5.1.8" class="ltx_td ltx_align_right ltx_border_t">37.8</td>
<td id="S4.T2.3.3.5.1.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T2.3.3.5.1.10" class="ltx_td ltx_align_right ltx_border_t">42.9</td>
<td id="S4.T2.3.3.5.1.11" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T2.3.3.5.1.12" class="ltx_td ltx_align_right ltx_border_t">77.6</td>
<td id="S4.T2.3.3.5.1.13" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">-</td>
<td id="S4.T2.3.3.5.1.14" class="ltx_td ltx_align_right ltx_border_t">70.8</td>
<td id="S4.T2.3.3.5.1.15" class="ltx_td ltx_align_center ltx_border_t">-</td>
</tr>
<tr id="S4.T2.3.3.6.2" class="ltx_tr">
<td id="S4.T2.3.3.6.2.1" class="ltx_td"></td>
<td id="S4.T2.3.3.6.2.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="S4.T2.3.3.6.2.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#FEFFFE;"><span id="S4.T2.3.3.6.2.3.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="S4.T2.3.3.6.2.4" class="ltx_td ltx_align_right">57.9</td>
<td id="S4.T2.3.3.6.2.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FCF2F1;"><span id="S4.T2.3.3.6.2.5.1" class="ltx_text" style="background-color:#FCF2F1;">-0.5%</span></td>
<td id="S4.T2.3.3.6.2.6" class="ltx_td ltx_align_right">52.1</td>
<td id="S4.T2.3.3.6.2.7" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#D0ECDF;"><span id="S4.T2.3.3.6.2.7.1" class="ltx_text" style="background-color:#D0ECDF;">1.8%</span></td>
<td id="S4.T2.3.3.6.2.8" class="ltx_td ltx_align_right">37.9</td>
<td id="S4.T2.3.3.6.2.9" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F9FDFB;"><span id="S4.T2.3.3.6.2.9.1" class="ltx_text" style="background-color:#F9FDFB;">0.3%</span></td>
<td id="S4.T2.3.3.6.2.10" class="ltx_td ltx_align_right">43.0</td>
<td id="S4.T2.3.3.6.2.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFFFE;"><span id="S4.T2.3.3.6.2.11.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="S4.T2.3.3.6.2.12" class="ltx_td ltx_align_right">77.1</td>
<td id="S4.T2.3.3.6.2.13" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FBEEED;"><span id="S4.T2.3.3.6.2.13.1" class="ltx_text" style="background-color:#FBEEED;">-0.6%</span></td>
<td id="S4.T2.3.3.6.2.14" class="ltx_td ltx_align_right">70.6</td>
<td id="S4.T2.3.3.6.2.15" class="ltx_td ltx_align_right" style="background-color:#FDF9F8;"><span id="S4.T2.3.3.6.2.15.1" class="ltx_text" style="background-color:#FDF9F8;">-0.2%</span></td>
</tr>
<tr id="S4.T2.3.3.7.3" class="ltx_tr">
<td id="S4.T2.3.3.7.3.1" class="ltx_td ltx_align_center"><span id="S4.T2.3.3.7.3.1.1" class="ltx_text">Aya 35B</span></td>
<td id="S4.T2.3.3.7.3.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="S4.T2.3.3.7.3.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#F1B7B3;"><span id="S4.T2.3.3.7.3.3.1" class="ltx_text" style="background-color:#F1B7B3;">-2.9%</span></td>
<td id="S4.T2.3.3.7.3.4" class="ltx_td ltx_align_right">56.6</td>
<td id="S4.T2.3.3.7.3.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F1B7B3;"><span id="S4.T2.3.3.7.3.5.1" class="ltx_text" style="background-color:#F1B7B3;">-2.7%</span></td>
<td id="S4.T2.3.3.7.3.6" class="ltx_td ltx_align_right">48.1</td>
<td id="S4.T2.3.3.7.3.7" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E67C73;"><span id="S4.T2.3.3.7.3.7.1" class="ltx_text" style="background-color:#E67C73;">-6.0%</span></td>
<td id="S4.T2.3.3.7.3.8" class="ltx_td ltx_align_right">37.2</td>
<td id="S4.T2.3.3.7.3.9" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F8DBD9;"><span id="S4.T2.3.3.7.3.9.1" class="ltx_text" style="background-color:#F8DBD9;">-1.4%</span></td>
<td id="S4.T2.3.3.7.3.10" class="ltx_td ltx_align_right">42.4</td>
<td id="S4.T2.3.3.7.3.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F8DFDC;"><span id="S4.T2.3.3.7.3.11.1" class="ltx_text" style="background-color:#F8DFDC;">-1.2%</span></td>
<td id="S4.T2.3.3.7.3.12" class="ltx_td ltx_align_right">73.0</td>
<td id="S4.T2.3.3.7.3.13" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E67C73;"><span id="S4.T2.3.3.7.3.13.1" class="ltx_text" style="background-color:#E67C73;">-5.9%</span></td>
<td id="S4.T2.3.3.7.3.14" class="ltx_td ltx_align_right">70.5</td>
<td id="S4.T2.3.3.7.3.15" class="ltx_td ltx_align_right" style="background-color:#FDF6F5;"><span id="S4.T2.3.3.7.3.15.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
</tr>
<tr id="S4.T2.3.3.8.4" class="ltx_tr">
<td id="S4.T2.3.3.8.4.1" class="ltx_td ltx_border_tt"></td>
<td id="S4.T2.3.3.8.4.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt">FP16</td>
<td id="S4.T2.3.3.8.4.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T2.3.3.8.4.4" class="ltx_td ltx_align_right ltx_border_tt">48.2</td>
<td id="S4.T2.3.3.8.4.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T2.3.3.8.4.6" class="ltx_td ltx_align_right ltx_border_tt">34.7</td>
<td id="S4.T2.3.3.8.4.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T2.3.3.8.4.8" class="ltx_td ltx_align_right ltx_border_tt">34.8</td>
<td id="S4.T2.3.3.8.4.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T2.3.3.8.4.10" class="ltx_td ltx_align_right ltx_border_tt">39.5</td>
<td id="S4.T2.3.3.8.4.11" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T2.3.3.8.4.12" class="ltx_td ltx_align_right ltx_border_tt">64.8</td>
<td id="S4.T2.3.3.8.4.13" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt">-</td>
<td id="S4.T2.3.3.8.4.14" class="ltx_td ltx_align_right ltx_border_tt">67.6</td>
<td id="S4.T2.3.3.8.4.15" class="ltx_td ltx_align_center ltx_border_tt">-</td>
</tr>
<tr id="S4.T2.3.3.9.5" class="ltx_tr">
<td id="S4.T2.3.3.9.5.1" class="ltx_td"></td>
<td id="S4.T2.3.3.9.5.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="S4.T2.3.3.9.5.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#F7FCFA;"><span id="S4.T2.3.3.9.5.3.1" class="ltx_text" style="background-color:#F7FCFA;">0.3%</span></td>
<td id="S4.T2.3.3.9.5.4" class="ltx_td ltx_align_right">47.8</td>
<td id="S4.T2.3.3.9.5.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FAE7E6;"><span id="S4.T2.3.3.9.5.5.1" class="ltx_text" style="background-color:#FAE7E6;">-0.9%</span></td>
<td id="S4.T2.3.3.9.5.6" class="ltx_td ltx_align_right">35.4</td>
<td id="S4.T2.3.3.9.5.7" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#C9E9D9;"><span id="S4.T2.3.3.9.5.7.1" class="ltx_text" style="background-color:#C9E9D9;">2.1%</span></td>
<td id="S4.T2.3.3.9.5.8" class="ltx_td ltx_align_right">34.8</td>
<td id="S4.T2.3.3.9.5.9" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FAFDFC;"><span id="S4.T2.3.3.9.5.9.1" class="ltx_text" style="background-color:#FAFDFC;">0.2%</span></td>
<td id="S4.T2.3.3.9.5.10" class="ltx_td ltx_align_right">39.7</td>
<td id="S4.T2.3.3.9.5.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F4FBF7;"><span id="S4.T2.3.3.9.5.11.1" class="ltx_text" style="background-color:#F4FBF7;">0.5%</span></td>
<td id="S4.T2.3.3.9.5.12" class="ltx_td ltx_align_right">64.6</td>
<td id="S4.T2.3.3.9.5.13" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FDF6F6;"><span id="S4.T2.3.3.9.5.13.1" class="ltx_text" style="background-color:#FDF6F6;">-0.3%</span></td>
<td id="S4.T2.3.3.9.5.14" class="ltx_td ltx_align_right">67.6</td>
<td id="S4.T2.3.3.9.5.15" class="ltx_td ltx_align_right" style="background-color:#FEFFFE;"><span id="S4.T2.3.3.9.5.15.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
</tr>
<tr id="S4.T2.3.3.10.6" class="ltx_tr">
<td id="S4.T2.3.3.10.6.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T2.3.3.10.6.1.1" class="ltx_text">Aya 8B</span></td>
<td id="S4.T2.3.3.10.6.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4</td>
<td id="S4.T2.3.3.10.6.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="background-color:#EC9E97;"><span id="S4.T2.3.3.10.6.3.1" class="ltx_text" style="background-color:#EC9E97;">-3.7%</span></td>
<td id="S4.T2.3.3.10.6.4" class="ltx_td ltx_align_right ltx_border_bb">46.7</td>
<td id="S4.T2.3.3.10.6.5" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#EEABA5;"><span id="S4.T2.3.3.10.6.5.1" class="ltx_text" style="background-color:#EEABA5;">-3.2%</span></td>
<td id="S4.T2.3.3.10.6.6" class="ltx_td ltx_align_right ltx_border_bb">32.1</td>
<td id="S4.T2.3.3.10.6.7" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#E67C73;"><span id="S4.T2.3.3.10.6.7.1" class="ltx_text" style="background-color:#E67C73;">-7.5%</span></td>
<td id="S4.T2.3.3.10.6.8" class="ltx_td ltx_align_right ltx_border_bb">34.1</td>
<td id="S4.T2.3.3.10.6.9" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F5CFCC;"><span id="S4.T2.3.3.10.6.9.1" class="ltx_text" style="background-color:#F5CFCC;">-1.8%</span></td>
<td id="S4.T2.3.3.10.6.10" class="ltx_td ltx_align_right ltx_border_bb">39.1</td>
<td id="S4.T2.3.3.10.6.11" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#FAE5E3;"><span id="S4.T2.3.3.10.6.11.1" class="ltx_text" style="background-color:#FAE5E3;">-1.0%</span></td>
<td id="S4.T2.3.3.10.6.12" class="ltx_td ltx_align_right ltx_border_bb">59.3</td>
<td id="S4.T2.3.3.10.6.13" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#E67C73;"><span id="S4.T2.3.3.10.6.13.1" class="ltx_text" style="background-color:#E67C73;">-8.5%</span></td>
<td id="S4.T2.3.3.10.6.14" class="ltx_td ltx_align_right ltx_border_bb">67.5</td>
<td id="S4.T2.3.3.10.6.15" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FEFAFA;"><span id="S4.T2.3.3.10.6.15.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span><span id="S4.T2.7.1" class="ltx_text ltx_font_bold">Per-dataset average performance across non-English languages for 35B and 8B Aya 23 models at varying levels of quantization.</span> %<math id="S4.T2.5.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S4.T2.5.m1.1b"><mi mathvariant="normal" id="S4.T2.5.m1.1.1" xref="S4.T2.5.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S4.T2.5.m1.1c"><ci id="S4.T2.5.m1.1.1.cmml" xref="S4.T2.5.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.5.m1.1d">\Delta</annotation></semantics></math> is relative performance vs. FP16. We follow the evaluation setup of <cite class="ltx_cite ltx_citemacro_citet">Aryabumi et al. (<a href="#bib.bib5" title="" class="ltx_ref">2024</a>)</cite> and evaluate on languages in the 23 languages list. On “Unseen Tasks” (XWinograd, XCOPA, XStoryCloze), we use all the available languages.
See Section <a href="#S3.SS1" title="3.1 Automatic Evaluation ‣ 3 Experiment Set-up ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a> for details and language list.
</figcaption>
</figure>
<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">To clearly see the many-faceted impact of quantization, we discuss our results by quantization level (§<a href="#S4.SS1" title="4.1 By Quantization Level ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>), by task (§<a href="#S4.SS2" title="4.2 By Task ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>), by language (§<a href="#S4.SS3" title="4.3 By Language ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>), by model size (§<a href="#S4.SS4" title="4.4 By Model Size ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4</span></a>), and by quantization strategy (§<a href="#S4.SS5" title="4.5 By Quantization Strategy ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.5</span></a>). We then report LLM-as-a-Judge and RM-as-a-Judge (§<a href="#S4.SS6" title="4.6 LLM/RM-as-a-Judge ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.6</span></a>) and human evaluation results (§<a href="#S4.SS7" title="4.7 Human Evaluation ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.7</span></a>).</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>By Quantization Level</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p"><span id="S4.SS1.p1.1.1" class="ltx_text ltx_font_italic">How do different levels of quantization affect downstream performance?</span></p>
</div>
<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Command R and R+</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px1.p1.3" class="ltx_p">In Table <a href="#S4.T1" title="Table 1 ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, we aggregate results of each metric for each level of quantization. We average scores across languages, then calculate the relative percentage drop from FP16.<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span>Ex. For 103B <span id="footnote9.1" class="ltx_text ltx_font_bold">W4-g</span> MGSM, scores were: {de: 71.2, es: 75.7, fr: 69.0, ja: 58.0, zh: 68.9}, thus the average score was 68.6—a 2.9% drop from FP16 (<math id="footnote9.m1.1" class="ltx_Math" alttext="\frac{68.6-70.6}{70.6}=-0.029" display="inline"><semantics id="footnote9.m1.1b"><mrow id="footnote9.m1.1.1" xref="footnote9.m1.1.1.cmml"><mfrac id="footnote9.m1.1.1.2" xref="footnote9.m1.1.1.2.cmml"><mrow id="footnote9.m1.1.1.2.2" xref="footnote9.m1.1.1.2.2.cmml"><mn id="footnote9.m1.1.1.2.2.2" xref="footnote9.m1.1.1.2.2.2.cmml">68.6</mn><mo id="footnote9.m1.1.1.2.2.1" xref="footnote9.m1.1.1.2.2.1.cmml">−</mo><mn id="footnote9.m1.1.1.2.2.3" xref="footnote9.m1.1.1.2.2.3.cmml">70.6</mn></mrow><mn id="footnote9.m1.1.1.2.3" xref="footnote9.m1.1.1.2.3.cmml">70.6</mn></mfrac><mo id="footnote9.m1.1.1.1" xref="footnote9.m1.1.1.1.cmml">=</mo><mrow id="footnote9.m1.1.1.3" xref="footnote9.m1.1.1.3.cmml"><mo id="footnote9.m1.1.1.3b" xref="footnote9.m1.1.1.3.cmml">−</mo><mn id="footnote9.m1.1.1.3.2" xref="footnote9.m1.1.1.3.2.cmml">0.029</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote9.m1.1c"><apply id="footnote9.m1.1.1.cmml" xref="footnote9.m1.1.1"><eq id="footnote9.m1.1.1.1.cmml" xref="footnote9.m1.1.1.1"></eq><apply id="footnote9.m1.1.1.2.cmml" xref="footnote9.m1.1.1.2"><divide id="footnote9.m1.1.1.2.1.cmml" xref="footnote9.m1.1.1.2"></divide><apply id="footnote9.m1.1.1.2.2.cmml" xref="footnote9.m1.1.1.2.2"><minus id="footnote9.m1.1.1.2.2.1.cmml" xref="footnote9.m1.1.1.2.2.1"></minus><cn type="float" id="footnote9.m1.1.1.2.2.2.cmml" xref="footnote9.m1.1.1.2.2.2">68.6</cn><cn type="float" id="footnote9.m1.1.1.2.2.3.cmml" xref="footnote9.m1.1.1.2.2.3">70.6</cn></apply><cn type="float" id="footnote9.m1.1.1.2.3.cmml" xref="footnote9.m1.1.1.2.3">70.6</cn></apply><apply id="footnote9.m1.1.1.3.cmml" xref="footnote9.m1.1.1.3"><minus id="footnote9.m1.1.1.3.1.cmml" xref="footnote9.m1.1.1.3"></minus><cn type="float" id="footnote9.m1.1.1.3.2.cmml" xref="footnote9.m1.1.1.3.2">0.029</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote9.m1.1d">\frac{68.6-70.6}{70.6}=-0.029</annotation></semantics></math>).</span></span></span> We discuss results of <span id="S4.SS1.SSS0.Px1.p1.3.1" class="ltx_text ltx_font_bold">W8</span>, <span id="S4.SS1.SSS0.Px1.p1.3.2" class="ltx_text ltx_font_bold">W8A8</span>, and <span id="S4.SS1.SSS0.Px1.p1.3.3" class="ltx_text ltx_font_bold">W4-g</span> quantization, which are variants available for both Command model sizes. Most results follow intuition: greater quantization leads to larger performance degradation: <math id="S4.SS1.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="-0.2\%" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.1.m1.1a"><mrow id="S4.SS1.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.cmml"><mo id="S4.SS1.SSS0.Px1.p1.1.m1.1.1a" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.cmml">−</mo><mrow id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.cmml"><mn id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.2" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.2.cmml">0.2</mn><mo id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.1" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.1.m1.1b"><apply id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1"><minus id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1"></minus><apply id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.2">0.2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.1.m1.1c">-0.2\%</annotation></semantics></math> for <span id="S4.SS1.SSS0.Px1.p1.3.4" class="ltx_text ltx_font_bold">W8</span>, <math id="S4.SS1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="-0.8\%" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.2.m2.1a"><mrow id="S4.SS1.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.cmml"><mo id="S4.SS1.SSS0.Px1.p1.2.m2.1.1a" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.cmml">−</mo><mrow id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml"><mn id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.2" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.2.cmml">0.8</mn><mo id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.1" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.2.m2.1b"><apply id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1"><minus id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1"></minus><apply id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2"><csymbol cd="latexml" id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.2">0.8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.2.m2.1c">-0.8\%</annotation></semantics></math> for <span id="S4.SS1.SSS0.Px1.p1.3.5" class="ltx_text ltx_font_bold">W8A8</span>, and <math id="S4.SS1.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="-0.9\%" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.3.m3.1a"><mrow id="S4.SS1.SSS0.Px1.p1.3.m3.1.1" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.cmml"><mo id="S4.SS1.SSS0.Px1.p1.3.m3.1.1a" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.cmml">−</mo><mrow id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.cmml"><mn id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.2" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.2.cmml">0.9</mn><mo id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.1" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.3.m3.1b"><apply id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1"><minus id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1"></minus><apply id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2"><csymbol cd="latexml" id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.2.2">0.9</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.3.m3.1c">-0.9\%</annotation></semantics></math> overall for the 103B model. An exception is <span id="S4.SS1.SSS0.Px1.p1.3.6" class="ltx_text ltx_font_bold">W8A8</span> for the 35B, which experiences a slight boost overall due to higher performance on translation and language confusion evaluations.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Aya 23 Models</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">Table <a href="#S4.T2" title="Table 2 ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> shows the aggregated results for Aya 23 models on the extended Aya evaluations at <span id="S4.SS1.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_bold">W8</span>, and <span id="S4.SS1.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_bold">W4</span> quantization.
We find a similar trend with Command models where <span id="S4.SS1.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_bold">W4</span> often leads to a larger drop compared to <span id="S4.SS1.SSS0.Px2.p1.1.4" class="ltx_text ltx_font_bold">W8</span>, consistent across tasks and languages. <span id="S4.SS1.SSS0.Px2.p1.1.5" class="ltx_text ltx_font_bold">W8</span>, however, does not substantially drop performance in any task.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>By Task</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p"><span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_italic">Are tasks differently affected by quantization?</span></p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.4" class="ltx_p">Results here reference Tables <a href="#S4.T1" title="Table 1 ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and <a href="#S4.T2" title="Table 2 ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, with full raw and relative results in Appendix <a href="#A1.SS2" title="A.2 Automatic Tasks - Full Results ‣ Appendix A Appendix ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.2</span></a>.
Mathematical reasoning as measured by MGSM is strikingly affected by quantization. Relative performance of the 35B <span id="S4.SS2.p2.4.1" class="ltx_text ltx_font_bold">W4-g</span> model is a dismal <math id="S4.SS2.p2.1.m1.1" class="ltx_Math" alttext="-13.1\%" display="inline"><semantics id="S4.SS2.p2.1.m1.1a"><mrow id="S4.SS2.p2.1.m1.1.1" xref="S4.SS2.p2.1.m1.1.1.cmml"><mo id="S4.SS2.p2.1.m1.1.1a" xref="S4.SS2.p2.1.m1.1.1.cmml">−</mo><mrow id="S4.SS2.p2.1.m1.1.1.2" xref="S4.SS2.p2.1.m1.1.1.2.cmml"><mn id="S4.SS2.p2.1.m1.1.1.2.2" xref="S4.SS2.p2.1.m1.1.1.2.2.cmml">13.1</mn><mo id="S4.SS2.p2.1.m1.1.1.2.1" xref="S4.SS2.p2.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.1.m1.1b"><apply id="S4.SS2.p2.1.m1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1"><minus id="S4.SS2.p2.1.m1.1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1"></minus><apply id="S4.SS2.p2.1.m1.1.1.2.cmml" xref="S4.SS2.p2.1.m1.1.1.2"><csymbol cd="latexml" id="S4.SS2.p2.1.m1.1.1.2.1.cmml" xref="S4.SS2.p2.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS2.p2.1.m1.1.1.2.2.cmml" xref="S4.SS2.p2.1.m1.1.1.2.2">13.1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.1.m1.1c">-13.1\%</annotation></semantics></math> on average, with as poor as <math id="S4.SS2.p2.2.m2.1" class="ltx_Math" alttext="-17.3\%" display="inline"><semantics id="S4.SS2.p2.2.m2.1a"><mrow id="S4.SS2.p2.2.m2.1.1" xref="S4.SS2.p2.2.m2.1.1.cmml"><mo id="S4.SS2.p2.2.m2.1.1a" xref="S4.SS2.p2.2.m2.1.1.cmml">−</mo><mrow id="S4.SS2.p2.2.m2.1.1.2" xref="S4.SS2.p2.2.m2.1.1.2.cmml"><mn id="S4.SS2.p2.2.m2.1.1.2.2" xref="S4.SS2.p2.2.m2.1.1.2.2.cmml">17.3</mn><mo id="S4.SS2.p2.2.m2.1.1.2.1" xref="S4.SS2.p2.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.2.m2.1b"><apply id="S4.SS2.p2.2.m2.1.1.cmml" xref="S4.SS2.p2.2.m2.1.1"><minus id="S4.SS2.p2.2.m2.1.1.1.cmml" xref="S4.SS2.p2.2.m2.1.1"></minus><apply id="S4.SS2.p2.2.m2.1.1.2.cmml" xref="S4.SS2.p2.2.m2.1.1.2"><csymbol cd="latexml" id="S4.SS2.p2.2.m2.1.1.2.1.cmml" xref="S4.SS2.p2.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS2.p2.2.m2.1.1.2.2.cmml" xref="S4.SS2.p2.2.m2.1.1.2.2">17.3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.2.m2.1c">-17.3\%</annotation></semantics></math> in Chinese. MGSM and Belebele are most greatly degraded for Aya 23 models with <span id="S4.SS2.p2.4.2" class="ltx_text ltx_font_bold">W4</span> quantization, dropping <math id="S4.SS2.p2.3.m3.1" class="ltx_Math" alttext="7.5\%" display="inline"><semantics id="S4.SS2.p2.3.m3.1a"><mrow id="S4.SS2.p2.3.m3.1.1" xref="S4.SS2.p2.3.m3.1.1.cmml"><mn id="S4.SS2.p2.3.m3.1.1.2" xref="S4.SS2.p2.3.m3.1.1.2.cmml">7.5</mn><mo id="S4.SS2.p2.3.m3.1.1.1" xref="S4.SS2.p2.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.3.m3.1b"><apply id="S4.SS2.p2.3.m3.1.1.cmml" xref="S4.SS2.p2.3.m3.1.1"><csymbol cd="latexml" id="S4.SS2.p2.3.m3.1.1.1.cmml" xref="S4.SS2.p2.3.m3.1.1.1">percent</csymbol><cn type="float" id="S4.SS2.p2.3.m3.1.1.2.cmml" xref="S4.SS2.p2.3.m3.1.1.2">7.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.3.m3.1c">7.5\%</annotation></semantics></math> and <math id="S4.SS2.p2.4.m4.1" class="ltx_Math" alttext="8.5\%" display="inline"><semantics id="S4.SS2.p2.4.m4.1a"><mrow id="S4.SS2.p2.4.m4.1.1" xref="S4.SS2.p2.4.m4.1.1.cmml"><mn id="S4.SS2.p2.4.m4.1.1.2" xref="S4.SS2.p2.4.m4.1.1.2.cmml">8.5</mn><mo id="S4.SS2.p2.4.m4.1.1.1" xref="S4.SS2.p2.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.4.m4.1b"><apply id="S4.SS2.p2.4.m4.1.1.cmml" xref="S4.SS2.p2.4.m4.1.1"><csymbol cd="latexml" id="S4.SS2.p2.4.m4.1.1.1.cmml" xref="S4.SS2.p2.4.m4.1.1.1">percent</csymbol><cn type="float" id="S4.SS2.p2.4.m4.1.1.2.cmml" xref="S4.SS2.p2.4.m4.1.1.2">8.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.4.m4.1c">8.5\%</annotation></semantics></math> on the 8B model. mMMLU is the next most greatly degraded task.</p>
</div>
<figure id="S4.T3" class="ltx_table">
<div id="S4.T3.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:123.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-66.7pt,19.1pt) scale(0.764648409620307,0.764648409620307) ;">
<table id="S4.T3.1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T3.1.1.1" class="ltx_tr">
<td id="S4.T3.1.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S4.T3.1.1.1.3" class="ltx_td ltx_border_tt"></td>
<td id="S4.T3.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.4.1" class="ltx_text ltx_font_bold">ar</span></td>
<td id="S4.T3.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.5.1" class="ltx_text ltx_font_bold">de</span></td>
<td id="S4.T3.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.6.1" class="ltx_text ltx_font_bold">es</span></td>
<td id="S4.T3.1.1.1.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.7.1" class="ltx_text ltx_font_bold">fr</span></td>
<td id="S4.T3.1.1.1.8" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.8.1" class="ltx_text ltx_font_bold">it</span></td>
<td id="S4.T3.1.1.1.9" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.9.1" class="ltx_text ltx_font_bold">ja</span></td>
<td id="S4.T3.1.1.1.10" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.10.1" class="ltx_text ltx_font_bold">ko</span></td>
<td id="S4.T3.1.1.1.11" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.11.1" class="ltx_text ltx_font_bold">pt</span></td>
<td id="S4.T3.1.1.1.12" class="ltx_td ltx_align_center ltx_border_rr ltx_border_tt"><span id="S4.T3.1.1.1.12.1" class="ltx_text ltx_font_bold">zh</span></td>
<td id="S4.T3.1.1.1.13" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.13.1" class="ltx_text ltx_font_bold">Avg</span></td>
<td id="S4.T3.1.1.1.14" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.1.1.1.14.1" class="ltx_text ltx_font_bold">Ltn/IE</span></td>
<td id="S4.T3.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt"><math id="S4.T3.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T3.1.1.1.1.m1.1a"><mo id="S4.T3.1.1.1.1.m1.1.1" xref="S4.T3.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T3.1.1.1.1.m1.1b"><not id="S4.T3.1.1.1.1.m1.1.1.cmml" xref="S4.T3.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.1.1.1.1.m1.1c">\neg</annotation></semantics></math></td>
</tr>
<tr id="S4.T3.1.1.2.1" class="ltx_tr">
<td id="S4.T3.1.1.2.1.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="S4.T3.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFEFE;"><span id="S4.T3.1.1.2.1.3.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="S4.T3.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDFEFE;"><span id="S4.T3.1.1.2.1.4.1" class="ltx_text" style="background-color:#FDFEFE;">0.1%</span></td>
<td id="S4.T3.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFFFF;"><span id="S4.T3.1.1.2.1.5.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
<td id="S4.T3.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="S4.T3.1.1.2.1.6.1" class="ltx_text" style="background-color:#FEFDFD;">0.0%</span></td>
<td id="S4.T3.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFEFE;"><span id="S4.T3.1.1.2.1.7.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="S4.T3.1.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFFFE;"><span id="S4.T3.1.1.2.1.8.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="S4.T3.1.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFEFE;"><span id="S4.T3.1.1.2.1.9.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="S4.T3.1.1.2.1.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF4F3;"><span id="S4.T3.1.1.2.1.10.1" class="ltx_text" style="background-color:#FCF4F3;">-0.4%</span></td>
<td id="S4.T3.1.1.2.1.11" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FDF8F8;"><span id="S4.T3.1.1.2.1.11.1" class="ltx_text" style="background-color:#FDF8F8;">-0.2%</span></td>
<td id="S4.T3.1.1.2.1.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="S4.T3.1.1.2.1.12.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="S4.T3.1.1.2.1.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="S4.T3.1.1.2.1.13.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="S4.T3.1.1.2.1.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="S4.T3.1.1.2.1.14.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
</tr>
<tr id="S4.T3.1.1.3.2" class="ltx_tr">
<td id="S4.T3.1.1.3.2.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.1.1.3.2.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8A8-sq</td>
<td id="S4.T3.1.1.3.2.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF0EF;"><span id="S4.T3.1.1.3.2.3.1" class="ltx_text" style="background-color:#FCF0EF;">-0.6%</span></td>
<td id="S4.T3.1.1.3.2.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAFDFC;"><span id="S4.T3.1.1.3.2.4.1" class="ltx_text" style="background-color:#FAFDFC;">0.2%</span></td>
<td id="S4.T3.1.1.3.2.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F6;"><span id="S4.T3.1.1.3.2.5.1" class="ltx_text" style="background-color:#FDF6F6;">-0.3%</span></td>
<td id="S4.T3.1.1.3.2.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCFEFD;"><span id="S4.T3.1.1.3.2.6.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="S4.T3.1.1.3.2.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEEEC;"><span id="S4.T3.1.1.3.2.7.1" class="ltx_text" style="background-color:#FBEEEC;">-0.6%</span></td>
<td id="S4.T3.1.1.3.2.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F5;"><span id="S4.T3.1.1.3.2.8.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="S4.T3.1.1.3.2.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="S4.T3.1.1.3.2.9.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="S4.T3.1.1.3.2.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEDEC;"><span id="S4.T3.1.1.3.2.10.1" class="ltx_text" style="background-color:#FBEDEC;">-0.7%</span></td>
<td id="S4.T3.1.1.3.2.11" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FBEAE9;"><span id="S4.T3.1.1.3.2.11.1" class="ltx_text" style="background-color:#FBEAE9;">-0.8%</span></td>
<td id="S4.T3.1.1.3.2.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F5;"><span id="S4.T3.1.1.3.2.12.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="S4.T3.1.1.3.2.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF8F7;"><span id="S4.T3.1.1.3.2.13.1" class="ltx_text" style="background-color:#FDF8F7;">-0.3%</span></td>
<td id="S4.T3.1.1.3.2.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F2;"><span id="S4.T3.1.1.3.2.14.1" class="ltx_text" style="background-color:#FCF3F2;">-0.4%</span></td>
</tr>
<tr id="S4.T3.1.1.4.3" class="ltx_tr">
<td id="S4.T3.1.1.4.3.1" class="ltx_td"></td>
<td id="S4.T3.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="S4.T3.1.1.4.3.3" class="ltx_td ltx_align_right" style="background-color:#F8DDDB;"><span id="S4.T3.1.1.4.3.3.1" class="ltx_text" style="background-color:#F8DDDB;">-1.3%</span></td>
<td id="S4.T3.1.1.4.3.4" class="ltx_td ltx_align_right" style="background-color:#FAE7E6;"><span id="S4.T3.1.1.4.3.4.1" class="ltx_text" style="background-color:#FAE7E6;">-0.9%</span></td>
<td id="S4.T3.1.1.4.3.5" class="ltx_td ltx_align_right" style="background-color:#FCF3F2;"><span id="S4.T3.1.1.4.3.5.1" class="ltx_text" style="background-color:#FCF3F2;">-0.5%</span></td>
<td id="S4.T3.1.1.4.3.6" class="ltx_td ltx_align_right" style="background-color:#FCF3F2;"><span id="S4.T3.1.1.4.3.6.1" class="ltx_text" style="background-color:#FCF3F2;">-0.5%</span></td>
<td id="S4.T3.1.1.4.3.7" class="ltx_td ltx_align_right" style="background-color:#FAE9E7;"><span id="S4.T3.1.1.4.3.7.1" class="ltx_text" style="background-color:#FAE9E7;">-0.8%</span></td>
<td id="S4.T3.1.1.4.3.8" class="ltx_td ltx_align_right" style="background-color:#FDF7F6;"><span id="S4.T3.1.1.4.3.8.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
<td id="S4.T3.1.1.4.3.9" class="ltx_td ltx_align_right" style="background-color:#F8DCD9;"><span id="S4.T3.1.1.4.3.9.1" class="ltx_text" style="background-color:#F8DCD9;">-1.3%</span></td>
<td id="S4.T3.1.1.4.3.10" class="ltx_td ltx_align_right" style="background-color:#FBEBE9;"><span id="S4.T3.1.1.4.3.10.1" class="ltx_text" style="background-color:#FBEBE9;">-0.8%</span></td>
<td id="S4.T3.1.1.4.3.11" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#FAE7E5;"><span id="S4.T3.1.1.4.3.11.1" class="ltx_text" style="background-color:#FAE7E5;">-0.9%</span></td>
<td id="S4.T3.1.1.4.3.12" class="ltx_td ltx_align_right" style="background-color:#FBEAE8;"><span id="S4.T3.1.1.4.3.12.1" class="ltx_text" style="background-color:#FBEAE8;">-0.8%</span></td>
<td id="S4.T3.1.1.4.3.13" class="ltx_td ltx_align_right" style="background-color:#FBEDEC;"><span id="S4.T3.1.1.4.3.13.1" class="ltx_text" style="background-color:#FBEDEC;">-0.7%</span></td>
<td id="S4.T3.1.1.4.3.14" class="ltx_td ltx_align_right" style="background-color:#FAE6E4;"><span id="S4.T3.1.1.4.3.14.1" class="ltx_text" style="background-color:#FAE6E4;">-0.9%</span></td>
</tr>
<tr id="S4.T3.1.1.5.4" class="ltx_tr">
<td id="S4.T3.1.1.5.4.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T3.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W4-g</td>
<td id="S4.T3.1.1.5.4.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE9E7;"><span id="S4.T3.1.1.5.4.3.1" class="ltx_text" style="background-color:#FAE9E7;">-0.8%</span></td>
<td id="S4.T3.1.1.5.4.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAFA;"><span id="S4.T3.1.1.5.4.4.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="S4.T3.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF5F4;"><span id="S4.T3.1.1.5.4.5.1" class="ltx_text" style="background-color:#FDF5F4;">-0.4%</span></td>
<td id="S4.T3.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFFFE;"><span id="S4.T3.1.1.5.4.6.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="S4.T3.1.1.5.4.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF5F5;"><span id="S4.T3.1.1.5.4.7.1" class="ltx_text" style="background-color:#FDF5F5;">-0.4%</span></td>
<td id="S4.T3.1.1.5.4.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F3;"><span id="S4.T3.1.1.5.4.8.1" class="ltx_text" style="background-color:#FCF3F3;">-0.4%</span></td>
<td id="S4.T3.1.1.5.4.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF0EF;"><span id="S4.T3.1.1.5.4.9.1" class="ltx_text" style="background-color:#FCF0EF;">-0.6%</span></td>
<td id="S4.T3.1.1.5.4.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8DFDD;"><span id="S4.T3.1.1.5.4.10.1" class="ltx_text" style="background-color:#F8DFDD;">-1.2%</span></td>
<td id="S4.T3.1.1.5.4.11" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FAE6E5;"><span id="S4.T3.1.1.5.4.11.1" class="ltx_text" style="background-color:#FAE6E5;">-0.9%</span></td>
<td id="S4.T3.1.1.5.4.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF1F0;"><span id="S4.T3.1.1.5.4.12.1" class="ltx_text" style="background-color:#FCF1F0;">-0.5%</span></td>
<td id="S4.T3.1.1.5.4.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF4F3;"><span id="S4.T3.1.1.5.4.13.1" class="ltx_text" style="background-color:#FCF4F3;">-0.4%</span></td>
<td id="S4.T3.1.1.5.4.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEDEB;"><span id="S4.T3.1.1.5.4.14.1" class="ltx_text" style="background-color:#FBEDEB;">-0.7%</span></td>
</tr>
<tr id="S4.T3.1.1.6.5" class="ltx_tr">
<td id="S4.T3.1.1.6.5.1" class="ltx_td ltx_align_left"><span id="S4.T3.1.1.6.5.1.1" class="ltx_text">103B</span></td>
<td id="S4.T3.1.1.6.5.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="S4.T3.1.1.6.5.3" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="S4.T3.1.1.6.5.3.1" class="ltx_text" style="background-color:#F9E4E2;">-1.0%</span></td>
<td id="S4.T3.1.1.6.5.4" class="ltx_td ltx_align_right" style="background-color:#FBEFED;"><span id="S4.T3.1.1.6.5.4.1" class="ltx_text" style="background-color:#FBEFED;">-0.6%</span></td>
<td id="S4.T3.1.1.6.5.5" class="ltx_td ltx_align_right" style="background-color:#FDFEFE;"><span id="S4.T3.1.1.6.5.5.1" class="ltx_text" style="background-color:#FDFEFE;">0.1%</span></td>
<td id="S4.T3.1.1.6.5.6" class="ltx_td ltx_align_right" style="background-color:#FAE9E7;"><span id="S4.T3.1.1.6.5.6.1" class="ltx_text" style="background-color:#FAE9E7;">-0.8%</span></td>
<td id="S4.T3.1.1.6.5.7" class="ltx_td ltx_align_right" style="background-color:#F9E0DD;"><span id="S4.T3.1.1.6.5.7.1" class="ltx_text" style="background-color:#F9E0DD;">-1.2%</span></td>
<td id="S4.T3.1.1.6.5.8" class="ltx_td ltx_align_right" style="background-color:#F7D9D7;"><span id="S4.T3.1.1.6.5.8.1" class="ltx_text" style="background-color:#F7D9D7;">-1.4%</span></td>
<td id="S4.T3.1.1.6.5.9" class="ltx_td ltx_align_right" style="background-color:#F0B2AC;"><span id="S4.T3.1.1.6.5.9.1" class="ltx_text" style="background-color:#F0B2AC;">-2.9%</span></td>
<td id="S4.T3.1.1.6.5.10" class="ltx_td ltx_align_right" style="background-color:#FBEAE9;"><span id="S4.T3.1.1.6.5.10.1" class="ltx_text" style="background-color:#FBEAE9;">-0.8%</span></td>
<td id="S4.T3.1.1.6.5.11" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F3C2BE;"><span id="S4.T3.1.1.6.5.11.1" class="ltx_text" style="background-color:#F3C2BE;">-2.3%</span></td>
<td id="S4.T3.1.1.6.5.12" class="ltx_td ltx_align_right" style="background-color:#F8DFDC;"><span id="S4.T3.1.1.6.5.12.1" class="ltx_text" style="background-color:#F8DFDC;">-1.2%</span></td>
<td id="S4.T3.1.1.6.5.13" class="ltx_td ltx_align_right" style="background-color:#FBEDEC;"><span id="S4.T3.1.1.6.5.13.1" class="ltx_text" style="background-color:#FBEDEC;">-0.7%</span></td>
<td id="S4.T3.1.1.6.5.14" class="ltx_td ltx_align_right" style="background-color:#F5CCC9;"><span id="S4.T3.1.1.6.5.14.1" class="ltx_text" style="background-color:#F5CCC9;">-1.9%</span></td>
</tr>
<tr id="S4.T3.1.1.7.6" class="ltx_tr">
<td id="S4.T3.1.1.7.6.1" class="ltx_td ltx_border_tt"></td>
<td id="S4.T3.1.1.7.6.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt">W8</td>
<td id="S4.T3.1.1.7.6.3" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#F9FDFB;"><span id="S4.T3.1.1.7.6.3.1" class="ltx_text" style="background-color:#F9FDFB;">0.3%</span></td>
<td id="S4.T3.1.1.7.6.4" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FCF2F1;"><span id="S4.T3.1.1.7.6.4.1" class="ltx_text" style="background-color:#FCF2F1;">-0.5%</span></td>
<td id="S4.T3.1.1.7.6.5" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FEFCFB;"><span id="S4.T3.1.1.7.6.5.1" class="ltx_text" style="background-color:#FEFCFB;">-0.1%</span></td>
<td id="S4.T3.1.1.7.6.6" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FEFAF9;"><span id="S4.T3.1.1.7.6.6.1" class="ltx_text" style="background-color:#FEFAF9;">-0.2%</span></td>
<td id="S4.T3.1.1.7.6.7" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FDF5F4;"><span id="S4.T3.1.1.7.6.7.1" class="ltx_text" style="background-color:#FDF5F4;">-0.4%</span></td>
<td id="S4.T3.1.1.7.6.8" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#F7FCFA;"><span id="S4.T3.1.1.7.6.8.1" class="ltx_text" style="background-color:#F7FCFA;">0.3%</span></td>
<td id="S4.T3.1.1.7.6.9" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FEFBFA;"><span id="S4.T3.1.1.7.6.9.1" class="ltx_text" style="background-color:#FEFBFA;">-0.1%</span></td>
<td id="S4.T3.1.1.7.6.10" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FEFFFE;"><span id="S4.T3.1.1.7.6.10.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="S4.T3.1.1.7.6.11" class="ltx_td ltx_align_right ltx_border_rr ltx_border_tt" style="background-color:#FDF8F7;"><span id="S4.T3.1.1.7.6.11.1" class="ltx_text" style="background-color:#FDF8F7;">-0.3%</span></td>
<td id="S4.T3.1.1.7.6.12" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FEFCFC;"><span id="S4.T3.1.1.7.6.12.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="S4.T3.1.1.7.6.13" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FDF9F8;"><span id="S4.T3.1.1.7.6.13.1" class="ltx_text" style="background-color:#FDF9F8;">-0.2%</span></td>
<td id="S4.T3.1.1.7.6.14" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FEFFFF;"><span id="S4.T3.1.1.7.6.14.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
</tr>
<tr id="S4.T3.1.1.8.7" class="ltx_tr">
<td id="S4.T3.1.1.8.7.1" class="ltx_td"></td>
<td id="S4.T3.1.1.8.7.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="S4.T3.1.1.8.7.3" class="ltx_td ltx_align_right" style="background-color:#CBEADB;"><span id="S4.T3.1.1.8.7.3.1" class="ltx_text" style="background-color:#CBEADB;">2.0%</span></td>
<td id="S4.T3.1.1.8.7.4" class="ltx_td ltx_align_right" style="background-color:#C0E6D3;"><span id="S4.T3.1.1.8.7.4.1" class="ltx_text" style="background-color:#C0E6D3;">2.5%</span></td>
<td id="S4.T3.1.1.8.7.5" class="ltx_td ltx_align_right" style="background-color:#EFF9F4;"><span id="S4.T3.1.1.8.7.5.1" class="ltx_text" style="background-color:#EFF9F4;">0.7%</span></td>
<td id="S4.T3.1.1.8.7.6" class="ltx_td ltx_align_right" style="background-color:#E5F5ED;"><span id="S4.T3.1.1.8.7.6.1" class="ltx_text" style="background-color:#E5F5ED;">1.0%</span></td>
<td id="S4.T3.1.1.8.7.7" class="ltx_td ltx_align_right" style="background-color:#E1F3EA;"><span id="S4.T3.1.1.8.7.7.1" class="ltx_text" style="background-color:#E1F3EA;">1.2%</span></td>
<td id="S4.T3.1.1.8.7.8" class="ltx_td ltx_align_right" style="background-color:#E2F4EB;"><span id="S4.T3.1.1.8.7.8.1" class="ltx_text" style="background-color:#E2F4EB;">1.1%</span></td>
<td id="S4.T3.1.1.8.7.9" class="ltx_td ltx_align_right" style="background-color:#E9F6F0;"><span id="S4.T3.1.1.8.7.9.1" class="ltx_text" style="background-color:#E9F6F0;">0.9%</span></td>
<td id="S4.T3.1.1.8.7.10" class="ltx_td ltx_align_right" style="background-color:#DAF0E6;"><span id="S4.T3.1.1.8.7.10.1" class="ltx_text" style="background-color:#DAF0E6;">1.4%</span></td>
<td id="S4.T3.1.1.8.7.11" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#E4F5ED;"><span id="S4.T3.1.1.8.7.11.1" class="ltx_text" style="background-color:#E4F5ED;">1.0%</span></td>
<td id="S4.T3.1.1.8.7.12" class="ltx_td ltx_align_right" style="background-color:#DEF2E8;"><span id="S4.T3.1.1.8.7.12.1" class="ltx_text" style="background-color:#DEF2E8;">1.3%</span></td>
<td id="S4.T3.1.1.8.7.13" class="ltx_td ltx_align_right" style="background-color:#DDF1E7;"><span id="S4.T3.1.1.8.7.13.1" class="ltx_text" style="background-color:#DDF1E7;">1.3%</span></td>
<td id="S4.T3.1.1.8.7.14" class="ltx_td ltx_align_right" style="background-color:#DFF2E9;"><span id="S4.T3.1.1.8.7.14.1" class="ltx_text" style="background-color:#DFF2E9;">1.3%</span></td>
</tr>
<tr id="S4.T3.1.1.9.8" class="ltx_tr">
<td id="S4.T3.1.1.9.8.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T3.1.1.9.8.1.1" class="ltx_text">35B</span></td>
<td id="S4.T3.1.1.9.8.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4-g</td>
<td id="S4.T3.1.1.9.8.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9E1DF;"><span id="S4.T3.1.1.9.8.3.1" class="ltx_text" style="background-color:#F9E1DF;">-1.1%</span></td>
<td id="S4.T3.1.1.9.8.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9E2E0;"><span id="S4.T3.1.1.9.8.4.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="S4.T3.1.1.9.8.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FDFFFE;"><span id="S4.T3.1.1.9.8.5.1" class="ltx_text" style="background-color:#FDFFFE;">0.1%</span></td>
<td id="S4.T3.1.1.9.8.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FDF7F6;"><span id="S4.T3.1.1.9.8.6.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
<td id="S4.T3.1.1.9.8.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FEFDFC;"><span id="S4.T3.1.1.9.8.7.1" class="ltx_text" style="background-color:#FEFDFC;">-0.1%</span></td>
<td id="S4.T3.1.1.9.8.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C3BF;"><span id="S4.T3.1.1.9.8.8.1" class="ltx_text" style="background-color:#F3C3BF;">-2.3%</span></td>
<td id="S4.T3.1.1.9.8.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F7D9D7;"><span id="S4.T3.1.1.9.8.9.1" class="ltx_text" style="background-color:#F7D9D7;">-1.4%</span></td>
<td id="S4.T3.1.1.9.8.10" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FBEFEE;"><span id="S4.T3.1.1.9.8.10.1" class="ltx_text" style="background-color:#FBEFEE;">-0.6%</span></td>
<td id="S4.T3.1.1.9.8.11" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#F8DBD9;"><span id="S4.T3.1.1.9.8.11.1" class="ltx_text" style="background-color:#F8DBD9;">-1.3%</span></td>
<td id="S4.T3.1.1.9.8.12" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE7E5;"><span id="S4.T3.1.1.9.8.12.1" class="ltx_text" style="background-color:#FAE7E5;">-0.9%</span></td>
<td id="S4.T3.1.1.9.8.13" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FDF4F3;"><span id="S4.T3.1.1.9.8.13.1" class="ltx_text" style="background-color:#FDF4F3;">-0.4%</span></td>
<td id="S4.T3.1.1.9.8.14" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F7D6D4;"><span id="S4.T3.1.1.9.8.14.1" class="ltx_text" style="background-color:#F7D6D4;">-1.5%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span><span id="S4.T3.4.1" class="ltx_text ltx_font_bold">Per-language relative performance (%<math id="S4.T3.4.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S4.T3.4.1.m1.1b"><mi mathvariant="normal" id="S4.T3.4.1.m1.1.1" xref="S4.T3.4.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S4.T3.4.1.m1.1c"><ci id="S4.T3.4.1.m1.1.1.cmml" xref="S4.T3.4.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.4.1.m1.1d">\Delta</annotation></semantics></math>) vs. FP16, averaged over mMMLU, FLORES, and Language Confusion tasks.</span> Ltn/IE are Latin-script/Indo-European languages: de, es, fr, it, pt. <math id="S4.T3.5.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T3.5.m1.1b"><mo id="S4.T3.5.m1.1.1" xref="S4.T3.5.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T3.5.m1.1c"><not id="S4.T3.5.m1.1.1.cmml" xref="S4.T3.5.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.5.m1.1d">\neg</annotation></semantics></math> are the rest: ar, ja, ko, zh.</figcaption>
</figure>
<figure id="S4.T4" class="ltx_table">
<div id="S4.T4.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:173.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(14.5pt,-5.8pt) scale(1.07162054017176,1.07162054017176) ;">
<table id="S4.T4.1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T4.1.1.1" class="ltx_tr">
<td id="S4.T4.1.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S4.T4.1.1.1.3" class="ltx_td ltx_border_r ltx_border_tt"></td>
<td id="S4.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T4.1.1.1.4.1" class="ltx_text ltx_font_bold">de</span></td>
<td id="S4.T4.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T4.1.1.1.5.1" class="ltx_text ltx_font_bold">es</span></td>
<td id="S4.T4.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T4.1.1.1.6.1" class="ltx_text ltx_font_bold">fr</span></td>
<td id="S4.T4.1.1.1.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T4.1.1.1.7.1" class="ltx_text ltx_font_bold">ja</span></td>
<td id="S4.T4.1.1.1.8" class="ltx_td ltx_align_center ltx_border_rr ltx_border_tt"><span id="S4.T4.1.1.1.8.1" class="ltx_text ltx_font_bold">zh</span></td>
<td id="S4.T4.1.1.1.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S4.T4.1.1.1.9.1" class="ltx_text ltx_font_bold">Avg</span></td>
<td id="S4.T4.1.1.1.10" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T4.1.1.1.10.1" class="ltx_text ltx_font_bold">Ltn/IE</span></td>
<td id="S4.T4.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt"><math id="S4.T4.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T4.1.1.1.1.m1.1a"><mo id="S4.T4.1.1.1.1.m1.1.1" xref="S4.T4.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T4.1.1.1.1.m1.1b"><not id="S4.T4.1.1.1.1.m1.1.1.cmml" xref="S4.T4.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.1.1.1.1.m1.1c">\neg</annotation></semantics></math></td>
</tr>
<tr id="S4.T4.1.1.2.1" class="ltx_tr">
<td id="S4.T4.1.1.2.1.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="S4.T4.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCFEFD;"><span id="S4.T4.1.1.2.1.3.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="S4.T4.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="S4.T4.1.1.2.1.4.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="S4.T4.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F6;"><span id="S4.T4.1.1.2.1.5.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
<td id="S4.T4.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF4F4;"><span id="S4.T4.1.1.2.1.6.1" class="ltx_text" style="background-color:#FDF4F4;">-0.4%</span></td>
<td id="S4.T4.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FDF9F8;"><span id="S4.T4.1.1.2.1.7.1" class="ltx_text" style="background-color:#FDF9F8;">-0.2%</span></td>
<td id="S4.T4.1.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAFA;"><span id="S4.T4.1.1.2.1.8.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="S4.T4.1.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="S4.T4.1.1.2.1.9.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="S4.T4.1.1.2.1.10" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="background-color:#FDF7F6;"><span id="S4.T4.1.1.2.1.10.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
</tr>
<tr id="S4.T4.1.1.3.2" class="ltx_tr">
<td id="S4.T4.1.1.3.2.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.1.1.3.2.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8A8-sq</td>
<td id="S4.T4.1.1.3.2.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F6FCF9;"><span id="S4.T4.1.1.3.2.3.1" class="ltx_text" style="background-color:#F6FCF9;">0.4%</span></td>
<td id="S4.T4.1.1.3.2.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE8E6;"><span id="S4.T4.1.1.3.2.4.1" class="ltx_text" style="background-color:#FAE8E6;">-0.9%</span></td>
<td id="S4.T4.1.1.3.2.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="S4.T4.1.1.3.2.5.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="S4.T4.1.1.3.2.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F6;"><span id="S4.T4.1.1.3.2.6.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
<td id="S4.T4.1.1.3.2.7" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#F8DEDC;"><span id="S4.T4.1.1.3.2.7.1" class="ltx_text" style="background-color:#F8DEDC;">-1.2%</span></td>
<td id="S4.T4.1.1.3.2.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F2;"><span id="S4.T4.1.1.3.2.8.1" class="ltx_text" style="background-color:#FCF3F2;">-0.4%</span></td>
<td id="S4.T4.1.1.3.2.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F9;"><span id="S4.T4.1.1.3.2.9.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="S4.T4.1.1.3.2.10" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="background-color:#FBEAE9;"><span id="S4.T4.1.1.3.2.10.1" class="ltx_text" style="background-color:#FBEAE9;">-0.8%</span></td>
</tr>
<tr id="S4.T4.1.1.4.3" class="ltx_tr">
<td id="S4.T4.1.1.4.3.1" class="ltx_td"></td>
<td id="S4.T4.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="S4.T4.1.1.4.3.3" class="ltx_td ltx_align_right" style="background-color:#FDF4F4;"><span id="S4.T4.1.1.4.3.3.1" class="ltx_text" style="background-color:#FDF4F4;">-0.4%</span></td>
<td id="S4.T4.1.1.4.3.4" class="ltx_td ltx_align_right" style="background-color:#FAE5E3;"><span id="S4.T4.1.1.4.3.4.1" class="ltx_text" style="background-color:#FAE5E3;">-1.0%</span></td>
<td id="S4.T4.1.1.4.3.5" class="ltx_td ltx_align_right" style="background-color:#FCF0EF;"><span id="S4.T4.1.1.4.3.5.1" class="ltx_text" style="background-color:#FCF0EF;">-0.6%</span></td>
<td id="S4.T4.1.1.4.3.6" class="ltx_td ltx_align_right" style="background-color:#FEFBFB;"><span id="S4.T4.1.1.4.3.6.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="S4.T4.1.1.4.3.7" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F8DDDB;"><span id="S4.T4.1.1.4.3.7.1" class="ltx_text" style="background-color:#F8DDDB;">-1.3%</span></td>
<td id="S4.T4.1.1.4.3.8" class="ltx_td ltx_align_right" style="background-color:#FBEDEC;"><span id="S4.T4.1.1.4.3.8.1" class="ltx_text" style="background-color:#FBEDEC;">-0.7%</span></td>
<td id="S4.T4.1.1.4.3.9" class="ltx_td ltx_align_right" style="background-color:#FBEEED;"><span id="S4.T4.1.1.4.3.9.1" class="ltx_text" style="background-color:#FBEEED;">-0.6%</span></td>
<td id="S4.T4.1.1.4.3.10" class="ltx_td ltx_nopad_r ltx_align_right" style="background-color:#FBECEB;"><span id="S4.T4.1.1.4.3.10.1" class="ltx_text" style="background-color:#FBECEB;">-0.7%</span></td>
</tr>
<tr id="S4.T4.1.1.5.4" class="ltx_tr">
<td id="S4.T4.1.1.5.4.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W4-g</td>
<td id="S4.T4.1.1.5.4.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F2;"><span id="S4.T4.1.1.5.4.3.1" class="ltx_text" style="background-color:#FCF3F2;">-0.5%</span></td>
<td id="S4.T4.1.1.5.4.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF1F0;"><span id="S4.T4.1.1.5.4.4.1" class="ltx_text" style="background-color:#FCF1F0;">-0.5%</span></td>
<td id="S4.T4.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F5;"><span id="S4.T4.1.1.5.4.5.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="S4.T4.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F6D3D0;"><span id="S4.T4.1.1.5.4.6.1" class="ltx_text" style="background-color:#F6D3D0;">-1.7%</span></td>
<td id="S4.T4.1.1.5.4.7" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#F9E2E0;"><span id="S4.T4.1.1.5.4.7.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="S4.T4.1.1.5.4.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE9E8;"><span id="S4.T4.1.1.5.4.8.1" class="ltx_text" style="background-color:#FAE9E8;">-0.8%</span></td>
<td id="S4.T4.1.1.5.4.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F2;"><span id="S4.T4.1.1.5.4.9.1" class="ltx_text" style="background-color:#FCF3F2;">-0.4%</span></td>
<td id="S4.T4.1.1.5.4.10" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="background-color:#F8DAD8;"><span id="S4.T4.1.1.5.4.10.1" class="ltx_text" style="background-color:#F8DAD8;">-1.4%</span></td>
</tr>
<tr id="S4.T4.1.1.6.5" class="ltx_tr">
<td id="S4.T4.1.1.6.5.1" class="ltx_td ltx_align_left"><span id="S4.T4.1.1.6.5.1.1" class="ltx_text">103B</span></td>
<td id="S4.T4.1.1.6.5.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="S4.T4.1.1.6.5.3" class="ltx_td ltx_align_right" style="background-color:#F3C1BD;"><span id="S4.T4.1.1.6.5.3.1" class="ltx_text" style="background-color:#F3C1BD;">-2.3%</span></td>
<td id="S4.T4.1.1.6.5.4" class="ltx_td ltx_align_right" style="background-color:#F9E2E0;"><span id="S4.T4.1.1.6.5.4.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="S4.T4.1.1.6.5.5" class="ltx_td ltx_align_right" style="background-color:#F6D3D0;"><span id="S4.T4.1.1.6.5.5.1" class="ltx_text" style="background-color:#F6D3D0;">-1.7%</span></td>
<td id="S4.T4.1.1.6.5.6" class="ltx_td ltx_align_right" style="background-color:#EFB0AA;"><span id="S4.T4.1.1.6.5.6.1" class="ltx_text" style="background-color:#EFB0AA;">-3.0%</span></td>
<td id="S4.T4.1.1.6.5.7" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#EDA29C;"><span id="S4.T4.1.1.6.5.7.1" class="ltx_text" style="background-color:#EDA29C;">-3.5%</span></td>
<td id="S4.T4.1.1.6.5.8" class="ltx_td ltx_align_right" style="background-color:#F3C2BD;"><span id="S4.T4.1.1.6.5.8.1" class="ltx_text" style="background-color:#F3C2BD;">-2.3%</span></td>
<td id="S4.T4.1.1.6.5.9" class="ltx_td ltx_align_right" style="background-color:#F6D2CF;"><span id="S4.T4.1.1.6.5.9.1" class="ltx_text" style="background-color:#F6D2CF;">-1.7%</span></td>
<td id="S4.T4.1.1.6.5.10" class="ltx_td ltx_nopad_r ltx_align_right" style="background-color:#EEA9A3;"><span id="S4.T4.1.1.6.5.10.1" class="ltx_text" style="background-color:#EEA9A3;">-3.3%</span></td>
</tr>
<tr id="S4.T4.1.1.7.6" class="ltx_tr">
<td id="S4.T4.1.1.7.6.1" class="ltx_td ltx_border_t"></td>
<td id="S4.T4.1.1.7.6.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="S4.T4.1.1.7.6.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEFED;"><span id="S4.T4.1.1.7.6.3.1" class="ltx_text" style="background-color:#FBEFED;">-0.6%</span></td>
<td id="S4.T4.1.1.7.6.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F7;"><span id="S4.T4.1.1.7.6.4.1" class="ltx_text" style="background-color:#FDF7F7;">-0.3%</span></td>
<td id="S4.T4.1.1.7.6.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="S4.T4.1.1.7.6.5.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="S4.T4.1.1.7.6.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF5F5;"><span id="S4.T4.1.1.7.6.6.1" class="ltx_text" style="background-color:#FDF5F5;">-0.4%</span></td>
<td id="S4.T4.1.1.7.6.7" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FEFFFF;"><span id="S4.T4.1.1.7.6.7.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
<td id="S4.T4.1.1.7.6.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF8F8;"><span id="S4.T4.1.1.7.6.8.1" class="ltx_text" style="background-color:#FDF8F8;">-0.2%</span></td>
<td id="S4.T4.1.1.7.6.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F6;"><span id="S4.T4.1.1.7.6.9.1" class="ltx_text" style="background-color:#FDF6F6;">-0.3%</span></td>
<td id="S4.T4.1.1.7.6.10" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="background-color:#FEFBFA;"><span id="S4.T4.1.1.7.6.10.1" class="ltx_text" style="background-color:#FEFBFA;">-0.2%</span></td>
</tr>
<tr id="S4.T4.1.1.8.7" class="ltx_tr">
<td id="S4.T4.1.1.8.7.1" class="ltx_td"></td>
<td id="S4.T4.1.1.8.7.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="S4.T4.1.1.8.7.3" class="ltx_td ltx_align_right" style="background-color:#DEF2E8;"><span id="S4.T4.1.1.8.7.3.1" class="ltx_text" style="background-color:#DEF2E8;">1.3%</span></td>
<td id="S4.T4.1.1.8.7.4" class="ltx_td ltx_align_right" style="background-color:#FCEFEE;"><span id="S4.T4.1.1.8.7.4.1" class="ltx_text" style="background-color:#FCEFEE;">-0.6%</span></td>
<td id="S4.T4.1.1.8.7.5" class="ltx_td ltx_align_right" style="background-color:#F9FDFB;"><span id="S4.T4.1.1.8.7.5.1" class="ltx_text" style="background-color:#F9FDFB;">0.3%</span></td>
<td id="S4.T4.1.1.8.7.6" class="ltx_td ltx_align_right" style="background-color:#FDF6F5;"><span id="S4.T4.1.1.8.7.6.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="S4.T4.1.1.8.7.7" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#FEFEFE;"><span id="S4.T4.1.1.8.7.7.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="S4.T4.1.1.8.7.8" class="ltx_td ltx_align_right" style="background-color:#FCFEFD;"><span id="S4.T4.1.1.8.7.8.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="S4.T4.1.1.8.7.9" class="ltx_td ltx_align_right" style="background-color:#F7FCFA;"><span id="S4.T4.1.1.8.7.9.1" class="ltx_text" style="background-color:#F7FCFA;">0.3%</span></td>
<td id="S4.T4.1.1.8.7.10" class="ltx_td ltx_nopad_r ltx_align_right" style="background-color:#FEFAFA;"><span id="S4.T4.1.1.8.7.10.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
</tr>
<tr id="S4.T4.1.1.9.8" class="ltx_tr">
<td id="S4.T4.1.1.9.8.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T4.1.1.9.8.1.1" class="ltx_text">35B</span></td>
<td id="S4.T4.1.1.9.8.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4-g</td>
<td id="S4.T4.1.1.9.8.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EC9E97;"><span id="S4.T4.1.1.9.8.3.1" class="ltx_text" style="background-color:#EC9E97;">-3.7%</span></td>
<td id="S4.T4.1.1.9.8.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F6D1CD;"><span id="S4.T4.1.1.9.8.4.1" class="ltx_text" style="background-color:#F6D1CD;">-1.8%</span></td>
<td id="S4.T4.1.1.9.8.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F6D1CE;"><span id="S4.T4.1.1.9.8.5.1" class="ltx_text" style="background-color:#F6D1CE;">-1.7%</span></td>
<td id="S4.T4.1.1.9.8.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EB9B94;"><span id="S4.T4.1.1.9.8.6.1" class="ltx_text" style="background-color:#EB9B94;">-3.8%</span></td>
<td id="S4.T4.1.1.9.8.7" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#EB968F;"><span id="S4.T4.1.1.9.8.7.1" class="ltx_text" style="background-color:#EB968F;">-4.0%</span></td>
<td id="S4.T4.1.1.9.8.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F0B0AB;"><span id="S4.T4.1.1.9.8.8.1" class="ltx_text" style="background-color:#F0B0AB;">-3.0%</span></td>
<td id="S4.T4.1.1.9.8.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C0BB;"><span id="S4.T4.1.1.9.8.9.1" class="ltx_text" style="background-color:#F3C0BB;">-2.4%</span></td>
<td id="S4.T4.1.1.9.8.10" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="background-color:#EB9891;"><span id="S4.T4.1.1.9.8.10.1" class="ltx_text" style="background-color:#EB9891;">-3.9%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span><span id="S4.T4.4.1" class="ltx_text ltx_font_bold">Per-language relative performance (%<math id="S4.T4.4.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S4.T4.4.1.m1.1b"><mi mathvariant="normal" id="S4.T4.4.1.m1.1.1" xref="S4.T4.4.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S4.T4.4.1.m1.1c"><ci id="S4.T4.4.1.m1.1.1.cmml" xref="S4.T4.4.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.4.1.m1.1d">\Delta</annotation></semantics></math>) vs. FP16, averaged over MGSM, mMMLU, FLORES, and Language Confusion tasks.</span> <span id="S4.T4.7.2" class="ltx_text ltx_font_bold">Ltn/IE</span> are Latin-script/Indo-European: de, es, fr. <math id="S4.T4.5.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T4.5.m1.1b"><mo id="S4.T4.5.m1.1.1" xref="S4.T4.5.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T4.5.m1.1c"><not id="S4.T4.5.m1.1.1.cmml" xref="S4.T4.5.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.5.m1.1d">\neg</annotation></semantics></math> are the rest: ja, zh.</figcaption>
</figure>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.2" class="ltx_p">On FLORES, the 103B Command model is more sensitive to quantization for the L2<math id="S4.SS2.p3.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.SS2.p3.1.m1.1a"><mo stretchy="false" id="S4.SS2.p3.1.m1.1.1" xref="S4.SS2.p3.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.1.m1.1b"><ci id="S4.SS2.p3.1.m1.1.1.cmml" xref="S4.SS2.p3.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.1.m1.1c">\rightarrow</annotation></semantics></math>En translation direction than for L2<math id="S4.SS2.p3.2.m2.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.SS2.p3.2.m2.1a"><mo stretchy="false" id="S4.SS2.p3.2.m2.1.1" xref="S4.SS2.p3.2.m2.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.SS2.p3.2.m2.1b"><ci id="S4.SS2.p3.2.m2.1.1.cmml" xref="S4.SS2.p3.2.m2.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p3.2.m2.1c">\rightarrow</annotation></semantics></math>En, though we see the opposite for the smaller 35B and Aya 23 models at <span id="S4.SS2.p3.2.1" class="ltx_text ltx_font_bold">W4</span>. Quantization does not noticeably impact unseen discriminative tasks (XWinograd, XCOPA, XStoryCloze: Table <a href="#A1.T17" title="Table A17 ‣ A.2 Automatic Tasks - Full Results ‣ Appendix A Appendix ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A17</span></a>).</p>
</div>
<div id="S4.SS2.p4" class="ltx_para">
<p id="S4.SS2.p4.1" class="ltx_p">Curiously, there are some fleeting performance boosts: an increase of 1.8–2.1% on MGSM and mild improvements on FLORES with <span id="S4.SS2.p4.1.1" class="ltx_text ltx_font_bold">W8</span> on Aya models, and a similar translation boost of the 35B Command model at <span id="S4.SS2.p4.1.2" class="ltx_text ltx_font_bold">W8A8</span>. Quantization generally has no effect or causes mild improvement on the monolingual language confusion task, and cross-lingual language confusion performance is boosted with greater quantization in some cases.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>By Language</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p"><span id="S4.SS3.p1.1.1" class="ltx_text ltx_font_italic">Are languages differently affected by quantization?</span></p>
</div>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.2" class="ltx_p">Table <a href="#S4.T3" title="Table 3 ‣ 4.2 By Task ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows performance averaged over mMMLU, FLORES, and Language Confusion tasks, with Table <a href="#S4.T4" title="Table 4 ‣ 4.2 By Task ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> further including MGSM for supported languages. Metrics are on different scales, so we average relative change (%<math id="S4.SS3.p2.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S4.SS3.p2.1.m1.1a"><mi mathvariant="normal" id="S4.SS3.p2.1.m1.1.1" xref="S4.SS3.p2.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.1.m1.1b"><ci id="S4.SS3.p2.1.m1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.1.m1.1c">\Delta</annotation></semantics></math>) rather than raw scores.<span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span>Ex. to arrive at <math id="footnote10.m1.1" class="ltx_Math" alttext="-1.3\%" display="inline"><semantics id="footnote10.m1.1b"><mrow id="footnote10.m1.1.1" xref="footnote10.m1.1.1.cmml"><mo id="footnote10.m1.1.1b" xref="footnote10.m1.1.1.cmml">−</mo><mrow id="footnote10.m1.1.1.2" xref="footnote10.m1.1.1.2.cmml"><mn id="footnote10.m1.1.1.2.2" xref="footnote10.m1.1.1.2.2.cmml">1.3</mn><mo id="footnote10.m1.1.1.2.1" xref="footnote10.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote10.m1.1c"><apply id="footnote10.m1.1.1.cmml" xref="footnote10.m1.1.1"><minus id="footnote10.m1.1.1.1.cmml" xref="footnote10.m1.1.1"></minus><apply id="footnote10.m1.1.1.2.cmml" xref="footnote10.m1.1.1.2"><csymbol cd="latexml" id="footnote10.m1.1.1.2.1.cmml" xref="footnote10.m1.1.1.2.1">percent</csymbol><cn type="float" id="footnote10.m1.1.1.2.2.cmml" xref="footnote10.m1.1.1.2.2">1.3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote10.m1.1d">-1.3\%</annotation></semantics></math> for 103B <span id="footnote10.1" class="ltx_text ltx_font_bold">W8A8</span> in Arabic, we average relative performance for mMMLU, FLORES En<math id="footnote10.m2.1" class="ltx_Math" alttext="\leftrightarrow" display="inline"><semantics id="footnote10.m2.1b"><mo stretchy="false" id="footnote10.m2.1.1" xref="footnote10.m2.1.1.cmml">↔</mo><annotation-xml encoding="MathML-Content" id="footnote10.m2.1c"><ci id="footnote10.m2.1.1.cmml" xref="footnote10.m2.1.1">↔</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote10.m2.1d">\leftrightarrow</annotation></semantics></math>L2, and Language Confusion tasks: <math id="footnote10.m3.1" class="ltx_Math" alttext="\text{Avg}(\{-2.2\%,-1.0\%,-1.3\%,0.0\%,-1.8\%\})=-1.3\%" display="inline"><semantics id="footnote10.m3.1b"><mrow id="footnote10.m3.1.1" xref="footnote10.m3.1.1.cmml"><mrow id="footnote10.m3.1.1.1" xref="footnote10.m3.1.1.1.cmml"><mtext id="footnote10.m3.1.1.1.3" xref="footnote10.m3.1.1.1.3a.cmml">Avg</mtext><mo lspace="0em" rspace="0em" id="footnote10.m3.1.1.1.2" xref="footnote10.m3.1.1.1.2.cmml">​</mo><mrow id="footnote10.m3.1.1.1.1.1" xref="footnote10.m3.1.1.1.cmml"><mo stretchy="false" id="footnote10.m3.1.1.1.1.1.2" xref="footnote10.m3.1.1.1.cmml">(</mo><mrow id="footnote10.m3.1.1.1.1.1.1.5" xref="footnote10.m3.1.1.1.1.1.1.6.cmml"><mo stretchy="false" id="footnote10.m3.1.1.1.1.1.1.5.6" xref="footnote10.m3.1.1.1.1.1.1.6.cmml">{</mo><mrow id="footnote10.m3.1.1.1.1.1.1.1.1" xref="footnote10.m3.1.1.1.1.1.1.1.1.cmml"><mo id="footnote10.m3.1.1.1.1.1.1.1.1b" xref="footnote10.m3.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="footnote10.m3.1.1.1.1.1.1.1.1.2" xref="footnote10.m3.1.1.1.1.1.1.1.1.2.cmml"><mn id="footnote10.m3.1.1.1.1.1.1.1.1.2.2" xref="footnote10.m3.1.1.1.1.1.1.1.1.2.2.cmml">2.2</mn><mo id="footnote10.m3.1.1.1.1.1.1.1.1.2.1" xref="footnote10.m3.1.1.1.1.1.1.1.1.2.1.cmml">%</mo></mrow></mrow><mo id="footnote10.m3.1.1.1.1.1.1.5.7" xref="footnote10.m3.1.1.1.1.1.1.6.cmml">,</mo><mrow id="footnote10.m3.1.1.1.1.1.1.2.2" xref="footnote10.m3.1.1.1.1.1.1.2.2.cmml"><mo id="footnote10.m3.1.1.1.1.1.1.2.2b" xref="footnote10.m3.1.1.1.1.1.1.2.2.cmml">−</mo><mrow id="footnote10.m3.1.1.1.1.1.1.2.2.2" xref="footnote10.m3.1.1.1.1.1.1.2.2.2.cmml"><mn id="footnote10.m3.1.1.1.1.1.1.2.2.2.2" xref="footnote10.m3.1.1.1.1.1.1.2.2.2.2.cmml">1.0</mn><mo id="footnote10.m3.1.1.1.1.1.1.2.2.2.1" xref="footnote10.m3.1.1.1.1.1.1.2.2.2.1.cmml">%</mo></mrow></mrow><mo id="footnote10.m3.1.1.1.1.1.1.5.8" xref="footnote10.m3.1.1.1.1.1.1.6.cmml">,</mo><mrow id="footnote10.m3.1.1.1.1.1.1.3.3" xref="footnote10.m3.1.1.1.1.1.1.3.3.cmml"><mo id="footnote10.m3.1.1.1.1.1.1.3.3b" xref="footnote10.m3.1.1.1.1.1.1.3.3.cmml">−</mo><mrow id="footnote10.m3.1.1.1.1.1.1.3.3.2" xref="footnote10.m3.1.1.1.1.1.1.3.3.2.cmml"><mn id="footnote10.m3.1.1.1.1.1.1.3.3.2.2" xref="footnote10.m3.1.1.1.1.1.1.3.3.2.2.cmml">1.3</mn><mo id="footnote10.m3.1.1.1.1.1.1.3.3.2.1" xref="footnote10.m3.1.1.1.1.1.1.3.3.2.1.cmml">%</mo></mrow></mrow><mo id="footnote10.m3.1.1.1.1.1.1.5.9" xref="footnote10.m3.1.1.1.1.1.1.6.cmml">,</mo><mrow id="footnote10.m3.1.1.1.1.1.1.4.4" xref="footnote10.m3.1.1.1.1.1.1.4.4.cmml"><mn id="footnote10.m3.1.1.1.1.1.1.4.4.2" xref="footnote10.m3.1.1.1.1.1.1.4.4.2.cmml">0.0</mn><mo id="footnote10.m3.1.1.1.1.1.1.4.4.1" xref="footnote10.m3.1.1.1.1.1.1.4.4.1.cmml">%</mo></mrow><mo id="footnote10.m3.1.1.1.1.1.1.5.10" xref="footnote10.m3.1.1.1.1.1.1.6.cmml">,</mo><mrow id="footnote10.m3.1.1.1.1.1.1.5.5" xref="footnote10.m3.1.1.1.1.1.1.5.5.cmml"><mo id="footnote10.m3.1.1.1.1.1.1.5.5b" xref="footnote10.m3.1.1.1.1.1.1.5.5.cmml">−</mo><mrow id="footnote10.m3.1.1.1.1.1.1.5.5.2" xref="footnote10.m3.1.1.1.1.1.1.5.5.2.cmml"><mn id="footnote10.m3.1.1.1.1.1.1.5.5.2.2" xref="footnote10.m3.1.1.1.1.1.1.5.5.2.2.cmml">1.8</mn><mo id="footnote10.m3.1.1.1.1.1.1.5.5.2.1" xref="footnote10.m3.1.1.1.1.1.1.5.5.2.1.cmml">%</mo></mrow></mrow><mo stretchy="false" id="footnote10.m3.1.1.1.1.1.1.5.11" xref="footnote10.m3.1.1.1.1.1.1.6.cmml">}</mo></mrow><mo stretchy="false" id="footnote10.m3.1.1.1.1.1.3" xref="footnote10.m3.1.1.1.cmml">)</mo></mrow></mrow><mo id="footnote10.m3.1.1.2" xref="footnote10.m3.1.1.2.cmml">=</mo><mrow id="footnote10.m3.1.1.3" xref="footnote10.m3.1.1.3.cmml"><mo id="footnote10.m3.1.1.3b" xref="footnote10.m3.1.1.3.cmml">−</mo><mrow id="footnote10.m3.1.1.3.2" xref="footnote10.m3.1.1.3.2.cmml"><mn id="footnote10.m3.1.1.3.2.2" xref="footnote10.m3.1.1.3.2.2.cmml">1.3</mn><mo id="footnote10.m3.1.1.3.2.1" xref="footnote10.m3.1.1.3.2.1.cmml">%</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote10.m3.1c"><apply id="footnote10.m3.1.1.cmml" xref="footnote10.m3.1.1"><eq id="footnote10.m3.1.1.2.cmml" xref="footnote10.m3.1.1.2"></eq><apply id="footnote10.m3.1.1.1.cmml" xref="footnote10.m3.1.1.1"><times id="footnote10.m3.1.1.1.2.cmml" xref="footnote10.m3.1.1.1.2"></times><ci id="footnote10.m3.1.1.1.3a.cmml" xref="footnote10.m3.1.1.1.3"><mtext id="footnote10.m3.1.1.1.3.cmml" xref="footnote10.m3.1.1.1.3">Avg</mtext></ci><set id="footnote10.m3.1.1.1.1.1.1.6.cmml" xref="footnote10.m3.1.1.1.1.1.1.5"><apply id="footnote10.m3.1.1.1.1.1.1.1.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.1.1"><minus id="footnote10.m3.1.1.1.1.1.1.1.1.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.1.1"></minus><apply id="footnote10.m3.1.1.1.1.1.1.1.1.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.1.1.2"><csymbol cd="latexml" id="footnote10.m3.1.1.1.1.1.1.1.1.2.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.1.1.2.1">percent</csymbol><cn type="float" id="footnote10.m3.1.1.1.1.1.1.1.1.2.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.1.1.2.2">2.2</cn></apply></apply><apply id="footnote10.m3.1.1.1.1.1.1.2.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.2.2"><minus id="footnote10.m3.1.1.1.1.1.1.2.2.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.2.2"></minus><apply id="footnote10.m3.1.1.1.1.1.1.2.2.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.2.2.2"><csymbol cd="latexml" id="footnote10.m3.1.1.1.1.1.1.2.2.2.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.2.2.2.1">percent</csymbol><cn type="float" id="footnote10.m3.1.1.1.1.1.1.2.2.2.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.2.2.2.2">1.0</cn></apply></apply><apply id="footnote10.m3.1.1.1.1.1.1.3.3.cmml" xref="footnote10.m3.1.1.1.1.1.1.3.3"><minus id="footnote10.m3.1.1.1.1.1.1.3.3.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.3.3"></minus><apply id="footnote10.m3.1.1.1.1.1.1.3.3.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.3.3.2"><csymbol cd="latexml" id="footnote10.m3.1.1.1.1.1.1.3.3.2.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.3.3.2.1">percent</csymbol><cn type="float" id="footnote10.m3.1.1.1.1.1.1.3.3.2.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.3.3.2.2">1.3</cn></apply></apply><apply id="footnote10.m3.1.1.1.1.1.1.4.4.cmml" xref="footnote10.m3.1.1.1.1.1.1.4.4"><csymbol cd="latexml" id="footnote10.m3.1.1.1.1.1.1.4.4.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.4.4.1">percent</csymbol><cn type="float" id="footnote10.m3.1.1.1.1.1.1.4.4.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.4.4.2">0.0</cn></apply><apply id="footnote10.m3.1.1.1.1.1.1.5.5.cmml" xref="footnote10.m3.1.1.1.1.1.1.5.5"><minus id="footnote10.m3.1.1.1.1.1.1.5.5.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.5.5"></minus><apply id="footnote10.m3.1.1.1.1.1.1.5.5.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.5.5.2"><csymbol cd="latexml" id="footnote10.m3.1.1.1.1.1.1.5.5.2.1.cmml" xref="footnote10.m3.1.1.1.1.1.1.5.5.2.1">percent</csymbol><cn type="float" id="footnote10.m3.1.1.1.1.1.1.5.5.2.2.cmml" xref="footnote10.m3.1.1.1.1.1.1.5.5.2.2">1.8</cn></apply></apply></set></apply><apply id="footnote10.m3.1.1.3.cmml" xref="footnote10.m3.1.1.3"><minus id="footnote10.m3.1.1.3.1.cmml" xref="footnote10.m3.1.1.3"></minus><apply id="footnote10.m3.1.1.3.2.cmml" xref="footnote10.m3.1.1.3.2"><csymbol cd="latexml" id="footnote10.m3.1.1.3.2.1.cmml" xref="footnote10.m3.1.1.3.2.1">percent</csymbol><cn type="float" id="footnote10.m3.1.1.3.2.2.cmml" xref="footnote10.m3.1.1.3.2.2">1.3</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote10.m3.1d">\text{Avg}(\{-2.2\%,-1.0\%,-1.3\%,0.0\%,-1.8\%\})=-1.3\%</annotation></semantics></math>.</span></span></span> We further separate into languages written in the Latin/Roman script (which also are the subset of Indo-European languages; <span id="S4.SS3.p2.2.1" class="ltx_text ltx_font_bold">Ltn/IE</span>) versus the rest (<math id="S4.SS3.p2.2.m2.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.SS3.p2.2.m2.1a"><mo id="S4.SS3.p2.2.m2.1.1" xref="S4.SS3.p2.2.m2.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.2.m2.1b"><not id="S4.SS3.p2.2.m2.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.2.m2.1c">\neg</annotation></semantics></math><span id="S4.SS3.p2.2.2" class="ltx_text ltx_font_bold">Ltn/IE</span>).</p>
</div>
<div id="S4.SS3.p3" class="ltx_para">
<p id="S4.SS3.p3.1" class="ltx_p"><span id="S4.SS3.p3.1.1" class="ltx_text ltx_font_bold">W4-g</span> causes considerable degradation across languages for the 35B Command model. A relationship between language and performance is apparent: <math id="S4.SS3.p3.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.SS3.p3.1.m1.1a"><mo id="S4.SS3.p3.1.m1.1.1" xref="S4.SS3.p3.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.1.m1.1b"><not id="S4.SS3.p3.1.m1.1.1.cmml" xref="S4.SS3.p3.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.1.m1.1c">\neg</annotation></semantics></math><span id="S4.SS3.p3.1.2" class="ltx_text ltx_font_bold">Ltn/IE</span> languages typically degrade more. Chinese, Japanese, and Korean are particularly harmed by <span id="S4.SS3.p3.1.3" class="ltx_text ltx_font_bold">W4</span> quantization of the 103B. The effect is seen consistently across all automatic metrics for Command, with limited exception. Table <a href="#S4.T5" title="Table 5 ‣ 4.5 By Quantization Strategy ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> is discussed more thoroughly in Section <a href="#S4.SS5" title="4.5 By Quantization Strategy ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.5</span></a>, but also shows this discrepancy. In the Appendix, we see the same for Aya 23 models at <span id="S4.SS3.p3.1.4" class="ltx_text ltx_font_bold">W4</span>.</p>
</div>
<div id="S4.SS3.p4" class="ltx_para">
<p id="S4.SS3.p4.1" class="ltx_p">Interestingly, <span id="S4.SS3.p4.1.1" class="ltx_text ltx_font_bold">W8A8</span> of the 35B Command model <span id="S4.SS3.p4.1.2" class="ltx_text ltx_font_italic">helps</span> on average across all languages. The magnitude is primarily due to an increase on cross-lingual language confusion. <span id="S4.SS3.p4.1.3" class="ltx_text ltx_font_bold">W8</span> also aids Aya 23 on MGSM (Table <a href="#A1.T6" title="Table A6 ‣ A.2 Automatic Tasks - Full Results ‣ Appendix A Appendix ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A6</span></a>) for <math id="S4.SS3.p4.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.SS3.p4.1.m1.1a"><mo id="S4.SS3.p4.1.m1.1.1" xref="S4.SS3.p4.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p4.1.m1.1b"><not id="S4.SS3.p4.1.m1.1.1.cmml" xref="S4.SS3.p4.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p4.1.m1.1c">\neg</annotation></semantics></math><span id="S4.SS3.p4.1.4" class="ltx_text ltx_font_bold">Ltn/IE</span> languages, and across languages on FLORES (Table <a href="#A1.T16" title="Table A16 ‣ A.2 Automatic Tasks - Full Results ‣ Appendix A Appendix ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A16</span></a>).</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>By Model Size</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p"><span id="S4.SS4.p1.1.1" class="ltx_text ltx_font_italic">How do model size and quantization level interact?</span></p>
</div>
<div id="S4.SS4.p2" class="ltx_para">
<p id="S4.SS4.p2.8" class="ltx_p">Across evaluations at the most extreme quantization (<span id="S4.SS4.p2.8.1" class="ltx_text ltx_font_bold">W4</span>/<span id="S4.SS4.p2.8.2" class="ltx_text ltx_font_bold">W4-g</span>), smaller models are more sensitive: <span id="S4.SS4.p2.8.3" class="ltx_text ltx_font_bold">W4-g</span> variants of 103B and 35B Command record <math id="S4.SS4.p2.1.m1.1" class="ltx_Math" alttext="-0.9\%" display="inline"><semantics id="S4.SS4.p2.1.m1.1a"><mrow id="S4.SS4.p2.1.m1.1.1" xref="S4.SS4.p2.1.m1.1.1.cmml"><mo id="S4.SS4.p2.1.m1.1.1a" xref="S4.SS4.p2.1.m1.1.1.cmml">−</mo><mrow id="S4.SS4.p2.1.m1.1.1.2" xref="S4.SS4.p2.1.m1.1.1.2.cmml"><mn id="S4.SS4.p2.1.m1.1.1.2.2" xref="S4.SS4.p2.1.m1.1.1.2.2.cmml">0.9</mn><mo id="S4.SS4.p2.1.m1.1.1.2.1" xref="S4.SS4.p2.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.1.m1.1b"><apply id="S4.SS4.p2.1.m1.1.1.cmml" xref="S4.SS4.p2.1.m1.1.1"><minus id="S4.SS4.p2.1.m1.1.1.1.cmml" xref="S4.SS4.p2.1.m1.1.1"></minus><apply id="S4.SS4.p2.1.m1.1.1.2.cmml" xref="S4.SS4.p2.1.m1.1.1.2"><csymbol cd="latexml" id="S4.SS4.p2.1.m1.1.1.2.1.cmml" xref="S4.SS4.p2.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS4.p2.1.m1.1.1.2.2.cmml" xref="S4.SS4.p2.1.m1.1.1.2.2">0.9</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.1.m1.1c">-0.9\%</annotation></semantics></math> and <math id="S4.SS4.p2.2.m2.1" class="ltx_Math" alttext="-2.8\%" display="inline"><semantics id="S4.SS4.p2.2.m2.1a"><mrow id="S4.SS4.p2.2.m2.1.1" xref="S4.SS4.p2.2.m2.1.1.cmml"><mo id="S4.SS4.p2.2.m2.1.1a" xref="S4.SS4.p2.2.m2.1.1.cmml">−</mo><mrow id="S4.SS4.p2.2.m2.1.1.2" xref="S4.SS4.p2.2.m2.1.1.2.cmml"><mn id="S4.SS4.p2.2.m2.1.1.2.2" xref="S4.SS4.p2.2.m2.1.1.2.2.cmml">2.8</mn><mo id="S4.SS4.p2.2.m2.1.1.2.1" xref="S4.SS4.p2.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.2.m2.1b"><apply id="S4.SS4.p2.2.m2.1.1.cmml" xref="S4.SS4.p2.2.m2.1.1"><minus id="S4.SS4.p2.2.m2.1.1.1.cmml" xref="S4.SS4.p2.2.m2.1.1"></minus><apply id="S4.SS4.p2.2.m2.1.1.2.cmml" xref="S4.SS4.p2.2.m2.1.1.2"><csymbol cd="latexml" id="S4.SS4.p2.2.m2.1.1.2.1.cmml" xref="S4.SS4.p2.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS4.p2.2.m2.1.1.2.2.cmml" xref="S4.SS4.p2.2.m2.1.1.2.2">2.8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.2.m2.1c">-2.8\%</annotation></semantics></math> performance relative to FP16 on average, with a stark difference of <math id="S4.SS4.p2.3.m3.1" class="ltx_Math" alttext="-2.9\%" display="inline"><semantics id="S4.SS4.p2.3.m3.1a"><mrow id="S4.SS4.p2.3.m3.1.1" xref="S4.SS4.p2.3.m3.1.1.cmml"><mo id="S4.SS4.p2.3.m3.1.1a" xref="S4.SS4.p2.3.m3.1.1.cmml">−</mo><mrow id="S4.SS4.p2.3.m3.1.1.2" xref="S4.SS4.p2.3.m3.1.1.2.cmml"><mn id="S4.SS4.p2.3.m3.1.1.2.2" xref="S4.SS4.p2.3.m3.1.1.2.2.cmml">2.9</mn><mo id="S4.SS4.p2.3.m3.1.1.2.1" xref="S4.SS4.p2.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.3.m3.1b"><apply id="S4.SS4.p2.3.m3.1.1.cmml" xref="S4.SS4.p2.3.m3.1.1"><minus id="S4.SS4.p2.3.m3.1.1.1.cmml" xref="S4.SS4.p2.3.m3.1.1"></minus><apply id="S4.SS4.p2.3.m3.1.1.2.cmml" xref="S4.SS4.p2.3.m3.1.1.2"><csymbol cd="latexml" id="S4.SS4.p2.3.m3.1.1.2.1.cmml" xref="S4.SS4.p2.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS4.p2.3.m3.1.1.2.2.cmml" xref="S4.SS4.p2.3.m3.1.1.2.2">2.9</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.3.m3.1c">-2.9\%</annotation></semantics></math> vs. <math id="S4.SS4.p2.4.m4.1" class="ltx_Math" alttext="-13.1\%" display="inline"><semantics id="S4.SS4.p2.4.m4.1a"><mrow id="S4.SS4.p2.4.m4.1.1" xref="S4.SS4.p2.4.m4.1.1.cmml"><mo id="S4.SS4.p2.4.m4.1.1a" xref="S4.SS4.p2.4.m4.1.1.cmml">−</mo><mrow id="S4.SS4.p2.4.m4.1.1.2" xref="S4.SS4.p2.4.m4.1.1.2.cmml"><mn id="S4.SS4.p2.4.m4.1.1.2.2" xref="S4.SS4.p2.4.m4.1.1.2.2.cmml">13.1</mn><mo id="S4.SS4.p2.4.m4.1.1.2.1" xref="S4.SS4.p2.4.m4.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.4.m4.1b"><apply id="S4.SS4.p2.4.m4.1.1.cmml" xref="S4.SS4.p2.4.m4.1.1"><minus id="S4.SS4.p2.4.m4.1.1.1.cmml" xref="S4.SS4.p2.4.m4.1.1"></minus><apply id="S4.SS4.p2.4.m4.1.1.2.cmml" xref="S4.SS4.p2.4.m4.1.1.2"><csymbol cd="latexml" id="S4.SS4.p2.4.m4.1.1.2.1.cmml" xref="S4.SS4.p2.4.m4.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS4.p2.4.m4.1.1.2.2.cmml" xref="S4.SS4.p2.4.m4.1.1.2.2">13.1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.4.m4.1c">-13.1\%</annotation></semantics></math> on MGSM. Aya 23 35B/8B record <math id="S4.SS4.p2.5.m5.1" class="ltx_Math" alttext="-2.9\%" display="inline"><semantics id="S4.SS4.p2.5.m5.1a"><mrow id="S4.SS4.p2.5.m5.1.1" xref="S4.SS4.p2.5.m5.1.1.cmml"><mo id="S4.SS4.p2.5.m5.1.1a" xref="S4.SS4.p2.5.m5.1.1.cmml">−</mo><mrow id="S4.SS4.p2.5.m5.1.1.2" xref="S4.SS4.p2.5.m5.1.1.2.cmml"><mn id="S4.SS4.p2.5.m5.1.1.2.2" xref="S4.SS4.p2.5.m5.1.1.2.2.cmml">2.9</mn><mo id="S4.SS4.p2.5.m5.1.1.2.1" xref="S4.SS4.p2.5.m5.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.5.m5.1b"><apply id="S4.SS4.p2.5.m5.1.1.cmml" xref="S4.SS4.p2.5.m5.1.1"><minus id="S4.SS4.p2.5.m5.1.1.1.cmml" xref="S4.SS4.p2.5.m5.1.1"></minus><apply id="S4.SS4.p2.5.m5.1.1.2.cmml" xref="S4.SS4.p2.5.m5.1.1.2"><csymbol cd="latexml" id="S4.SS4.p2.5.m5.1.1.2.1.cmml" xref="S4.SS4.p2.5.m5.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS4.p2.5.m5.1.1.2.2.cmml" xref="S4.SS4.p2.5.m5.1.1.2.2">2.9</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.5.m5.1c">-2.9\%</annotation></semantics></math> vs. <math id="S4.SS4.p2.6.m6.1" class="ltx_Math" alttext="-3.7\%" display="inline"><semantics id="S4.SS4.p2.6.m6.1a"><mrow id="S4.SS4.p2.6.m6.1.1" xref="S4.SS4.p2.6.m6.1.1.cmml"><mo id="S4.SS4.p2.6.m6.1.1a" xref="S4.SS4.p2.6.m6.1.1.cmml">−</mo><mrow id="S4.SS4.p2.6.m6.1.1.2" xref="S4.SS4.p2.6.m6.1.1.2.cmml"><mn id="S4.SS4.p2.6.m6.1.1.2.2" xref="S4.SS4.p2.6.m6.1.1.2.2.cmml">3.7</mn><mo id="S4.SS4.p2.6.m6.1.1.2.1" xref="S4.SS4.p2.6.m6.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.6.m6.1b"><apply id="S4.SS4.p2.6.m6.1.1.cmml" xref="S4.SS4.p2.6.m6.1.1"><minus id="S4.SS4.p2.6.m6.1.1.1.cmml" xref="S4.SS4.p2.6.m6.1.1"></minus><apply id="S4.SS4.p2.6.m6.1.1.2.cmml" xref="S4.SS4.p2.6.m6.1.1.2"><csymbol cd="latexml" id="S4.SS4.p2.6.m6.1.1.2.1.cmml" xref="S4.SS4.p2.6.m6.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS4.p2.6.m6.1.1.2.2.cmml" xref="S4.SS4.p2.6.m6.1.1.2.2">3.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.6.m6.1c">-3.7\%</annotation></semantics></math> on average, with their largest gap occurring in Belebele (<math id="S4.SS4.p2.7.m7.1" class="ltx_Math" alttext="-5.9\%" display="inline"><semantics id="S4.SS4.p2.7.m7.1a"><mrow id="S4.SS4.p2.7.m7.1.1" xref="S4.SS4.p2.7.m7.1.1.cmml"><mo id="S4.SS4.p2.7.m7.1.1a" xref="S4.SS4.p2.7.m7.1.1.cmml">−</mo><mrow id="S4.SS4.p2.7.m7.1.1.2" xref="S4.SS4.p2.7.m7.1.1.2.cmml"><mn id="S4.SS4.p2.7.m7.1.1.2.2" xref="S4.SS4.p2.7.m7.1.1.2.2.cmml">5.9</mn><mo id="S4.SS4.p2.7.m7.1.1.2.1" xref="S4.SS4.p2.7.m7.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.7.m7.1b"><apply id="S4.SS4.p2.7.m7.1.1.cmml" xref="S4.SS4.p2.7.m7.1.1"><minus id="S4.SS4.p2.7.m7.1.1.1.cmml" xref="S4.SS4.p2.7.m7.1.1"></minus><apply id="S4.SS4.p2.7.m7.1.1.2.cmml" xref="S4.SS4.p2.7.m7.1.1.2"><csymbol cd="latexml" id="S4.SS4.p2.7.m7.1.1.2.1.cmml" xref="S4.SS4.p2.7.m7.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS4.p2.7.m7.1.1.2.2.cmml" xref="S4.SS4.p2.7.m7.1.1.2.2">5.9</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.7.m7.1c">-5.9\%</annotation></semantics></math> vs. <math id="S4.SS4.p2.8.m8.1" class="ltx_Math" alttext="-8.5\%" display="inline"><semantics id="S4.SS4.p2.8.m8.1a"><mrow id="S4.SS4.p2.8.m8.1.1" xref="S4.SS4.p2.8.m8.1.1.cmml"><mo id="S4.SS4.p2.8.m8.1.1a" xref="S4.SS4.p2.8.m8.1.1.cmml">−</mo><mrow id="S4.SS4.p2.8.m8.1.1.2" xref="S4.SS4.p2.8.m8.1.1.2.cmml"><mn id="S4.SS4.p2.8.m8.1.1.2.2" xref="S4.SS4.p2.8.m8.1.1.2.2.cmml">8.5</mn><mo id="S4.SS4.p2.8.m8.1.1.2.1" xref="S4.SS4.p2.8.m8.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p2.8.m8.1b"><apply id="S4.SS4.p2.8.m8.1.1.cmml" xref="S4.SS4.p2.8.m8.1.1"><minus id="S4.SS4.p2.8.m8.1.1.1.cmml" xref="S4.SS4.p2.8.m8.1.1"></minus><apply id="S4.SS4.p2.8.m8.1.1.2.cmml" xref="S4.SS4.p2.8.m8.1.1.2"><csymbol cd="latexml" id="S4.SS4.p2.8.m8.1.1.2.1.cmml" xref="S4.SS4.p2.8.m8.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS4.p2.8.m8.1.1.2.2.cmml" xref="S4.SS4.p2.8.m8.1.1.2.2">8.5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p2.8.m8.1c">-8.5\%</annotation></semantics></math>). (Refer back to Tables <a href="#S4.T1" title="Table 1 ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and <a href="#S4.T2" title="Table 2 ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.)</p>
</div>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>By Quantization Strategy</h3>

<div id="S4.SS5.p1" class="ltx_para">
<p id="S4.SS5.p1.1" class="ltx_p"><span id="S4.SS5.p1.1.1" class="ltx_text ltx_font_italic">How do techniques like SmoothQuant and group-wise scaling affect downstream performance?</span></p>
</div>
<figure id="S4.T5" class="ltx_table">
<div id="S4.T5.9" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:85.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-103.6pt,20.4pt) scale(0.676590456199741,0.676590456199741) ;">
<table id="S4.T5.9.9" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T5.9.9.10.1" class="ltx_tr">
<th id="S4.T5.9.9.10.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_r ltx_border_tt"></th>
<td id="S4.T5.9.9.10.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S4.T5.9.9.10.1.3" class="ltx_td ltx_border_r ltx_border_tt"></td>
<td id="S4.T5.9.9.10.1.4" class="ltx_td ltx_border_tt"></td>
<td id="S4.T5.9.9.10.1.5" class="ltx_td ltx_border_r ltx_border_tt"></td>
<td id="S4.T5.9.9.10.1.6" class="ltx_td ltx_border_tt"></td>
<td id="S4.T5.9.9.10.1.7" class="ltx_td ltx_border_r ltx_border_tt"></td>
<td id="S4.T5.9.9.10.1.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="4"><span id="S4.T5.9.9.10.1.8.1" class="ltx_text ltx_font_bold">FLORES</span></td>
<td id="S4.T5.9.9.10.1.9" class="ltx_td ltx_align_center ltx_border_tt" colspan="4"><span id="S4.T5.9.9.10.1.9.1" class="ltx_text ltx_font_bold">Language Confusion</span></td>
</tr>
<tr id="S4.T5.2.2.2" class="ltx_tr">
<th id="S4.T5.2.2.2.3" class="ltx_td ltx_th ltx_th_row ltx_border_r"></th>
<td id="S4.T5.2.2.2.4" class="ltx_td ltx_align_center ltx_border_r" colspan="2"><span id="S4.T5.2.2.2.4.1" class="ltx_text ltx_font_bold">Avg. Rel. %</span></td>
<td id="S4.T5.2.2.2.5" class="ltx_td ltx_align_center ltx_border_r" colspan="2"><span id="S4.T5.2.2.2.5.1" class="ltx_text ltx_font_bold">mMMLU</span></td>
<td id="S4.T5.2.2.2.6" class="ltx_td ltx_align_center ltx_border_r" colspan="2"><span id="S4.T5.2.2.2.6.1" class="ltx_text ltx_font_bold">MGSM</span></td>
<td id="S4.T5.1.1.1.1" class="ltx_td ltx_align_center ltx_border_r" colspan="2"><span id="S4.T5.1.1.1.1.1" class="ltx_text ltx_font_bold">En <math id="S4.T5.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T5.1.1.1.1.1.m1.1a"><mo stretchy="false" id="S4.T5.1.1.1.1.1.m1.1.1" xref="S4.T5.1.1.1.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T5.1.1.1.1.1.m1.1b"><ci id="S4.T5.1.1.1.1.1.m1.1.1.cmml" xref="S4.T5.1.1.1.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.1.1.1.1.1.m1.1c">\rightarrow</annotation></semantics></math> L2</span></td>
<td id="S4.T5.2.2.2.2" class="ltx_td ltx_align_center ltx_border_r" colspan="2"><span id="S4.T5.2.2.2.2.1" class="ltx_text ltx_font_bold">L2 <math id="S4.T5.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T5.2.2.2.2.1.m1.1a"><mo stretchy="false" id="S4.T5.2.2.2.2.1.m1.1.1" xref="S4.T5.2.2.2.2.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T5.2.2.2.2.1.m1.1b"><ci id="S4.T5.2.2.2.2.1.m1.1.1.cmml" xref="S4.T5.2.2.2.2.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.2.2.2.2.1.m1.1c">\rightarrow</annotation></semantics></math> En</span></td>
<td id="S4.T5.2.2.2.7" class="ltx_td ltx_align_center ltx_border_r" colspan="2"><span id="S4.T5.2.2.2.7.1" class="ltx_text ltx_font_bold">Monolingual</span></td>
<td id="S4.T5.2.2.2.8" class="ltx_td ltx_align_center" colspan="2"><span id="S4.T5.2.2.2.8.1" class="ltx_text ltx_font_bold">Cross-lingual</span></td>
</tr>
<tr id="S4.T5.9.9.9" class="ltx_tr">
<th id="S4.T5.9.9.9.8" class="ltx_td ltx_th ltx_th_row ltx_border_r"></th>
<td id="S4.T5.9.9.9.9" class="ltx_td ltx_align_left">Ltn/IE</td>
<td id="S4.T5.3.3.3.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T5.3.3.3.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T5.3.3.3.1.m1.1a"><mo id="S4.T5.3.3.3.1.m1.1.1" xref="S4.T5.3.3.3.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T5.3.3.3.1.m1.1b"><not id="S4.T5.3.3.3.1.m1.1.1.cmml" xref="S4.T5.3.3.3.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.3.3.3.1.m1.1c">\neg</annotation></semantics></math></td>
<td id="S4.T5.9.9.9.10" class="ltx_td ltx_align_left">Ltn/IE</td>
<td id="S4.T5.4.4.4.2" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T5.4.4.4.2.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T5.4.4.4.2.m1.1a"><mo id="S4.T5.4.4.4.2.m1.1.1" xref="S4.T5.4.4.4.2.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T5.4.4.4.2.m1.1b"><not id="S4.T5.4.4.4.2.m1.1.1.cmml" xref="S4.T5.4.4.4.2.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.4.4.4.2.m1.1c">\neg</annotation></semantics></math></td>
<td id="S4.T5.9.9.9.11" class="ltx_td ltx_align_left">Ltn/IE</td>
<td id="S4.T5.5.5.5.3" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T5.5.5.5.3.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T5.5.5.5.3.m1.1a"><mo id="S4.T5.5.5.5.3.m1.1.1" xref="S4.T5.5.5.5.3.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T5.5.5.5.3.m1.1b"><not id="S4.T5.5.5.5.3.m1.1.1.cmml" xref="S4.T5.5.5.5.3.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.5.5.5.3.m1.1c">\neg</annotation></semantics></math></td>
<td id="S4.T5.9.9.9.12" class="ltx_td ltx_align_left">Ltn/IE</td>
<td id="S4.T5.6.6.6.4" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T5.6.6.6.4.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T5.6.6.6.4.m1.1a"><mo id="S4.T5.6.6.6.4.m1.1.1" xref="S4.T5.6.6.6.4.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T5.6.6.6.4.m1.1b"><not id="S4.T5.6.6.6.4.m1.1.1.cmml" xref="S4.T5.6.6.6.4.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.6.6.6.4.m1.1c">\neg</annotation></semantics></math></td>
<td id="S4.T5.9.9.9.13" class="ltx_td ltx_align_left">Ltn/IE</td>
<td id="S4.T5.7.7.7.5" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T5.7.7.7.5.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T5.7.7.7.5.m1.1a"><mo id="S4.T5.7.7.7.5.m1.1.1" xref="S4.T5.7.7.7.5.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T5.7.7.7.5.m1.1b"><not id="S4.T5.7.7.7.5.m1.1.1.cmml" xref="S4.T5.7.7.7.5.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.7.7.7.5.m1.1c">\neg</annotation></semantics></math></td>
<td id="S4.T5.9.9.9.14" class="ltx_td ltx_align_left">Ltn/IE</td>
<td id="S4.T5.8.8.8.6" class="ltx_td ltx_align_center ltx_border_r"><math id="S4.T5.8.8.8.6.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T5.8.8.8.6.m1.1a"><mo id="S4.T5.8.8.8.6.m1.1.1" xref="S4.T5.8.8.8.6.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T5.8.8.8.6.m1.1b"><not id="S4.T5.8.8.8.6.m1.1.1.cmml" xref="S4.T5.8.8.8.6.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.8.8.8.6.m1.1c">\neg</annotation></semantics></math></td>
<td id="S4.T5.9.9.9.15" class="ltx_td ltx_align_left">Ltn/IE</td>
<td id="S4.T5.9.9.9.7" class="ltx_td ltx_align_center"><math id="S4.T5.9.9.9.7.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T5.9.9.9.7.m1.1a"><mo id="S4.T5.9.9.9.7.m1.1.1" xref="S4.T5.9.9.9.7.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T5.9.9.9.7.m1.1b"><not id="S4.T5.9.9.9.7.m1.1.1.cmml" xref="S4.T5.9.9.9.7.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.9.9.9.7.m1.1c">\neg</annotation></semantics></math></td>
</tr>
<tr id="S4.T5.9.9.11.2" class="ltx_tr">
<th id="S4.T5.9.9.11.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">W8A8</th>
<td id="S4.T5.9.9.11.2.2" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBECEB;"><span id="S4.T5.9.9.11.2.2.1" class="ltx_text" style="background-color:#FBECEB;">-0.7%</span></td>
<td id="S4.T5.9.9.11.2.3" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F9E4E2;"><span id="S4.T5.9.9.11.2.3.1" class="ltx_text" style="background-color:#F9E4E2;">-1.0%</span></td>
<td id="S4.T5.9.9.11.2.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8DBD9;"><span id="S4.T5.9.9.11.2.4.1" class="ltx_text" style="background-color:#F8DBD9;">-1.3%</span></td>
<td id="S4.T5.9.9.11.2.5" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F4C8C4;"><span id="S4.T5.9.9.11.2.5.1" class="ltx_text" style="background-color:#F4C8C4;">-2.1%</span></td>
<td id="S4.T5.9.9.11.2.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE8E6;"><span id="S4.T5.9.9.11.2.6.1" class="ltx_text" style="background-color:#FAE8E6;">-0.9%</span></td>
<td id="S4.T5.9.9.11.2.7" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F8DDDB;"><span id="S4.T5.9.9.11.2.7.1" class="ltx_text" style="background-color:#F8DDDB;">-1.3%</span></td>
<td id="S4.T5.9.9.11.2.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="S4.T5.9.9.11.2.8.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="S4.T5.9.9.11.2.9" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FCFEFD;"><span id="S4.T5.9.9.11.2.9.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="S4.T5.9.9.11.2.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F9E4E2;"><span id="S4.T5.9.9.11.2.10.1" class="ltx_text" style="background-color:#F9E4E2;">-1.0%</span></td>
<td id="S4.T5.9.9.11.2.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F7D5D2;"><span id="S4.T5.9.9.11.2.11.1" class="ltx_text" style="background-color:#F7D5D2;">-1.6%</span></td>
<td id="S4.T5.9.9.11.2.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFFFF;"><span id="S4.T5.9.9.11.2.12.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
<td id="S4.T5.9.9.11.2.13" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F6FCF9;"><span id="S4.T5.9.9.11.2.13.1" class="ltx_text" style="background-color:#F6FCF9;">0.4%</span></td>
<td id="S4.T5.9.9.11.2.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE6E4;"><span id="S4.T5.9.9.11.2.14.1" class="ltx_text" style="background-color:#FAE6E4;">-0.9%</span></td>
<td id="S4.T5.9.9.11.2.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F7D6D3;"><span id="S4.T5.9.9.11.2.15.1" class="ltx_text" style="background-color:#F7D6D3;">-1.6%</span></td>
</tr>
<tr id="S4.T5.9.9.12.3" class="ltx_tr">
<th id="S4.T5.9.9.12.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="S4.T5.9.9.12.3.2" class="ltx_td ltx_align_right" style="background-color:#FCF3F2;"><span id="S4.T5.9.9.12.3.2.1" class="ltx_text" style="background-color:#FCF3F2;">-0.4%</span></td>
<td id="S4.T5.9.9.12.3.3" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FBECEB;"><span id="S4.T5.9.9.12.3.3.1" class="ltx_text" style="background-color:#FBECEB;">-0.7%</span></td>
<td id="S4.T5.9.9.12.3.4" class="ltx_td ltx_align_right" style="background-color:#FDF5F5;"><span id="S4.T5.9.9.12.3.4.1" class="ltx_text" style="background-color:#FDF5F5;">-0.4%</span></td>
<td id="S4.T5.9.9.12.3.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FBEAE9;"><span id="S4.T5.9.9.12.3.5.1" class="ltx_text" style="background-color:#FBEAE9;">-0.8%</span></td>
<td id="S4.T5.9.9.12.3.6" class="ltx_td ltx_align_right" style="background-color:#F8DBD9;"><span id="S4.T5.9.9.12.3.6.1" class="ltx_text" style="background-color:#F8DBD9;">-1.3%</span></td>
<td id="S4.T5.9.9.12.3.7" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F5CDCA;"><span id="S4.T5.9.9.12.3.7.1" class="ltx_text" style="background-color:#F5CDCA;">-1.9%</span></td>
<td id="S4.T5.9.9.12.3.8" class="ltx_td ltx_align_right" style="background-color:#FAFDFB;"><span id="S4.T5.9.9.12.3.8.1" class="ltx_text" style="background-color:#FAFDFB;">0.2%</span></td>
<td id="S4.T5.9.9.12.3.9" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFEFD;"><span id="S4.T5.9.9.12.3.9.1" class="ltx_text" style="background-color:#FEFEFD;">0.0%</span></td>
<td id="S4.T5.9.9.12.3.10" class="ltx_td ltx_align_right" style="background-color:#F9E2E0;"><span id="S4.T5.9.9.12.3.10.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="S4.T5.9.9.12.3.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F6D4D1;"><span id="S4.T5.9.9.12.3.11.1" class="ltx_text" style="background-color:#F6D4D1;">-1.6%</span></td>
<td id="S4.T5.9.9.12.3.12" class="ltx_td ltx_align_right" style="background-color:#FEFCFB;"><span id="S4.T5.9.9.12.3.12.1" class="ltx_text" style="background-color:#FEFCFB;">-0.1%</span></td>
<td id="S4.T5.9.9.12.3.13" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FCFEFD;"><span id="S4.T5.9.9.12.3.13.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="S4.T5.9.9.12.3.14" class="ltx_td ltx_align_right" style="background-color:#FEFFFE;"><span id="S4.T5.9.9.12.3.14.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="S4.T5.9.9.12.3.15" class="ltx_td ltx_align_right" style="background-color:#FEFFFF;"><span id="S4.T5.9.9.12.3.15.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
</tr>
<tr id="S4.T5.9.9.13.4" class="ltx_tr">
<th id="S4.T5.9.9.13.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">W4</th>
<td id="S4.T5.9.9.13.4.2" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F5CDCA;"><span id="S4.T5.9.9.13.4.2.1" class="ltx_text" style="background-color:#F5CDCA;">-1.9%</span></td>
<td id="S4.T5.9.9.13.4.3" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#EEA8A2;"><span id="S4.T5.9.9.13.4.3.1" class="ltx_text" style="background-color:#EEA8A2;">-3.3%</span></td>
<td id="S4.T5.9.9.13.4.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#EB9992;"><span id="S4.T5.9.9.13.4.4.1" class="ltx_text" style="background-color:#EB9992;">-3.9%</span></td>
<td id="S4.T5.9.9.13.4.5" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#E67D74;"><span id="S4.T5.9.9.13.4.5.1" class="ltx_text" style="background-color:#E67D74;">-4.9%</span></td>
<td id="S4.T5.9.9.13.4.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E67C73;"><span id="S4.T5.9.9.13.4.6.1" class="ltx_text" style="background-color:#E67C73;">-8.0%</span></td>
<td id="S4.T5.9.9.13.4.7" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#E67C73;"><span id="S4.T5.9.9.13.4.7.1" class="ltx_text" style="background-color:#E67C73;">-10.2%</span></td>
<td id="S4.T5.9.9.13.4.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8DCDA;"><span id="S4.T5.9.9.13.4.8.1" class="ltx_text" style="background-color:#F8DCDA;">-1.3%</span></td>
<td id="S4.T5.9.9.13.4.9" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F4C9C5;"><span id="S4.T5.9.9.13.4.9.1" class="ltx_text" style="background-color:#F4C9C5;">-2.0%</span></td>
<td id="S4.T5.9.9.13.4.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F9E1DF;"><span id="S4.T5.9.9.13.4.10.1" class="ltx_text" style="background-color:#F9E1DF;">-1.1%</span></td>
<td id="S4.T5.9.9.13.4.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F3C2BD;"><span id="S4.T5.9.9.13.4.11.1" class="ltx_text" style="background-color:#F3C2BD;">-2.3%</span></td>
<td id="S4.T5.9.9.13.4.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCFEFD;"><span id="S4.T5.9.9.13.4.12.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="S4.T5.9.9.13.4.13" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FCFEFD;"><span id="S4.T5.9.9.13.4.13.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="S4.T5.9.9.13.4.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#B4E1CB;"><span id="S4.T5.9.9.13.4.14.1" class="ltx_text" style="background-color:#B4E1CB;">2.9%</span></td>
<td id="S4.T5.9.9.13.4.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F3;"><span id="S4.T5.9.9.13.4.15.1" class="ltx_text" style="background-color:#FCF3F3;">-0.4%</span></td>
</tr>
<tr id="S4.T5.9.9.14.5" class="ltx_tr">
<th id="S4.T5.9.9.14.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">W4-g</th>
<td id="S4.T5.9.9.14.5.2" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FBEEED;"><span id="S4.T5.9.9.14.5.2.1" class="ltx_text" style="background-color:#FBEEED;">-0.6%</span></td>
<td id="S4.T5.9.9.14.5.3" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F8DAD8;"><span id="S4.T5.9.9.14.5.3.1" class="ltx_text" style="background-color:#F8DAD8;">-1.4%</span></td>
<td id="S4.T5.9.9.14.5.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9E2E0;"><span id="S4.T5.9.9.14.5.4.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="S4.T5.9.9.14.5.5" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F5CCC9;"><span id="S4.T5.9.9.14.5.5.1" class="ltx_text" style="background-color:#F5CCC9;">-1.9%</span></td>
<td id="S4.T5.9.9.14.5.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F6CFCC;"><span id="S4.T5.9.9.14.5.6.1" class="ltx_text" style="background-color:#F6CFCC;">-1.8%</span></td>
<td id="S4.T5.9.9.14.5.7" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#E67E75;"><span id="S4.T5.9.9.14.5.7.1" class="ltx_text" style="background-color:#E67E75;">-4.9%</span></td>
<td id="S4.T5.9.9.14.5.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FBFEFC;"><span id="S4.T5.9.9.14.5.8.1" class="ltx_text" style="background-color:#FBFEFC;">0.2%</span></td>
<td id="S4.T5.9.9.14.5.9" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#ECF8F2;"><span id="S4.T5.9.9.14.5.9.1" class="ltx_text" style="background-color:#ECF8F2;">0.7%</span></td>
<td id="S4.T5.9.9.14.5.10" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FDF6F5;"><span id="S4.T5.9.9.14.5.10.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="S4.T5.9.9.14.5.11" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#FAE9E7;"><span id="S4.T5.9.9.14.5.11.1" class="ltx_text" style="background-color:#FAE9E7;">-0.8%</span></td>
<td id="S4.T5.9.9.14.5.12" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FDFEFE;"><span id="S4.T5.9.9.14.5.12.1" class="ltx_text" style="background-color:#FDFEFE;">0.1%</span></td>
<td id="S4.T5.9.9.14.5.13" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#FEFCFC;"><span id="S4.T5.9.9.14.5.13.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="S4.T5.9.9.14.5.14" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE7E5;"><span id="S4.T5.9.9.14.5.14.1" class="ltx_text" style="background-color:#FAE7E5;">-0.9%</span></td>
<td id="S4.T5.9.9.14.5.15" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F8DCD9;"><span id="S4.T5.9.9.14.5.15.1" class="ltx_text" style="background-color:#F8DCD9;">-1.3%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span><span id="S4.T5.13.1" class="ltx_text ltx_font_bold">Effect of mitigation strategies on W8A8 and W4 quantization on the 103B model.</span> Percentage points off FP16 baseline for W8A8-sq vs. naive W8A8 and W4-g vs. W4, broken down by Latin-script/Indo-European languages (Ltn/IE) versus others (<math id="S4.T5.11.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T5.11.m1.1b"><mo id="S4.T5.11.m1.1.1" xref="S4.T5.11.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T5.11.m1.1c"><not id="S4.T5.11.m1.1.1.cmml" xref="S4.T5.11.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T5.11.m1.1d">\neg</annotation></semantics></math>). Avg. Rel. % reports averaged performance all datasets.</figcaption>
</figure>
<figure id="S4.T6" class="ltx_table">
<div id="S4.T6.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:110.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-136.0pt,34.7pt) scale(0.614457569252135,0.614457569252135) ;">
<table id="S4.T6.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T6.1.1.1" class="ltx_tr">
<th id="S4.T6.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt"></th>
<th id="S4.T6.1.1.1.3" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt"></th>
<th id="S4.T6.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S4.T6.1.1.1.4.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="S4.T6.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S4.T6.1.1.1.5.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="S4.T6.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S4.T6.1.1.1.6.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="S4.T6.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_rr ltx_border_tt" colspan="2"><span id="S4.T6.1.1.1.7.1" class="ltx_text ltx_font_bold">ko</span></th>
<th id="S4.T6.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S4.T6.1.1.1.8.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="S4.T6.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S4.T6.1.1.1.9.1" class="ltx_text ltx_font_bold">Ltn/IE</span></th>
<th id="S4.T6.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><math id="S4.T6.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T6.1.1.1.1.m1.1a"><mo id="S4.T6.1.1.1.1.m1.1.1" xref="S4.T6.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T6.1.1.1.1.m1.1b"><not id="S4.T6.1.1.1.1.m1.1.1.cmml" xref="S4.T6.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T6.1.1.1.1.m1.1c">\neg</annotation></semantics></math></th>
</tr>
<tr id="S4.T6.1.1.2.1" class="ltx_tr">
<th id="S4.T6.1.1.2.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row"></th>
<th id="S4.T6.1.1.2.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r"></th>
<th id="S4.T6.1.1.2.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T6.1.1.2.1.3.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="S4.T6.1.1.2.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T6.1.1.2.1.4.1" class="ltx_text ltx_font_bold">RM</span></th>
<th id="S4.T6.1.1.2.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T6.1.1.2.1.5.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="S4.T6.1.1.2.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T6.1.1.2.1.6.1" class="ltx_text ltx_font_bold">RM</span></th>
<th id="S4.T6.1.1.2.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T6.1.1.2.1.7.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="S4.T6.1.1.2.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T6.1.1.2.1.8.1" class="ltx_text ltx_font_bold">RM</span></th>
<th id="S4.T6.1.1.2.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T6.1.1.2.1.9.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="S4.T6.1.1.2.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_rr"><span id="S4.T6.1.1.2.1.10.1" class="ltx_text ltx_font_bold">RM</span></th>
<th id="S4.T6.1.1.2.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T6.1.1.2.1.11.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="S4.T6.1.1.2.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T6.1.1.2.1.12.1" class="ltx_text ltx_font_bold">RM</span></th>
<th id="S4.T6.1.1.2.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T6.1.1.2.1.13.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="S4.T6.1.1.2.1.14" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T6.1.1.2.1.14.1" class="ltx_text ltx_font_bold">RM</span></th>
<th id="S4.T6.1.1.2.1.15" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T6.1.1.2.1.15.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="S4.T6.1.1.2.1.16" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T6.1.1.2.1.16.1" class="ltx_text ltx_font_bold">RM</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T6.1.1.3.1" class="ltx_tr">
<th id="S4.T6.1.1.3.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="S4.T6.1.1.3.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">W8</th>
<td id="S4.T6.1.1.3.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F3FAF7;"><span id="S4.T6.1.1.3.1.3.1" class="ltx_text" style="background-color:#F3FAF7;">1.0%</span></td>
<td id="S4.T6.1.1.3.1.4" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FEFAFA;"><span id="S4.T6.1.1.3.1.4.1" class="ltx_text" style="background-color:#FEFAFA;">-0.7%</span></td>
<td id="S4.T6.1.1.3.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F2BCB7;"><span id="S4.T6.1.1.3.1.5.1" class="ltx_text" style="background-color:#F2BCB7;">-10.2%</span></td>
<td id="S4.T6.1.1.3.1.6" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#A1D9BE;"><span id="S4.T6.1.1.3.1.6.1" class="ltx_text" style="background-color:#A1D9BE;">7.5%</span></td>
<td id="S4.T6.1.1.3.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8DBD9;"><span id="S4.T6.1.1.3.1.7.1" class="ltx_text" style="background-color:#F8DBD9;">-5.4%</span></td>
<td id="S4.T6.1.1.3.1.8" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#BBE4D0;"><span id="S4.T6.1.1.3.1.8.1" class="ltx_text" style="background-color:#BBE4D0;">5.4%</span></td>
<td id="S4.T6.1.1.3.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#A2D9BE;"><span id="S4.T6.1.1.3.1.9.1" class="ltx_text" style="background-color:#A2D9BE;">7.5%</span></td>
<td id="S4.T6.1.1.3.1.10" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#F7D9D6;"><span id="S4.T6.1.1.3.1.10.1" class="ltx_text" style="background-color:#F7D9D6;">-5.8%</span></td>
<td id="S4.T6.1.1.3.1.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F2;"><span id="S4.T6.1.1.3.1.11.1" class="ltx_text" style="background-color:#FCF3F2;">-1.8%</span></td>
<td id="S4.T6.1.1.3.1.12" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#EBF7F1;"><span id="S4.T6.1.1.3.1.12.1" class="ltx_text" style="background-color:#EBF7F1;">1.6%</span></td>
<td id="S4.T6.1.1.3.1.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F9E1DF;"><span id="S4.T6.1.1.3.1.13.1" class="ltx_text" style="background-color:#F9E1DF;">-4.6%</span></td>
<td id="S4.T6.1.1.3.1.14" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#D5EEE2;"><span id="S4.T6.1.1.3.1.14.1" class="ltx_text" style="background-color:#D5EEE2;">3.4%</span></td>
<td id="S4.T6.1.1.3.1.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F3FAF7;"><span id="S4.T6.1.1.3.1.15.1" class="ltx_text" style="background-color:#F3FAF7;">1.0%</span></td>
<td id="S4.T6.1.1.3.1.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="S4.T6.1.1.3.1.16.1" class="ltx_text" style="background-color:#FEFDFD;">-0.2%</span></td>
</tr>
<tr id="S4.T6.1.1.4.2" class="ltx_tr">
<th id="S4.T6.1.1.4.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S4.T6.1.1.4.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="S4.T6.1.1.4.2.3" class="ltx_td ltx_align_right" style="background-color:#E8867E;"><span id="S4.T6.1.1.4.2.3.1" class="ltx_text" style="background-color:#E8867E;">-18.4%</span></td>
<td id="S4.T6.1.1.4.2.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F8DDDB;"><span id="S4.T6.1.1.4.2.4.1" class="ltx_text" style="background-color:#F8DDDB;">-5.1%</span></td>
<td id="S4.T6.1.1.4.2.5" class="ltx_td ltx_align_right" style="background-color:#FAE6E4;"><span id="S4.T6.1.1.4.2.5.1" class="ltx_text" style="background-color:#FAE6E4;">-3.7%</span></td>
<td id="S4.T6.1.1.4.2.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#CCEBDC;"><span id="S4.T6.1.1.4.2.6.1" class="ltx_text" style="background-color:#CCEBDC;">4.1%</span></td>
<td id="S4.T6.1.1.4.2.7" class="ltx_td ltx_align_right" style="background-color:#E6F5EE;"><span id="S4.T6.1.1.4.2.7.1" class="ltx_text" style="background-color:#E6F5EE;">2.0%</span></td>
<td id="S4.T6.1.1.4.2.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#C4E7D6;"><span id="S4.T6.1.1.4.2.8.1" class="ltx_text" style="background-color:#C4E7D6;">4.7%</span></td>
<td id="S4.T6.1.1.4.2.9" class="ltx_td ltx_align_right" style="background-color:#D0ECDF;"><span id="S4.T6.1.1.4.2.9.1" class="ltx_text" style="background-color:#D0ECDF;">3.7%</span></td>
<td id="S4.T6.1.1.4.2.10" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F8DDDB;"><span id="S4.T6.1.1.4.2.10.1" class="ltx_text" style="background-color:#F8DDDB;">-5.1%</span></td>
<td id="S4.T6.1.1.4.2.11" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="S4.T6.1.1.4.2.11.1" class="ltx_text" style="background-color:#F9E4E2;">-4.1%</span></td>
<td id="S4.T6.1.1.4.2.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFCFC;"><span id="S4.T6.1.1.4.2.12.1" class="ltx_text" style="background-color:#FEFCFC;">-0.3%</span></td>
<td id="S4.T6.1.1.4.2.13" class="ltx_td ltx_align_right" style="background-color:#F1B6B1;"><span id="S4.T6.1.1.4.2.13.1" class="ltx_text" style="background-color:#F1B6B1;">-11.0%</span></td>
<td id="S4.T6.1.1.4.2.14" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFBFB;"><span id="S4.T6.1.1.4.2.14.1" class="ltx_text" style="background-color:#FEFBFB;">-0.5%</span></td>
<td id="S4.T6.1.1.4.2.15" class="ltx_td ltx_align_right" style="background-color:#DBF1E6;"><span id="S4.T6.1.1.4.2.15.1" class="ltx_text" style="background-color:#DBF1E6;">2.9%</span></td>
<td id="S4.T6.1.1.4.2.16" class="ltx_td ltx_align_right" style="background-color:#FEFDFD;"><span id="S4.T6.1.1.4.2.16.1" class="ltx_text" style="background-color:#FEFDFD;">-0.2%</span></td>
</tr>
<tr id="S4.T6.1.1.5.3" class="ltx_tr">
<th id="S4.T6.1.1.5.3.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S4.T6.1.1.5.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4-g</th>
<td id="S4.T6.1.1.5.3.3" class="ltx_td ltx_align_right" style="background-color:#F1BAB5;"><span id="S4.T6.1.1.5.3.3.1" class="ltx_text" style="background-color:#F1BAB5;">-10.5%</span></td>
<td id="S4.T6.1.1.5.3.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E98F88;"><span id="S4.T6.1.1.5.3.4.1" class="ltx_text" style="background-color:#E98F88;">-17.0%</span></td>
<td id="S4.T6.1.1.5.3.5" class="ltx_td ltx_align_right" style="background-color:#EA928A;"><span id="S4.T6.1.1.5.3.5.1" class="ltx_text" style="background-color:#EA928A;">-16.6%</span></td>
<td id="S4.T6.1.1.5.3.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E6F5EE;"><span id="S4.T6.1.1.5.3.6.1" class="ltx_text" style="background-color:#E6F5EE;">2.0%</span></td>
<td id="S4.T6.1.1.5.3.7" class="ltx_td ltx_align_right" style="background-color:#EB9A93;"><span id="S4.T6.1.1.5.3.7.1" class="ltx_text" style="background-color:#EB9A93;">-15.3%</span></td>
<td id="S4.T6.1.1.5.3.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FFFFFF;"><span id="S4.T6.1.1.5.3.8.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="S4.T6.1.1.5.3.9" class="ltx_td ltx_align_right" style="background-color:#F7D9D6;"><span id="S4.T6.1.1.5.3.9.1" class="ltx_text" style="background-color:#F7D9D6;">-5.8%</span></td>
<td id="S4.T6.1.1.5.3.10" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#EB9891;"><span id="S4.T6.1.1.5.3.10.1" class="ltx_text" style="background-color:#EB9891;">-15.6%</span></td>
<td id="S4.T6.1.1.5.3.11" class="ltx_td ltx_align_right" style="background-color:#EFB0AA;"><span id="S4.T6.1.1.5.3.11.1" class="ltx_text" style="background-color:#EFB0AA;">-12.1%</span></td>
<td id="S4.T6.1.1.5.3.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F5CCC9;"><span id="S4.T6.1.1.5.3.12.1" class="ltx_text" style="background-color:#F5CCC9;">-7.7%</span></td>
<td id="S4.T6.1.1.5.3.13" class="ltx_td ltx_align_right" style="background-color:#EEA6A0;"><span id="S4.T6.1.1.5.3.13.1" class="ltx_text" style="background-color:#EEA6A0;">-13.6%</span></td>
<td id="S4.T6.1.1.5.3.14" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F5CECA;"><span id="S4.T6.1.1.5.3.14.1" class="ltx_text" style="background-color:#F5CECA;">-7.5%</span></td>
<td id="S4.T6.1.1.5.3.15" class="ltx_td ltx_align_right" style="background-color:#F1B9B5;"><span id="S4.T6.1.1.5.3.15.1" class="ltx_text" style="background-color:#F1B9B5;">-10.5%</span></td>
<td id="S4.T6.1.1.5.3.16" class="ltx_td ltx_align_right" style="background-color:#F5CBC8;"><span id="S4.T6.1.1.5.3.16.1" class="ltx_text" style="background-color:#F5CBC8;">-7.8%</span></td>
</tr>
<tr id="S4.T6.1.1.6.4" class="ltx_tr">
<th id="S4.T6.1.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T6.1.1.6.4.1.1" class="ltx_text">Internal</span></th>
<th id="S4.T6.1.1.6.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4</th>
<td id="S4.T6.1.1.6.4.3" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.3.1" class="ltx_text" style="background-color:#E67C73;">-30.2%</span></td>
<td id="S4.T6.1.1.6.4.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.4.1" class="ltx_text" style="background-color:#E67C73;">-20.4%</span></td>
<td id="S4.T6.1.1.6.4.5" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.5.1" class="ltx_text" style="background-color:#E67C73;">-33.0%</span></td>
<td id="S4.T6.1.1.6.4.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E98F88;"><span id="S4.T6.1.1.6.4.6.1" class="ltx_text" style="background-color:#E98F88;">-17.0%</span></td>
<td id="S4.T6.1.1.6.4.7" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.7.1" class="ltx_text" style="background-color:#E67C73;">-21.7%</span></td>
<td id="S4.T6.1.1.6.4.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.8.1" class="ltx_text" style="background-color:#E67C73;">-20.0%</span></td>
<td id="S4.T6.1.1.6.4.9" class="ltx_td ltx_align_right" style="background-color:#E7847C;"><span id="S4.T6.1.1.6.4.9.1" class="ltx_text" style="background-color:#E7847C;">-18.6%</span></td>
<td id="S4.T6.1.1.6.4.10" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.10.1" class="ltx_text" style="background-color:#E67C73;">-27.6%</span></td>
<td id="S4.T6.1.1.6.4.11" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.11.1" class="ltx_text" style="background-color:#E67C73;">-25.9%</span></td>
<td id="S4.T6.1.1.6.4.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.12.1" class="ltx_text" style="background-color:#E67C73;">-21.2%</span></td>
<td id="S4.T6.1.1.6.4.13" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.13.1" class="ltx_text" style="background-color:#E67C73;">-31.6%</span></td>
<td id="S4.T6.1.1.6.4.14" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E7847C;"><span id="S4.T6.1.1.6.4.14.1" class="ltx_text" style="background-color:#E7847C;">-18.7%</span></td>
<td id="S4.T6.1.1.6.4.15" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.15.1" class="ltx_text" style="background-color:#E67C73;">-20.2%</span></td>
<td id="S4.T6.1.1.6.4.16" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="S4.T6.1.1.6.4.16.1" class="ltx_text" style="background-color:#E67C73;">-23.8%</span></td>
</tr>
<tr id="S4.T6.1.1.7.5" class="ltx_tr">
<th id="S4.T6.1.1.7.5.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="S4.T6.1.1.7.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">W8</th>
<td id="S4.T6.1.1.7.5.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F5;"><span id="S4.T6.1.1.7.5.3.1" class="ltx_text" style="background-color:#FDF6F5;">-1.3%</span></td>
<td id="S4.T6.1.1.7.5.4" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#E6F5EE;"><span id="S4.T6.1.1.7.5.4.1" class="ltx_text" style="background-color:#E6F5EE;">2.0%</span></td>
<td id="S4.T6.1.1.7.5.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#A3DABF;"><span id="S4.T6.1.1.7.5.5.1" class="ltx_text" style="background-color:#A3DABF;">7.3%</span></td>
<td id="S4.T6.1.1.7.5.6" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FAE4E2;"><span id="S4.T6.1.1.7.5.6.1" class="ltx_text" style="background-color:#FAE4E2;">-4.0%</span></td>
<td id="S4.T6.1.1.7.5.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F7D7D5;"><span id="S4.T6.1.1.7.5.7.1" class="ltx_text" style="background-color:#F7D7D5;">-6.0%</span></td>
<td id="S4.T6.1.1.7.5.8" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F8DCD9;"><span id="S4.T6.1.1.7.5.8.1" class="ltx_text" style="background-color:#F8DCD9;">-5.3%</span></td>
<td id="S4.T6.1.1.7.5.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#DEF2E8;"><span id="S4.T6.1.1.7.5.9.1" class="ltx_text" style="background-color:#DEF2E8;">2.7%</span></td>
<td id="S4.T6.1.1.7.5.10" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#E6F5EE;"><span id="S4.T6.1.1.7.5.10.1" class="ltx_text" style="background-color:#E6F5EE;">2.0%</span></td>
<td id="S4.T6.1.1.7.5.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F7FCFA;"><span id="S4.T6.1.1.7.5.11.1" class="ltx_text" style="background-color:#F7FCFA;">0.7%</span></td>
<td id="S4.T6.1.1.7.5.12" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FDF6F5;"><span id="S4.T6.1.1.7.5.12.1" class="ltx_text" style="background-color:#FDF6F5;">-1.3%</span></td>
<td id="S4.T6.1.1.7.5.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#DAF0E5;"><span id="S4.T6.1.1.7.5.13.1" class="ltx_text" style="background-color:#DAF0E5;">3.0%</span></td>
<td id="S4.T6.1.1.7.5.14" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FDF8F8;"><span id="S4.T6.1.1.7.5.14.1" class="ltx_text" style="background-color:#FDF8F8;">-1.0%</span></td>
<td id="S4.T6.1.1.7.5.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF4F3;"><span id="S4.T6.1.1.7.5.15.1" class="ltx_text" style="background-color:#FCF4F3;">-1.7%</span></td>
<td id="S4.T6.1.1.7.5.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF4F3;"><span id="S4.T6.1.1.7.5.16.1" class="ltx_text" style="background-color:#FCF4F3;">-1.7%</span></td>
</tr>
<tr id="S4.T6.1.1.8.6" class="ltx_tr">
<th id="S4.T6.1.1.8.6.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S4.T6.1.1.8.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="S4.T6.1.1.8.6.3" class="ltx_td ltx_align_right" style="background-color:#EB9A93;"><span id="S4.T6.1.1.8.6.3.1" class="ltx_text" style="background-color:#EB9A93;">-15.3%</span></td>
<td id="S4.T6.1.1.8.6.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F4C6C2;"><span id="S4.T6.1.1.8.6.4.1" class="ltx_text" style="background-color:#F4C6C2;">-8.7%</span></td>
<td id="S4.T6.1.1.8.6.5" class="ltx_td ltx_align_right" style="background-color:#92D3B4;"><span id="S4.T6.1.1.8.6.5.1" class="ltx_text" style="background-color:#92D3B4;">8.7%</span></td>
<td id="S4.T6.1.1.8.6.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F5CAC7;"><span id="S4.T6.1.1.8.6.6.1" class="ltx_text" style="background-color:#F5CAC7;">-8.0%</span></td>
<td id="S4.T6.1.1.8.6.7" class="ltx_td ltx_align_right" style="background-color:#FDF6F5;"><span id="S4.T6.1.1.8.6.7.1" class="ltx_text" style="background-color:#FDF6F5;">-1.3%</span></td>
<td id="S4.T6.1.1.8.6.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#EFF9F4;"><span id="S4.T6.1.1.8.6.8.1" class="ltx_text" style="background-color:#EFF9F4;">1.3%</span></td>
<td id="S4.T6.1.1.8.6.9" class="ltx_td ltx_align_right" style="background-color:#F5CAC7;"><span id="S4.T6.1.1.8.6.9.1" class="ltx_text" style="background-color:#F5CAC7;">-8.0%</span></td>
<td id="S4.T6.1.1.8.6.10" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F9E0DE;"><span id="S4.T6.1.1.8.6.10.1" class="ltx_text" style="background-color:#F9E0DE;">-4.7%</span></td>
<td id="S4.T6.1.1.8.6.11" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="S4.T6.1.1.8.6.11.1" class="ltx_text" style="background-color:#F9E4E2;">-4.0%</span></td>
<td id="S4.T6.1.1.8.6.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F8DEDC;"><span id="S4.T6.1.1.8.6.12.1" class="ltx_text" style="background-color:#F8DEDC;">-5.0%</span></td>
<td id="S4.T6.1.1.8.6.13" class="ltx_td ltx_align_right" style="background-color:#FAE9E7;"><span id="S4.T6.1.1.8.6.13.1" class="ltx_text" style="background-color:#FAE9E7;">-3.3%</span></td>
<td id="S4.T6.1.1.8.6.14" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F4C8C4;"><span id="S4.T6.1.1.8.6.14.1" class="ltx_text" style="background-color:#F4C8C4;">-8.3%</span></td>
<td id="S4.T6.1.1.8.6.15" class="ltx_td ltx_align_right" style="background-color:#F9E0DE;"><span id="S4.T6.1.1.8.6.15.1" class="ltx_text" style="background-color:#F9E0DE;">-4.7%</span></td>
<td id="S4.T6.1.1.8.6.16" class="ltx_td ltx_align_right" style="background-color:#FCF4F3;"><span id="S4.T6.1.1.8.6.16.1" class="ltx_text" style="background-color:#FCF4F3;">-1.7%</span></td>
</tr>
<tr id="S4.T6.1.1.9.7" class="ltx_tr">
<th id="S4.T6.1.1.9.7.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S4.T6.1.1.9.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8</th>
<td id="S4.T6.1.1.9.7.3" class="ltx_td ltx_align_right" style="background-color:#FAE8E7;"><span id="S4.T6.1.1.9.7.3.1" class="ltx_text" style="background-color:#FAE8E7;">-3.4%</span></td>
<td id="S4.T6.1.1.9.7.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#DEF2E8;"><span id="S4.T6.1.1.9.7.4.1" class="ltx_text" style="background-color:#DEF2E8;">2.7%</span></td>
<td id="S4.T6.1.1.9.7.5" class="ltx_td ltx_align_right" style="background-color:#57BB8A;"><span id="S4.T6.1.1.9.7.5.1" class="ltx_text" style="background-color:#57BB8A;">13.3%</span></td>
<td id="S4.T6.1.1.9.7.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FAE9E7;"><span id="S4.T6.1.1.9.7.6.1" class="ltx_text" style="background-color:#FAE9E7;">-3.3%</span></td>
<td id="S4.T6.1.1.9.7.7" class="ltx_td ltx_align_right" style="background-color:#DEF2E8;"><span id="S4.T6.1.1.9.7.7.1" class="ltx_text" style="background-color:#DEF2E8;">2.7%</span></td>
<td id="S4.T6.1.1.9.7.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FDF6F5;"><span id="S4.T6.1.1.9.7.8.1" class="ltx_text" style="background-color:#FDF6F5;">-1.3%</span></td>
<td id="S4.T6.1.1.9.7.9" class="ltx_td ltx_align_right" style="background-color:#BCE4D1;"><span id="S4.T6.1.1.9.7.9.1" class="ltx_text" style="background-color:#BCE4D1;">5.3%</span></td>
<td id="S4.T6.1.1.9.7.10" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#FAE9E7;"><span id="S4.T6.1.1.9.7.10.1" class="ltx_text" style="background-color:#FAE9E7;">-3.3%</span></td>
<td id="S4.T6.1.1.9.7.11" class="ltx_td ltx_align_right" style="background-color:#C7E9D8;"><span id="S4.T6.1.1.9.7.11.1" class="ltx_text" style="background-color:#C7E9D8;">4.5%</span></td>
<td id="S4.T6.1.1.9.7.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FDF6F5;"><span id="S4.T6.1.1.9.7.12.1" class="ltx_text" style="background-color:#FDF6F5;">-1.3%</span></td>
<td id="S4.T6.1.1.9.7.13" class="ltx_td ltx_align_right" style="background-color:#C1E6D4;"><span id="S4.T6.1.1.9.7.13.1" class="ltx_text" style="background-color:#C1E6D4;">5.0%</span></td>
<td id="S4.T6.1.1.9.7.14" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFCFC;"><span id="S4.T6.1.1.9.7.14.1" class="ltx_text" style="background-color:#FEFCFC;">-0.3%</span></td>
<td id="S4.T6.1.1.9.7.15" class="ltx_td ltx_align_right" style="background-color:#CDEBDC;"><span id="S4.T6.1.1.9.7.15.1" class="ltx_text" style="background-color:#CDEBDC;">4.0%</span></td>
<td id="S4.T6.1.1.9.7.16" class="ltx_td ltx_align_right" style="background-color:#FCEFEE;"><span id="S4.T6.1.1.9.7.16.1" class="ltx_text" style="background-color:#FCEFEE;">-2.3%</span></td>
</tr>
<tr id="S4.T6.1.1.10.8" class="ltx_tr">
<th id="S4.T6.1.1.10.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S4.T6.1.1.10.8.1.1" class="ltx_text">Dolly</span></th>
<th id="S4.T6.1.1.10.8.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">W4-g</th>
<td id="S4.T6.1.1.10.8.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F5CECB;"><span id="S4.T6.1.1.10.8.3.1" class="ltx_text" style="background-color:#F5CECB;">-7.4%</span></td>
<td id="S4.T6.1.1.10.8.4" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#FBEDEC;"><span id="S4.T6.1.1.10.8.4.1" class="ltx_text" style="background-color:#FBEDEC;">-2.7%</span></td>
<td id="S4.T6.1.1.10.8.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE4E2;"><span id="S4.T6.1.1.10.8.5.1" class="ltx_text" style="background-color:#FAE4E2;">-4.0%</span></td>
<td id="S4.T6.1.1.10.8.6" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#C5E8D7;"><span id="S4.T6.1.1.10.8.6.1" class="ltx_text" style="background-color:#C5E8D7;">4.7%</span></td>
<td id="S4.T6.1.1.10.8.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EB9A93;"><span id="S4.T6.1.1.10.8.7.1" class="ltx_text" style="background-color:#EB9A93;">-15.3%</span></td>
<td id="S4.T6.1.1.10.8.8" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#EB9A93;"><span id="S4.T6.1.1.10.8.8.1" class="ltx_text" style="background-color:#EB9A93;">-15.3%</span></td>
<td id="S4.T6.1.1.10.8.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F0B4AF;"><span id="S4.T6.1.1.10.8.9.1" class="ltx_text" style="background-color:#F0B4AF;">-11.3%</span></td>
<td id="S4.T6.1.1.10.8.10" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#F8DCD9;"><span id="S4.T6.1.1.10.8.10.1" class="ltx_text" style="background-color:#F8DCD9;">-5.3%</span></td>
<td id="S4.T6.1.1.10.8.11" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C0BC;"><span id="S4.T6.1.1.10.8.11.1" class="ltx_text" style="background-color:#F3C0BC;">-9.5%</span></td>
<td id="S4.T6.1.1.10.8.12" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F9E0DE;"><span id="S4.T6.1.1.10.8.12.1" class="ltx_text" style="background-color:#F9E0DE;">-4.7%</span></td>
<td id="S4.T6.1.1.10.8.13" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F7D9D7;"><span id="S4.T6.1.1.10.8.13.1" class="ltx_text" style="background-color:#F7D9D7;">-5.7%</span></td>
<td id="S4.T6.1.1.10.8.14" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F3FAF7;"><span id="S4.T6.1.1.10.8.14.1" class="ltx_text" style="background-color:#F3FAF7;">1.0%</span></td>
<td id="S4.T6.1.1.10.8.15" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EEA7A1;"><span id="S4.T6.1.1.10.8.15.1" class="ltx_text" style="background-color:#EEA7A1;">-13.3%</span></td>
<td id="S4.T6.1.1.10.8.16" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F2BBB6;"><span id="S4.T6.1.1.10.8.16.1" class="ltx_text" style="background-color:#F2BBB6;">-10.3%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span><span id="S4.T6.5.1" class="ltx_text ltx_font_bold">Relative performance vs. FP16 of 103B quantized models according to <span id="S4.T6.5.1.1" class="ltx_text ltx_font_italic">LLM/RM-as-a-Judge</span></span> over <span id="S4.T6.6.2" class="ltx_text ltx_font_italic">Internal</span> and <span id="S4.T6.7.3" class="ltx_text ltx_font_italic">Aya Dolly</span> subsampled test sets. Raw win-rates in Table <a href="#A1.T20" title="Table A20 ‣ A.3 RM/LLM-as-a-Judge and Human Evaluation - Full Results ‣ Appendix A Appendix ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A20</span></a>.</figcaption>
</figure>
<div id="S4.SS5.p2" class="ltx_para">
<p id="S4.SS5.p2.1" class="ltx_p">Table <a href="#S4.T5" title="Table 5 ‣ 4.5 By Quantization Strategy ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> shows the effect of using SmoothQuant and Group-Wise scaling strategies.
We evaluate variants of the 103B Command model with SmoothQuant (<span id="S4.SS5.p2.1.1" class="ltx_text ltx_font_bold">W8A8-sq</span>), and a more naive <span id="S4.SS5.p2.1.2" class="ltx_text ltx_font_bold">W4</span> variant using per-column quantization instead of group-wise scaling. We compare <span id="S4.SS5.p2.1.3" class="ltx_text ltx_font_bold">W8A8-sq</span> to <span id="S4.SS5.p2.1.4" class="ltx_text ltx_font_bold">W8A8</span>, and <span id="S4.SS5.p2.1.5" class="ltx_text ltx_font_bold">W4-g</span> to <span id="S4.SS5.p2.1.6" class="ltx_text ltx_font_bold">W4</span>.</p>
</div>
<div id="S4.SS5.p3" class="ltx_para">
<p id="S4.SS5.p3.1" class="ltx_p">On average and across mMMLU, MGSM, and FLORES, Group-Wise scaling greatly improves over column-wise <span id="S4.SS5.p3.1.1" class="ltx_text ltx_font_bold">W4</span>, recovering over 6 percentage points lost on MGSM for <span id="S4.SS5.p3.1.2" class="ltx_text ltx_font_bold">Ltn/IE</span> languages. SmoothQuant has a similar effect on average and for mMMLU, though to a lesser degree. That said, SmoothQuant harms MGSM scores slightly, and Group-Wise scaling degrades cross-lingual language confusion. We again observe that <math id="S4.SS5.p3.1.m1.1" class="ltx_Math" alttext="\neg\text{Ltn/IE}" display="inline"><semantics id="S4.SS5.p3.1.m1.1a"><mrow id="S4.SS5.p3.1.m1.1.1" xref="S4.SS5.p3.1.m1.1.1.cmml"><mo rspace="0.167em" id="S4.SS5.p3.1.m1.1.1.1" xref="S4.SS5.p3.1.m1.1.1.1.cmml">¬</mo><mtext class="ltx_mathvariant_bold" id="S4.SS5.p3.1.m1.1.1.2" xref="S4.SS5.p3.1.m1.1.1.2a.cmml">Ltn/IE</mtext></mrow><annotation-xml encoding="MathML-Content" id="S4.SS5.p3.1.m1.1b"><apply id="S4.SS5.p3.1.m1.1.1.cmml" xref="S4.SS5.p3.1.m1.1.1"><not id="S4.SS5.p3.1.m1.1.1.1.cmml" xref="S4.SS5.p3.1.m1.1.1.1"></not><ci id="S4.SS5.p3.1.m1.1.1.2a.cmml" xref="S4.SS5.p3.1.m1.1.1.2"><mtext class="ltx_mathvariant_bold" id="S4.SS5.p3.1.m1.1.1.2.cmml" xref="S4.SS5.p3.1.m1.1.1.2">Ltn/IE</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p3.1.m1.1c">\neg\text{Ltn/IE}</annotation></semantics></math> languages suffer more in nearly all cases.</p>
</div>
<div id="S4.SS5.p4" class="ltx_para">
<p id="S4.SS5.p4.1" class="ltx_p">On cross-lingual language confusion, strategies aimed to retain performance have different effects: SmoothQuant recovers all lost from naive <span id="S4.SS5.p4.1.1" class="ltx_text ltx_font_bold">W8A8</span>, but Group-Wise scaling is actively damaging. In contrast, <span id="S4.SS5.p4.1.2" class="ltx_text ltx_font_bold">W4</span> benefits <span id="S4.SS5.p4.1.3" class="ltx_text ltx_font_bold">Ltn/IE</span> and Arabic on cross-lingual language confusion, but worsens the rest.<span id="footnote11" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">11</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">11</sup><span class="ltx_tag ltx_tag_note">11</span>Full results in are Table <a href="#A1.T19" title="Table A19 ‣ A.2 Automatic Tasks - Full Results ‣ Appendix A Appendix ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A19</span></a>.</span></span></span></p>
</div>
<div id="S4.SS5.p5" class="ltx_para">
<p id="S4.SS5.p5.1" class="ltx_p">Thus, while the quantization strategies tend to aid performance overall, there may be adverse effects on specific tasks.
More research is needed to understand this, but it is intriguing to consider the effect that lower-precision might have on the ability to produce output in a desired language, and maintain that language once decoding begins.</p>
</div>
</section>
<section id="S4.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.6 </span>LLM/RM-as-a-Judge</h3>

<div id="S4.SS6.p1" class="ltx_para">
<p id="S4.SS6.p1.1" class="ltx_p">Table <a href="#S4.T6" title="Table 6 ‣ 4.5 By Quantization Strategy ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> shows relative performance of quantized variants of the 103B Command model evaluated with LLM- and RM-as-a-Judge.<span id="footnote12" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">12</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">12</sup><span class="ltx_tag ltx_tag_note">12</span>Calculation: <math id="footnote12.m1.1" class="ltx_Math" alttext="\frac{\text{Quantized Win Rate}-50}{50}" display="inline"><semantics id="footnote12.m1.1b"><mfrac id="footnote12.m1.1.1" xref="footnote12.m1.1.1.cmml"><mrow id="footnote12.m1.1.1.2" xref="footnote12.m1.1.1.2.cmml"><mtext id="footnote12.m1.1.1.2.2" xref="footnote12.m1.1.1.2.2a.cmml">Quantized Win Rate</mtext><mo id="footnote12.m1.1.1.2.1" xref="footnote12.m1.1.1.2.1.cmml">−</mo><mn id="footnote12.m1.1.1.2.3" xref="footnote12.m1.1.1.2.3.cmml">50</mn></mrow><mn id="footnote12.m1.1.1.3" xref="footnote12.m1.1.1.3.cmml">50</mn></mfrac><annotation-xml encoding="MathML-Content" id="footnote12.m1.1c"><apply id="footnote12.m1.1.1.cmml" xref="footnote12.m1.1.1"><divide id="footnote12.m1.1.1.1.cmml" xref="footnote12.m1.1.1"></divide><apply id="footnote12.m1.1.1.2.cmml" xref="footnote12.m1.1.1.2"><minus id="footnote12.m1.1.1.2.1.cmml" xref="footnote12.m1.1.1.2.1"></minus><ci id="footnote12.m1.1.1.2.2a.cmml" xref="footnote12.m1.1.1.2.2"><mtext mathsize="70%" id="footnote12.m1.1.1.2.2.cmml" xref="footnote12.m1.1.1.2.2">Quantized Win Rate</mtext></ci><cn type="integer" id="footnote12.m1.1.1.2.3.cmml" xref="footnote12.m1.1.1.2.3">50</cn></apply><cn type="integer" id="footnote12.m1.1.1.3.cmml" xref="footnote12.m1.1.1.3">50</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote12.m1.1d">\frac{\text{Quantized Win Rate}-50}{50}</annotation></semantics></math>, as 50 is the expected win-rate of two FP16 models compared.</span></span></span> In nearly all cases, the LLM and RM agree that <span id="S4.SS6.p1.1.1" class="ltx_text ltx_font_bold">W4</span> and <span id="S4.SS6.p1.1.2" class="ltx_text ltx_font_bold">W4-g</span> severely harm performance on our challenging <span id="S4.SS6.p1.1.3" class="ltx_text ltx_font_italic">Internal</span> test set. Performance is also severely degraded for <math id="S4.SS6.p1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.SS6.p1.1.m1.1a"><mo id="S4.SS6.p1.1.m1.1.1" xref="S4.SS6.p1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.SS6.p1.1.m1.1b"><not id="S4.SS6.p1.1.m1.1.1.cmml" xref="S4.SS6.p1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.SS6.p1.1.m1.1c">\neg</annotation></semantics></math><span id="S4.SS6.p1.1.4" class="ltx_text ltx_font_bold">Ltn/IE</span> languages on <span id="S4.SS6.p1.1.5" class="ltx_text ltx_font_italic">Dolly</span> with <span id="S4.SS6.p1.1.6" class="ltx_text ltx_font_bold">W4-g</span>, and French with <span id="S4.SS6.p1.1.7" class="ltx_text ltx_font_bold">W8A8-sq</span>. On average across languages, the LLM and RM agree on the ranking of model quality over <span id="S4.SS6.p1.1.8" class="ltx_text ltx_font_italic">Internal</span>. Results on the easier <span id="S4.SS6.p1.1.9" class="ltx_text ltx_font_italic">Dolly</span> test set are less clear-cut: The LLM reports greater degradation for <em id="S4.SS6.p1.1.10" class="ltx_emph ltx_font_italic">Internal</em> than <span id="S4.SS6.p1.1.11" class="ltx_text ltx_font_italic">Dolly</span> overall, but the RM disagrees for <span id="S4.SS6.p1.1.12" class="ltx_text ltx_font_bold">W8</span> and <span id="S4.SS6.p1.1.13" class="ltx_text ltx_font_bold">W8A8-sq</span>. Perhaps <span id="S4.SS6.p1.1.14" class="ltx_text ltx_font_italic">Dolly</span> prompts are easy enough that models output similar responses, creating more noise in the judgments; future work could examine this hypothesis. Furthermore, on multiple instances, the LLM and RM disagree on whether performance <span id="S4.SS6.p1.1.15" class="ltx_text ltx_font_italic">improves</span> or <span id="S4.SS6.p1.1.16" class="ltx_text ltx_font_italic">worsens</span>, given the same setting. Comparisons between the two differing methods of automated evaluation are worthy of further study.</p>
</div>
</section>
<section id="S4.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.7 </span>Human Evaluation</h3>

<figure id="S4.T7" class="ltx_table">
<div id="S4.T7.1" class="ltx_inline-block ltx_transformed_outer" style="width:203.8pt;height:62.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-104.2pt,31.9pt) scale(0.494364157485127,0.494364157485127) ;">
<table id="S4.T7.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T7.1.1.1" class="ltx_tr">
<th id="S4.T7.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt"></th>
<th id="S4.T7.1.1.1.3" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt"></th>
<th id="S4.T7.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T7.1.1.1.4.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="S4.T7.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T7.1.1.1.5.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="S4.T7.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T7.1.1.1.6.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="S4.T7.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt"><span id="S4.T7.1.1.1.7.1" class="ltx_text ltx_font_bold">ko</span></th>
<th id="S4.T7.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T7.1.1.1.8.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="S4.T7.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T7.1.1.1.9.1" class="ltx_text ltx_font_bold">Ltn/IE</span></th>
<th id="S4.T7.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="S4.T7.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="S4.T7.1.1.1.1.m1.1a"><mo id="S4.T7.1.1.1.1.m1.1.1" xref="S4.T7.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="S4.T7.1.1.1.1.m1.1b"><not id="S4.T7.1.1.1.1.m1.1.1.cmml" xref="S4.T7.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="S4.T7.1.1.1.1.m1.1c">\neg</annotation></semantics></math></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T7.1.1.2.1" class="ltx_tr">
<th id="S4.T7.1.1.2.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="S4.T7.1.1.2.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">W8</th>
<td id="S4.T7.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F5CECB;"><span id="S4.T7.1.1.2.1.3.1" class="ltx_text" style="background-color:#F5CECB;">-7.4%</span></td>
<td id="S4.T7.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F7FCFA;"><span id="S4.T7.1.1.2.1.4.1" class="ltx_text" style="background-color:#F7FCFA;">0.6%</span></td>
<td id="S4.T7.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#98D6B7;"><span id="S4.T7.1.1.2.1.5.1" class="ltx_text" style="background-color:#98D6B7;">7.4%</span></td>
<td id="S4.T7.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#F0B0AB;"><span id="S4.T7.1.1.2.1.6.1" class="ltx_text" style="background-color:#F0B0AB;">-12.0%</span></td>
<td id="S4.T7.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBECEB;"><span id="S4.T7.1.1.2.1.7.1" class="ltx_text" style="background-color:#FBECEB;">-2.8%</span></td>
<td id="S4.T7.1.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE8E7;"><span id="S4.T7.1.1.2.1.8.1" class="ltx_text" style="background-color:#FAE8E7;">-3.4%</span></td>
<td id="S4.T7.1.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCEFEE;"><span id="S4.T7.1.1.2.1.9.1" class="ltx_text" style="background-color:#FCEFEE;">-2.3%</span></td>
</tr>
<tr id="S4.T7.1.1.3.2" class="ltx_tr">
<th id="S4.T7.1.1.3.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S4.T7.1.1.3.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">W8A8-sq</th>
<td id="S4.T7.1.1.3.2.3" class="ltx_td ltx_align_right" style="background-color:#F3C1BD;"><span id="S4.T7.1.1.3.2.3.1" class="ltx_text" style="background-color:#F3C1BD;">-9.4%</span></td>
<td id="S4.T7.1.1.3.2.4" class="ltx_td ltx_align_right" style="background-color:#F5CECB;"><span id="S4.T7.1.1.3.2.4.1" class="ltx_text" style="background-color:#F5CECB;">-7.4%</span></td>
<td id="S4.T7.1.1.3.2.5" class="ltx_td ltx_align_right" style="background-color:#FCF1F1;"><span id="S4.T7.1.1.3.2.5.1" class="ltx_text" style="background-color:#FCF1F1;">-2.0%</span></td>
<td id="S4.T7.1.1.3.2.6" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#C7E9D8;"><span id="S4.T7.1.1.3.2.6.1" class="ltx_text" style="background-color:#C7E9D8;">4.0%</span></td>
<td id="S4.T7.1.1.3.2.7" class="ltx_td ltx_align_right" style="background-color:#FAE6E5;"><span id="S4.T7.1.1.3.2.7.1" class="ltx_text" style="background-color:#FAE6E5;">-3.7%</span></td>
<td id="S4.T7.1.1.3.2.8" class="ltx_td ltx_align_right" style="background-color:#F4C7C4;"><span id="S4.T7.1.1.3.2.8.1" class="ltx_text" style="background-color:#F4C7C4;">-8.4%</span></td>
<td id="S4.T7.1.1.3.2.9" class="ltx_td ltx_align_right" style="background-color:#F2FAF6;"><span id="S4.T7.1.1.3.2.9.1" class="ltx_text" style="background-color:#F2FAF6;">1.0%</span></td>
</tr>
<tr id="S4.T7.1.1.4.3" class="ltx_tr">
<th id="S4.T7.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T7.1.1.4.3.1.1" class="ltx_text">Internal</span></th>
<th id="S4.T7.1.1.4.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">W4-g</th>
<td id="S4.T7.1.1.4.3.3" class="ltx_td ltx_align_right" style="background-color:#EA928A;"><span id="S4.T7.1.1.4.3.3.1" class="ltx_text" style="background-color:#EA928A;">-16.6%</span></td>
<td id="S4.T7.1.1.4.3.4" class="ltx_td ltx_align_right" style="background-color:#F9E0DE;"><span id="S4.T7.1.1.4.3.4.1" class="ltx_text" style="background-color:#F9E0DE;">-4.6%</span></td>
<td id="S4.T7.1.1.4.3.5" class="ltx_td ltx_align_right" style="background-color:#EB968F;"><span id="S4.T7.1.1.4.3.5.1" class="ltx_text" style="background-color:#EB968F;">-16.0%</span></td>
<td id="S4.T7.1.1.4.3.6" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F9E0DE;"><span id="S4.T7.1.1.4.3.6.1" class="ltx_text" style="background-color:#F9E0DE;">-4.6%</span></td>
<td id="S4.T7.1.1.4.3.7" class="ltx_td ltx_align_right" style="background-color:#F1BAB5;"><span id="S4.T7.1.1.4.3.7.1" class="ltx_text" style="background-color:#F1BAB5;">-10.5%</span></td>
<td id="S4.T7.1.1.4.3.8" class="ltx_td ltx_align_right" style="background-color:#F1B9B4;"><span id="S4.T7.1.1.4.3.8.1" class="ltx_text" style="background-color:#F1B9B4;">-10.6%</span></td>
<td id="S4.T7.1.1.4.3.9" class="ltx_td ltx_align_right" style="background-color:#F2BBB6;"><span id="S4.T7.1.1.4.3.9.1" class="ltx_text" style="background-color:#F2BBB6;">-10.3%</span></td>
</tr>
<tr id="S4.T7.1.1.5.4" class="ltx_tr">
<th id="S4.T7.1.1.5.4.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="S4.T7.1.1.5.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">W8</th>
<td id="S4.T7.1.1.5.4.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F7FCFA;"><span id="S4.T7.1.1.5.4.3.1" class="ltx_text" style="background-color:#F7FCFA;">0.6%</span></td>
<td id="S4.T7.1.1.5.4.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8DBD9;"><span id="S4.T7.1.1.5.4.4.1" class="ltx_text" style="background-color:#F8DBD9;">-5.4%</span></td>
<td id="S4.T7.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#57BB8A;"><span id="S4.T7.1.1.5.4.5.1" class="ltx_text" style="background-color:#57BB8A;">12.0%</span></td>
<td id="S4.T7.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FFFFFF;"><span id="S4.T7.1.1.5.4.6.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="S4.T7.1.1.5.4.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E6F5EE;"><span id="S4.T7.1.1.5.4.7.1" class="ltx_text" style="background-color:#E6F5EE;">1.8%</span></td>
<td id="S4.T7.1.1.5.4.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEFEE;"><span id="S4.T7.1.1.5.4.8.1" class="ltx_text" style="background-color:#FBEFEE;">-2.4%</span></td>
<td id="S4.T7.1.1.5.4.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#ABDDC5;"><span id="S4.T7.1.1.5.4.9.1" class="ltx_text" style="background-color:#ABDDC5;">6.0%</span></td>
</tr>
<tr id="S4.T7.1.1.6.5" class="ltx_tr">
<th id="S4.T7.1.1.6.5.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S4.T7.1.1.6.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">W8A8-sq</th>
<td id="S4.T7.1.1.6.5.3" class="ltx_td ltx_align_right" style="background-color:#F5CECB;"><span id="S4.T7.1.1.6.5.3.1" class="ltx_text" style="background-color:#F5CECB;">-7.4%</span></td>
<td id="S4.T7.1.1.6.5.4" class="ltx_td ltx_align_right" style="background-color:#F4C6C2;"><span id="S4.T7.1.1.6.5.4.1" class="ltx_text" style="background-color:#F4C6C2;">-8.6%</span></td>
<td id="S4.T7.1.1.6.5.5" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="S4.T7.1.1.6.5.5.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="S4.T7.1.1.6.5.6" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#FAE8E7;"><span id="S4.T7.1.1.6.5.6.1" class="ltx_text" style="background-color:#FAE8E7;">-3.4%</span></td>
<td id="S4.T7.1.1.6.5.7" class="ltx_td ltx_align_right" style="background-color:#F8DFDD;"><span id="S4.T7.1.1.6.5.7.1" class="ltx_text" style="background-color:#F8DFDD;">-4.8%</span></td>
<td id="S4.T7.1.1.6.5.8" class="ltx_td ltx_align_right" style="background-color:#F5CAC7;"><span id="S4.T7.1.1.6.5.8.1" class="ltx_text" style="background-color:#F5CAC7;">-8.0%</span></td>
<td id="S4.T7.1.1.6.5.9" class="ltx_td ltx_align_right" style="background-color:#FCF3F3;"><span id="S4.T7.1.1.6.5.9.1" class="ltx_text" style="background-color:#FCF3F3;">-1.7%</span></td>
</tr>
<tr id="S4.T7.1.1.7.6" class="ltx_tr">
<th id="S4.T7.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S4.T7.1.1.7.6.1.1" class="ltx_text">Dolly</span></th>
<th id="S4.T7.1.1.7.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">W4-g</th>
<td id="S4.T7.1.1.7.6.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C1BD;"><span id="S4.T7.1.1.7.6.3.1" class="ltx_text" style="background-color:#F3C1BD;">-9.4%</span></td>
<td id="S4.T7.1.1.7.6.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FDF5F5;"><span id="S4.T7.1.1.7.6.4.1" class="ltx_text" style="background-color:#FDF5F5;">-1.4%</span></td>
<td id="S4.T7.1.1.7.6.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#DBF1E6;"><span id="S4.T7.1.1.7.6.5.1" class="ltx_text" style="background-color:#DBF1E6;">2.6%</span></td>
<td id="S4.T7.1.1.7.6.6" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#F5CAC7;"><span id="S4.T7.1.1.7.6.6.1" class="ltx_text" style="background-color:#F5CAC7;">-8.0%</span></td>
<td id="S4.T7.1.1.7.6.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9E4E2;"><span id="S4.T7.1.1.7.6.7.1" class="ltx_text" style="background-color:#F9E4E2;">-4.1%</span></td>
<td id="S4.T7.1.1.7.6.8" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F8DBD9;"><span id="S4.T7.1.1.7.6.8.1" class="ltx_text" style="background-color:#F8DBD9;">-5.4%</span></td>
<td id="S4.T7.1.1.7.6.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FBEDEC;"><span id="S4.T7.1.1.7.6.9.1" class="ltx_text" style="background-color:#FBEDEC;">-2.7%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 7: </span><span id="S4.T7.5.1" class="ltx_text ltx_font_bold">Relative performance vs. FP16 of 103B quantized models according to <span id="S4.T7.5.1.1" class="ltx_text ltx_font_italic">human evaluators</span></span> over <span id="S4.T7.6.2" class="ltx_text ltx_font_italic">Internal</span> and <span id="S4.T7.7.3" class="ltx_text ltx_font_italic">Aya Dolly</span> subsampled test sets.</figcaption>
</figure>
<div id="S4.SS7.p1" class="ltx_para">
<p id="S4.SS7.p1.3" class="ltx_p">Human evaluation paints a similar picture in Table <a href="#S4.T7" title="Table 7 ‣ 4.7 Human Evaluation ‣ 4 Results ‣ How Does Quantization Affect Multilingual LLMs?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, with some outliers. Average performance drops steadily across evaluated languages on the <em id="S4.SS7.p1.3.1" class="ltx_emph ltx_font_italic">Internal</em> test set, which has more difficult prompts. The sharpest decline is in French, with <math id="S4.SS7.p1.1.m1.1" class="ltx_Math" alttext="-16.6\%" display="inline"><semantics id="S4.SS7.p1.1.m1.1a"><mrow id="S4.SS7.p1.1.m1.1.1" xref="S4.SS7.p1.1.m1.1.1.cmml"><mo id="S4.SS7.p1.1.m1.1.1a" xref="S4.SS7.p1.1.m1.1.1.cmml">−</mo><mrow id="S4.SS7.p1.1.m1.1.1.2" xref="S4.SS7.p1.1.m1.1.1.2.cmml"><mn id="S4.SS7.p1.1.m1.1.1.2.2" xref="S4.SS7.p1.1.m1.1.1.2.2.cmml">16.6</mn><mo id="S4.SS7.p1.1.m1.1.1.2.1" xref="S4.SS7.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS7.p1.1.m1.1b"><apply id="S4.SS7.p1.1.m1.1.1.cmml" xref="S4.SS7.p1.1.m1.1.1"><minus id="S4.SS7.p1.1.m1.1.1.1.cmml" xref="S4.SS7.p1.1.m1.1.1"></minus><apply id="S4.SS7.p1.1.m1.1.1.2.cmml" xref="S4.SS7.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S4.SS7.p1.1.m1.1.1.2.1.cmml" xref="S4.SS7.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS7.p1.1.m1.1.1.2.2.cmml" xref="S4.SS7.p1.1.m1.1.1.2.2">16.6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS7.p1.1.m1.1c">-16.6\%</annotation></semantics></math> at <span id="S4.SS7.p1.3.2" class="ltx_text ltx_font_bold">W4-g</span>. Curiously, there is an initial <math id="S4.SS7.p1.2.m2.1" class="ltx_Math" alttext="7.4\%" display="inline"><semantics id="S4.SS7.p1.2.m2.1a"><mrow id="S4.SS7.p1.2.m2.1.1" xref="S4.SS7.p1.2.m2.1.1.cmml"><mn id="S4.SS7.p1.2.m2.1.1.2" xref="S4.SS7.p1.2.m2.1.1.2.cmml">7.4</mn><mo id="S4.SS7.p1.2.m2.1.1.1" xref="S4.SS7.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS7.p1.2.m2.1b"><apply id="S4.SS7.p1.2.m2.1.1.cmml" xref="S4.SS7.p1.2.m2.1.1"><csymbol cd="latexml" id="S4.SS7.p1.2.m2.1.1.1.cmml" xref="S4.SS7.p1.2.m2.1.1.1">percent</csymbol><cn type="float" id="S4.SS7.p1.2.m2.1.1.2.cmml" xref="S4.SS7.p1.2.m2.1.1.2">7.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS7.p1.2.m2.1c">7.4\%</annotation></semantics></math> boost for Japanese with <span id="S4.SS7.p1.3.3" class="ltx_text ltx_font_bold">W8</span>, but it falls to <math id="S4.SS7.p1.3.m3.1" class="ltx_Math" alttext="-16.0\%" display="inline"><semantics id="S4.SS7.p1.3.m3.1a"><mrow id="S4.SS7.p1.3.m3.1.1" xref="S4.SS7.p1.3.m3.1.1.cmml"><mo id="S4.SS7.p1.3.m3.1.1a" xref="S4.SS7.p1.3.m3.1.1.cmml">−</mo><mrow id="S4.SS7.p1.3.m3.1.1.2" xref="S4.SS7.p1.3.m3.1.1.2.cmml"><mn id="S4.SS7.p1.3.m3.1.1.2.2" xref="S4.SS7.p1.3.m3.1.1.2.2.cmml">16.0</mn><mo id="S4.SS7.p1.3.m3.1.1.2.1" xref="S4.SS7.p1.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS7.p1.3.m3.1b"><apply id="S4.SS7.p1.3.m3.1.1.cmml" xref="S4.SS7.p1.3.m3.1.1"><minus id="S4.SS7.p1.3.m3.1.1.1.cmml" xref="S4.SS7.p1.3.m3.1.1"></minus><apply id="S4.SS7.p1.3.m3.1.1.2.cmml" xref="S4.SS7.p1.3.m3.1.1.2"><csymbol cd="latexml" id="S4.SS7.p1.3.m3.1.1.2.1.cmml" xref="S4.SS7.p1.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS7.p1.3.m3.1.1.2.2.cmml" xref="S4.SS7.p1.3.m3.1.1.2.2">16.0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS7.p1.3.m3.1c">-16.0\%</annotation></semantics></math> with more extreme quantization. Interestingly, human annotators generally prefer outputs of quantized models on <span id="S4.SS7.p1.3.4" class="ltx_text ltx_font_italic">Dolly</span> prompts in Japanese, too, but disprefer those in other languages. We see more pronounced degradation on <em id="S4.SS7.p1.3.5" class="ltx_emph ltx_font_italic">Internal</em> overall, with an average relative drop of 5.7% versus 2.4% for Dolly.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Related Work</h2>

<section id="S5.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Impact of Compression on Multilingual Tasks</h4>

<div id="S5.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p1.1" class="ltx_p">There is a scarcity of research examining the impact of compression and quantization on multilingual tasks. <cite class="ltx_cite ltx_citemacro_citet">Paglieri et al. (<a href="#bib.bib46" title="" class="ltx_ref">2024</a>)</cite> study the impact of multilingual calibration sets on quantization, but their evaluation is English-only. <cite class="ltx_cite ltx_citemacro_citet">Ramesh et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite> study the effect of compression on multilingual model fairness in terms of classification accuracy, showing that while monolingual evaluation indicates a negative impact, multilingual evaluation differs across languages and dimensions. <cite class="ltx_cite ltx_citemacro_citet">Kharazmi et al. (<a href="#bib.bib28" title="" class="ltx_ref">2023</a>)</cite> show that recovering compression-caused performance loss of LSTMs is harder in a multilingual setting than monolingually. In machine translation, <cite class="ltx_cite ltx_citemacro_citet">Diddee et al. (<a href="#bib.bib17" title="" class="ltx_ref">2022</a>)</cite> show that distillation has a varied effect by language due to dependence on priors such as amount of synthetic data used and confidence of the teacher models, while quantization exhibits more consistent performance trends across languages. Our work is the first, to our knowledge, to study the effect of quantization on LLMs and for open-ended generation.</p>
</div>
<div id="S5.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p2.1" class="ltx_p">More broadly, multilingual data is an example of long tail data. Prior work shows that compression techniques like quantization and sparsity amplify disparate treatment of long-tail rare features <cite class="ltx_cite ltx_citemacro_citep">(Hooker et al., <a href="#bib.bib24" title="" class="ltx_ref">2019</a>; Ahia et al., <a href="#bib.bib2" title="" class="ltx_ref">2021</a>; Ogueji et al., <a href="#bib.bib44" title="" class="ltx_ref">2022</a>; Hooker et al., <a href="#bib.bib25" title="" class="ltx_ref">2020</a>)</cite>. <cite class="ltx_cite ltx_citemacro_citet">Ogueji et al. (<a href="#bib.bib44" title="" class="ltx_ref">2022</a>)</cite> show that depending on how out of distribution the task data is, sparsity-based compression can sometimes avoid overfitting to the training data, making a model better suited to the downstream task. <cite class="ltx_cite ltx_citemacro_citet">Ahia et al. (<a href="#bib.bib2" title="" class="ltx_ref">2021</a>)</cite> find that sparsity preserves machine translation performance on frequent sentences, but disparately impacts infrequent sentences.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Quantization of LLMs</h4>

<div id="S5.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p1.1" class="ltx_p">A recent line of work has emerged on techniques to improve performance of quantized LLMs, with the sole focus on English models and data for tuning and evaluation <cite class="ltx_cite ltx_citemacro_citep">(Ahmadian et al., <a href="#bib.bib4" title="" class="ltx_ref">2024</a>; Dettmers et al., <a href="#bib.bib15" title="" class="ltx_ref">2022</a>; Xiao et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>; Bondarenko et al., <a href="#bib.bib8" title="" class="ltx_ref">2024</a>; Gong et al., <a href="#bib.bib21" title="" class="ltx_ref">2024</a>)</cite>. Even the most recent <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib33" title="" class="ltx_ref">2024</a>; Liu et al., <a href="#bib.bib37" title="" class="ltx_ref">2024</a>)</cite> omit the multilingual dimension without acknowledging the limitation. Multilinguality and compression are both integral parts of LLMs, and our work explores this new territory.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Model design choices</h4>

<div id="S5.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px3.p1.1" class="ltx_p">We consider how design choices such as quantization impact performance for users of different languages. A wider body of work examines how design choices impact performance on underrepresented features or subgroups. <cite class="ltx_cite ltx_citemacro_citet">Zhuang et al. (<a href="#bib.bib59" title="" class="ltx_ref">2021</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Nelaturu et al. (<a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite> find that hardware choice incurs disparate impact on underrepresented features. <cite class="ltx_cite ltx_citemacro_citet">Wang et al. (<a href="#bib.bib55" title="" class="ltx_ref">2022</a>)</cite> establish that distillation imposes similar trade-offs, but the disproportionate harm to the long-tail could be mitigated by modifying the student-teacher objective.
<cite class="ltx_cite ltx_citemacro_citet">Ko et al. (<a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite> evaluate the positive role of ensembling disproportionately favoring underrepresented attributes. <cite class="ltx_cite ltx_citemacro_citet">Bagdasaryan and Shmatikov (<a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite> show that differential privacy techniques like gradient clipping and noise injection disproportionately impact underrepresented features.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion &amp; Future Work</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">We examine widely adopted quantization techniques for model compression and ask, <span id="S6.p1.1.1" class="ltx_text ltx_font_italic">How does quantization impact different languages?</span> We perform an extensive study of quantization in state-of-the-art multilingual LLMs—from 8 billion to 103 billion parameters—in 20+ languages using automatic metrics, LLM-as-a-Judge, RM-as-a-Judge, and human evaluation. We find that: (1) Damage from quantization is much worse than appears from automatic metrics: even when not observed automatically, human evaluators notice it. (2) Quantization affects languages to varying degrees, with non-Latin script languages more severely affected on automatic benchmarks. (3) Challenging tasks degrade fast and severely: math performance is strikingly reduced, as are responses on realistic challenging prompts judged by humans. On a bright note, quantization occasionally brings performance benefits.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p">Our results urge attention to multilingual performance at all stages of system design. Researchers might extend our work to consider the impact of other decisions on multilingual performance, including on languages excluded from training and out-of-distribution tasks. By being mindful of the impact on long-tail features, we’ll build better systems to serve the world.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Limitations</h2>

<section id="S7.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Generality of findings</h4>

<div id="S7.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S7.SS0.SSS0.Px1.p1.1" class="ltx_p">Due to the number of methods, languages, and benchmarks we examine, we focus our evaluation on models from two families (Command R and Aya). As we observe similar trends across these models, our findings are likely to generalize to other LLMs. Nevertheless, models that have been optimized differently or trained with a focus on specific tasks such as code or mathematical reasoning may behave differently.</p>
</div>
</section>
<section id="S7.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Under-represented languages</h4>

<div id="S7.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S7.SS0.SSS0.Px2.p1.1" class="ltx_p">For our study, we focused on languages that were supported by the models we evaluated. Performance deterioration is likely even larger for languages that are not or severely under-represented in the pre-training data. For such languages, evaluation is also more challenging due to poor availability of benchmark data and human annotators.</p>
</div>
</section>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Achiam et al. (2023)</span>
<span class="ltx_bibblock">
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.

</span>
<span class="ltx_bibblock">Gpt-4 technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.08774</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ahia et al. (2021)</span>
<span class="ltx_bibblock">
Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2110.03036" title="" class="ltx_ref ltx_href">The low-resource double bind: An empirical study of pruning for low-resource machine translation</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2110.03036.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ahia et al. (2023)</span>
<span class="ltx_bibblock">
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.13707" title="" class="ltx_ref ltx_href">Do all languages cost the same? tokenization in the era of commercial language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2305.13707.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ahmadian et al. (2024)</span>
<span class="ltx_bibblock">
Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. 2024.

</span>
<span class="ltx_bibblock">Intriguing properties of quantization at scale.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 37th International Conference on Neural Information Processing Systems</em>, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Aryabumi et al. (2024)</span>
<span class="ltx_bibblock">
Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2405.15032" title="" class="ltx_ref ltx_href">Aya 23: Open weight releases to further multilingual progress</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2405.15032.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bagdasaryan and Shmatikov (2019)</span>
<span class="ltx_bibblock">
Eugene Bagdasaryan and Vitaly Shmatikov. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/1905.12101" title="" class="ltx_ref ltx_href">Differential privacy has disparate impact on model accuracy</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:1905.12101.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bandarkar et al. (2023)</span>
<span class="ltx_bibblock">
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2308.16884" title="" class="ltx_ref ltx_href">The belebele benchmark: a parallel reading comprehension dataset in 122 language variants</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2308.16884.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bondarenko et al. (2024)</span>
<span class="ltx_bibblock">
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2024.

</span>
<span class="ltx_bibblock">Quantizable transformers: Removing outliers by helping attention heads do nothing.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. (2021)</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2110.14168</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Conover et al. (2023)</span>
<span class="ltx_bibblock">
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm" title="" class="ltx_ref ltx_href">Free dolly: Introducing the world’s first truly open instruction-tuned llm</a>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Costa-jussà et al. (2022a)</span>
<span class="ltx_bibblock">
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022a.

</span>
<span class="ltx_bibblock">No language left behind: Scaling human-centered machine translation.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2207.04672</em>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Costa-jussà et al. (2022b)</span>
<span class="ltx_bibblock">
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022b.

</span>
<span class="ltx_bibblock">No language left behind: Scaling human-centered machine translation.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2207.04672</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dac Lai et al. (2023)</span>
<span class="ltx_bibblock">
Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023.

</span>
<span class="ltx_bibblock">Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv e-prints</em>, pages arXiv–2307.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Deng et al. (2023)</span>
<span class="ltx_bibblock">
Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023.

</span>
<span class="ltx_bibblock">Multilingual jailbreak challenges in large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.06474</em>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2022)</span>
<span class="ltx_bibblock">
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.

</span>
<span class="ltx_bibblock">Llm. int8 (): 8-bit matrix multiplication for transformers at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2208.07339</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2023)</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2305.14314" title="" class="ltx_ref ltx_href">Qlora: Efficient finetuning of quantized llms</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2305.14314.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Diddee et al. (2022)</span>
<span class="ltx_bibblock">
Harshita Diddee, Sandipan Dandapat, Monojit Choudhury, Tanuja Ganu, and Kalika Bali. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/2022.wmt-1.80" title="" class="ltx_ref ltx_href">Too brittle to touch: Comparing the stability of quantization and distillation towards developing low-resource MT models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Seventh Conference on Machine Translation (WMT)</em>, pages 870–885, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Durmus et al. (2023)</span>
<span class="ltx_bibblock">
Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2023.

</span>
<span class="ltx_bibblock">Towards measuring the representation of subjective global opinions in language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv</em>, abs/2306.16388.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Frantar et al. (2022)</span>
<span class="ltx_bibblock">
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022.

</span>
<span class="ltx_bibblock">GPTQ: Accurate post-training compression for generative pretrained transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.17323</em>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2023)</span>
<span class="ltx_bibblock">
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.5281/zenodo.10256836" title="" class="ltx_ref ltx_href">A framework for few-shot language model evaluation</a>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gong et al. (2024)</span>
<span class="ltx_bibblock">
Zhuocheng Gong, Jiahao Liu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. 2024.

</span>
<span class="ltx_bibblock">What makes quantization for large language model hard? an empirical study from the lens of perturbation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI Conference on Artificial Intelligence</em>, volume 38, pages 18082–18089.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Held et al. (2023)</span>
<span class="ltx_bibblock">
William Held, Camille Harris, Michael Best, and Diyi Yang. 2023.

</span>
<span class="ltx_bibblock">A material lens on coloniality in nlp.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv</em>, abs/2311.08391.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2020)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2009.03300</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hooker et al. (2019)</span>
<span class="ltx_bibblock">
Sara Hooker, Aaron C. Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:226812844" title="" class="ltx_ref ltx_href">What do compressed deep neural networks forget</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv: Learning</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hooker et al. (2020)</span>
<span class="ltx_bibblock">
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2010.03058" title="" class="ltx_ref ltx_href">Characterising bias in compressed models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2010.03058.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Joulin et al. (2016)</span>
<span class="ltx_bibblock">
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016.

</span>
<span class="ltx_bibblock">Bag of tricks for efficient text classification.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1607.01759</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khandelwal et al. (2023)</span>
<span class="ltx_bibblock">
Khyati Khandelwal, Manuel Tonneau, Andrew M. Bean, Hannah Rose Kirk, and Scott A. Hale. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:262013517" title="" class="ltx_ref ltx_href">Casteist but not racist? quantifying disparities in large language model bias between india and the west</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2309.08573.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kharazmi et al. (2023)</span>
<span class="ltx_bibblock">
Pegah Kharazmi, Zhewei Zhao, Clement Chung, and Samridhi Choudhary. 2023.

</span>
<span class="ltx_bibblock">Distill-quantize-tune-leveraging large teachers for low-footprint efficient multilingual nlu on edge.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)</em>, pages 1–5. IEEE.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khondaker et al. (2023)</span>
<span class="ltx_bibblock">
Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023.

</span>
<span class="ltx_bibblock">Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">arXiv</em>, abs/2305.14976.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ko et al. (2023)</span>
<span class="ltx_bibblock">
Wei-Yin Ko, Daniel D’souza, Karina Nguyen, Randall Balestriero, and Sara Hooker. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2303.00586" title="" class="ltx_ref ltx_href">Fair-ensemble: When fairness naturally emerges from deep ensembling</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2303.00586.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kotek et al. (2023)</span>
<span class="ltx_bibblock">
Hadas Kotek, Rikker Dockum, and David Q. Sun. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:261276445" title="" class="ltx_ref ltx_href">Gender bias and stereotypes in large language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Proceedings of The ACM Collective Intelligence Conference</em>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023a)</span>
<span class="ltx_bibblock">
Haoran Li, Yulin Chen, Jinglong Luo, Yan Kang, Xiaojin Zhang, Qi Hu, Chunkit Chan, and Yangqiu Song. 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:264145758" title="" class="ltx_ref ltx_href">Privacy in large language models: Attacks, defenses and future directions</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2310.10383.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2024)</span>
<span class="ltx_bibblock">
Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. 2024.

</span>
<span class="ltx_bibblock">Evaluating quantized large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.18158</em>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023b)</span>
<span class="ltx_bibblock">
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023b.

</span>
<span class="ltx_bibblock">Alpacaeval: An automatic evaluator of instruction-following models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/alpaca_eval" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/alpaca_eval</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. (2024)</span>
<span class="ltx_bibblock">
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024.

</span>
<span class="ltx_bibblock">Awq: Activation-aware weight quantization for llm compression and acceleration.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">MLSys</em>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. (2022)</span>
<span class="ltx_bibblock">
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.emnlp-main.616" title="" class="ltx_ref ltx_href">Few-shot learning with multilingual generative language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</em>, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2024)</span>
<span class="ltx_bibblock">
Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/2024.lrec-main.461" title="" class="ltx_ref ltx_href">Do emergent abilities exist in quantized large language models: An empirical study</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)</em>, pages 5174–5190, Torino, Italia. ELRA and ICCL.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lukas et al. (2023)</span>
<span class="ltx_bibblock">
Nils Lukas, A. Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-B’eguelin. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://api.semanticscholar.org/CorpusID:256459554" title="" class="ltx_ref ltx_href">Analyzing leakage of personally identifiable information in language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">2023 IEEE Symposium on Security and Privacy (SP)</em>, pages 346–363.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Marchisio et al. (2024)</span>
<span class="ltx_bibblock">
Kelly Marchisio, Wei-Yin Ko, Alexandre Bérard, Théo Dehaze, and Sebastian Ruder. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2406.20052" title="" class="ltx_ref ltx_href">Understanding and mitigating language confusion in llms</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2406.20052.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Muennighoff et al. (2023)</span>
<span class="ltx_bibblock">
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. 2023.

</span>
<span class="ltx_bibblock">Crosslingual generalization through multitask finetuning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">The 61st Annual Meeting Of The Association For Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nasr et al. (2023)</span>
<span class="ltx_bibblock">
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023.

</span>
<span class="ltx_bibblock">Scalable extraction of training data from (production) language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv</em>, abs/2311.17035.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nelaturu et al. (2023)</span>
<span class="ltx_bibblock">
Sree Harsha Nelaturu, Nishaanth Kanna Ravichandran, Cuong Tran, Sara Hooker, and Ferdinando Fioretto. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2312.03886" title="" class="ltx_ref ltx_href">On the fairness impacts of hardware selection in machine learning</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2312.03886.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nicholas and Bhatia (2023)</span>
<span class="ltx_bibblock">
Gabriel Nicholas and Aliya Bhatia. 2023.

</span>
<span class="ltx_bibblock">Lost in translation: Large language models in non-english content analysis.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">arXiv</em>, abs/2306.07377.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ogueji et al. (2022)</span>
<span class="ltx_bibblock">
Kelechi Ogueji, Orevaoghene Ahia, Gbemileke Onilude, Sebastian Gehrmann, Sara Hooker, and Julia Kreutzer. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2022.emnlp-main.619" title="" class="ltx_ref ltx_href">Intriguing properties of compression on multilingual models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</em>, pages 9092–9110, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ojo et al. (2023)</span>
<span class="ltx_bibblock">
Jessica Ojo, Kelechi Ogueji, Pontus Stenetorp, and David I. Adelani. 2023.

</span>
<span class="ltx_bibblock">How good are large language models on african languages?

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv</em>, abs/2311.07978.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paglieri et al. (2024)</span>
<span class="ltx_bibblock">
Davide Paglieri, Saurabh Dash, Tim Rocktäschel, and Jack Parker-Holder. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2405.20835" title="" class="ltx_ref ltx_href">Outliers and calibration sets have diminishing effect on quantization of modern llms</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2405.20835.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ponti et al. (2020)</span>
<span class="ltx_bibblock">
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2020.emnlp-main.185" title="" class="ltx_ref ltx_href">Xcopa: A multilingual dataset for causal commonsense reasoning</a>.

</span>
<span class="ltx_bibblock">pages 2362–2376.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Post (2018)</span>
<span class="ltx_bibblock">
Matt Post. 2018.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/W18-6319" title="" class="ltx_ref ltx_href">A call for clarity in reporting BLEU scores</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Third Conference on Machine Translation: Research Papers</em>, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pozzobon et al. (2024)</span>
<span class="ltx_bibblock">
Luiza Pozzobon, Patrick Lewis, Sara Hooker, and Beyza Ermis. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2403.03893" title="" class="ltx_ref ltx_href">From one to many: Expanding the scope of toxicity mitigation in language models</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2403.03893.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ramesh et al. (2023)</span>
<span class="ltx_bibblock">
Krithika Ramesh, Arnav Chavan, Shrey Pandit, and Sunayana Sitaram. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://doi.org/10.18653/v1/2023.acl-long.878" title="" class="ltx_ref ltx_href">A comparative study on the impact of model compression techniques on fairness in language models</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pages 15762–15782, Toronto, Canada. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schwartz et al. (2022)</span>
<span class="ltx_bibblock">
Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, Patrick Hall, et al. 2022.

</span>
<span class="ltx_bibblock">Towards a standard for identifying and managing bias in artificial intelligence.

</span>
<span class="ltx_bibblock"><em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">NIST special publication</em>, 1270(10.6028).

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shi et al. (2023)</span>
<span class="ltx_bibblock">
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openreview.net/forum?id=fR3wGCk-IXp" title="" class="ltx_ref ltx_href">Language models are multilingual chain-of-thought reasoners</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations</em>.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Singh et al. (2024)</span>
<span class="ltx_bibblock">
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. 2024.

</span>
<span class="ltx_bibblock">Aya dataset: An open-access collection for multilingual instruction tuning.

</span>
<span class="ltx_bibblock"><em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.06619</em>.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vashishtha et al. (2023)</span>
<span class="ltx_bibblock">
Aniket Vashishtha, Kabir Ahuja, and Sunayana Sitaram. 2023.

</span>
<span class="ltx_bibblock">On evaluating and mitigating gender biases in multilingual settings.

</span>
<span class="ltx_bibblock"><em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">arXiv</em>, abs/2307.01503.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Serena Wang, Harikrishna Narasimhan, Yichen Zhou, Sara Hooker, Michal Lukasik, and Aditya Krishna Menon. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2206.06479" title="" class="ltx_ref ltx_href">Robust distillation for worst-class performance</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2206.06479.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiao et al. (2023)</span>
<span class="ltx_bibblock">
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023.

</span>
<span class="ltx_bibblock">SmoothQuant: Accurate and efficient post-training quantization for large language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 40th International Conference on Machine Learning</em>.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yong et al. (2023)</span>
<span class="ltx_bibblock">
Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2023.

</span>
<span class="ltx_bibblock">Low-resource languages jailbreak GPT-4.

</span>
<span class="ltx_bibblock"><em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">arXiv</em>, abs/2310.02446.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. (2023)</span>
<span class="ltx_bibblock">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023.

</span>
<span class="ltx_bibblock">Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.

</span>
<span class="ltx_bibblock"><em id="bib.bib58.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhuang et al. (2021)</span>
<span class="ltx_bibblock">
Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. 2021.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2106.11872" title="" class="ltx_ref ltx_href">Randomness in neural network training: Characterizing the impact of tooling</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2106.11872.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Üstün et al. (2024)</span>
<span class="ltx_bibblock">
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2402.07827" title="" class="ltx_ref ltx_href">Aya model: An instruction finetuned open-access multilingual language model</a>.

</span>
<span class="ltx_bibblock"><em id="bib.bib60.1.1" class="ltx_emph ltx_font_italic">Preprint</em>, arXiv:2402.07827.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Appendix</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Prompts for mMMLU and LLM-as-a-Judge</h3>

<figure id="A1.T1" class="ltx_table">
<div id="A1.T1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:263.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(79.3pt,-48.3pt) scale(1.57647930502418,1.57647930502418) ;">
<table id="A1.T1.1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T1.1.1.1.1" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb ltx_border_tt">
<table id="A1.T1.1.1.1.1.1.1" class="ltx_tabular ltx_align_middle">
<tr id="A1.T1.1.1.1.1.1.1.1" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">The following are multiple choice questions (with answers)</td>
</tr>
<tr id="A1.T1.1.1.1.1.1.1.2" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left">about clinical knowledge.</td>
</tr>
<tr id="A1.T1.1.1.1.1.1.1.3" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1.1.3.1" class="ltx_td ltx_nopad_r ltx_align_left">다음 중 파제트병에 대한 설명으로 옳은 것은 무엇입니까?</td>
</tr>
<tr id="A1.T1.1.1.1.1.1.1.4" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1.1.4.1" class="ltx_td ltx_nopad_r ltx_align_left">A. 긴 뼈가 휘어지는 것이 특징</td>
</tr>
<tr id="A1.T1.1.1.1.1.1.1.5" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1.1.5.1" class="ltx_td ltx_nopad_r ltx_align_left">B. 척수압박은 흔한 합병증이다</td>
</tr>
<tr id="A1.T1.1.1.1.1.1.1.6" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1.1.6.1" class="ltx_td ltx_nopad_r ltx_align_left">C. 심부전은 알려진 합병증이 아니다</td>
</tr>
<tr id="A1.T1.1.1.1.1.1.1.7" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1.1.7.1" class="ltx_td ltx_nopad_r ltx_align_left">D. 병적 골절은 특징이 아닙니다.</td>
</tr>
<tr id="A1.T1.1.1.1.1.1.1.8" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1.1.8.1" class="ltx_td ltx_nopad_r ltx_align_left">Answer: B</td>
</tr>
<tr id="A1.T1.1.1.1.1.1.1.9" class="ltx_tr">
<td id="A1.T1.1.1.1.1.1.1.9.1" class="ltx_td ltx_nopad_r ltx_align_left">…</td>
</tr>
</table>
</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A1: </span><span id="A1.T1.3.1" class="ltx_text ltx_font_bold">mMMLU prompt</span>. Following <cite class="ltx_cite ltx_citemacro_citet">Achiam et al. (<a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite>, letter choices and “Answer” are kept in English.</figcaption>
</figure>
<figure id="A1.T2" class="ltx_table">
<table id="A1.T2.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T2.1.1.1" class="ltx_tr">
<td id="A1.T2.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T2.1.1.1.1.1" class="ltx_text ltx_font_bold">Example Prompt</span></td>
</tr>
<tr id="A1.T2.1.2.2" class="ltx_tr">
<td id="A1.T2.1.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb ltx_border_t">
<table id="A1.T2.1.2.2.1.1" class="ltx_tabular ltx_align_middle">
<tr id="A1.T2.1.2.2.1.1.1" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.1.1.1.1" class="ltx_p" style="width:433.6pt;">I want you to create a leaderboard of different large-language models. To do so, I will give you the conversations (prompts) given to the models, and the responses of two models. Please rank the models based on which responses would be preferred by humans. All inputs and outputs should be python dictionaries.</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.2" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.2.1.1.1" class="ltx_p" style="width:433.6pt;">Here is the prompt:</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.3" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.3.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.3.1.1.1" class="ltx_p" style="width:433.6pt;">{</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.4" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.4.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.4.1.1.1" class="ltx_p" style="width:433.6pt;">       "conversation": """User: La tomate est-elle un fruit ou un légume?""",</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.5" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.5.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.5.1.1.1" class="ltx_p" style="width:433.6pt;">}</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.6" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.6.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.6.1.1.1" class="ltx_p" style="width:433.6pt;">Here are the outputs of the models:</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.7" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.7.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.7.1.1.1" class="ltx_p" style="width:433.6pt;">[</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.8" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.8.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.8.1.1.1" class="ltx_p" style="width:433.6pt;">       {</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.9" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.9.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.9.1.1.1" class="ltx_p" style="width:433.6pt;">        "model": "model_1",</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.10" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.10.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.10.1.1.1" class="ltx_p" style="width:433.6pt;">        "answer": """La tomate est un fruit. Plus précisément, il s’agit d’un fruit charnu, issu de la transformation de l’ovaire de la fleur du plant de tomate."""</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.11" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.11.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.11.1.1.1" class="ltx_p" style="width:433.6pt;">       },</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.12" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.12.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.12.1.1.1" class="ltx_p" style="width:433.6pt;">       {</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.13" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.13.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.13.1.1.1" class="ltx_p" style="width:433.6pt;">        "model": "model_2",</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.14" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.14.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.14.1.1.1" class="ltx_p" style="width:433.6pt;">        "answer": """La tomate est un fruit du point de vue botanique, car elle contient des graines et se développe à partir de la fleur d’une plante. Cependant, en cuisine, on considère souvent la tomate comme un légume en raison de son utilisation dans des plats salés et de sa saveur moins sucrée par rapport à d’autres fruits."""</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.15" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.15.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.15.1.1.1" class="ltx_p" style="width:433.6pt;">       }</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.16" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.16.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.16.1.1.1" class="ltx_p" style="width:433.6pt;">]</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.17" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.17.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.17.1.1.1" class="ltx_p" style="width:433.6pt;">Now please rank the models by the quality of their answers, so that the model with rank 1 has the best output. Then return a list of the model names and ranks, i.e., produce the following output:</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.18" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.18.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.18.1.1.1" class="ltx_p" style="width:433.6pt;">[</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.19" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.19.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.19.1.1.1" class="ltx_p" style="width:433.6pt;">       {’model’: &lt;model-name&gt;, ’rank’: &lt;model-rank&gt;},</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.20" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.20.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.20.1.1.1" class="ltx_p" style="width:433.6pt;">       {’model’: &lt;model-name&gt;, ’rank’: &lt;model-rank&gt;}</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.21" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.21.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.21.1.1.1" class="ltx_p" style="width:433.6pt;">]</span>
</span>
</td>
</tr>
<tr id="A1.T2.1.2.2.1.1.22" class="ltx_tr">
<td id="A1.T2.1.2.2.1.1.22.1" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="A1.T2.1.2.2.1.1.22.1.1" class="ltx_inline-block ltx_align_top">
<span id="A1.T2.1.2.2.1.1.22.1.1.1" class="ltx_p" style="width:433.6pt;">Your response must be a valid Python dictionary and should contain nothing else because we will directly execute it in Python. Please provide the ranking that the majority of humans would give.</span>
</span>
</td>
</tr>
</table>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A2: </span><span id="A1.T2.3.1" class="ltx_text ltx_font_bold">Example Input for LLM-as-a-Judge.</span> Template derived from <cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib34" title="" class="ltx_ref">2023b</a>)</cite>: <a target="_blank" href="https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evaluators_configs/gpt-3.5-turbo-1106_ranking/ranking_prompt.txt" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evaluators_configs/gpt-3.5-turbo-1106_ranking/ranking_prompt.txt</a></figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Automatic Tasks - Full Results</h3>

<figure id="A1.T3" class="ltx_table">
<div id="A1.T3.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:301.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(74.3pt,-51.6pt) scale(1.52102658798657,1.52102658798657) ;">
<table id="A1.T3.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T3.1.1.1.1" class="ltx_tr">
<th id="A1.T3.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="A1.T3.1.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="A1.T3.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T3.1.1.1.1.3.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T3.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T3.1.1.1.1.4.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T3.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T3.1.1.1.1.5.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T3.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T3.1.1.1.1.6.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T3.1.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T3.1.1.1.1.7.1" class="ltx_text ltx_font_bold">zh</span></th>
<th id="A1.T3.1.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T3.1.1.1.1.8.1" class="ltx_text ltx_font_bold">Avg</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T3.1.1.2.1" class="ltx_tr">
<td id="A1.T3.1.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="6"><span id="A1.T3.1.1.2.1.1.1" class="ltx_text">103B</span></td>
<td id="A1.T3.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T3.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t">72.6</td>
<td id="A1.T3.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t">76.6</td>
<td id="A1.T3.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t">70.6</td>
<td id="A1.T3.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t">63.0</td>
<td id="A1.T3.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t">70.2</td>
<td id="A1.T3.1.1.2.1.8" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.2.1.8.1" class="ltx_text" style="background-color:#E6E6E6;">70.6</span></td>
</tr>
<tr id="A1.T3.1.1.3.2" class="ltx_tr">
<td id="A1.T3.1.1.3.2.1" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T3.1.1.3.2.2" class="ltx_td ltx_align_right">72.8</td>
<td id="A1.T3.1.1.3.2.3" class="ltx_td ltx_align_right">75.9</td>
<td id="A1.T3.1.1.3.2.4" class="ltx_td ltx_align_right">69.5</td>
<td id="A1.T3.1.1.3.2.5" class="ltx_td ltx_align_right">61.3</td>
<td id="A1.T3.1.1.3.2.6" class="ltx_td ltx_align_right">70.2</td>
<td id="A1.T3.1.1.3.2.7" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.3.2.7.1" class="ltx_text" style="background-color:#E6E6E6;">69.9</span></td>
</tr>
<tr id="A1.T3.1.1.4.3" class="ltx_tr">
<td id="A1.T3.1.1.4.3.1" class="ltx_td ltx_align_left ltx_border_r">W8A8-sq</td>
<td id="A1.T3.1.1.4.3.2" class="ltx_td ltx_align_right">73.4</td>
<td id="A1.T3.1.1.4.3.3" class="ltx_td ltx_align_right">73.8</td>
<td id="A1.T3.1.1.4.3.4" class="ltx_td ltx_align_right">69.6</td>
<td id="A1.T3.1.1.4.3.5" class="ltx_td ltx_align_right">62.9</td>
<td id="A1.T3.1.1.4.3.6" class="ltx_td ltx_align_right">67.7</td>
<td id="A1.T3.1.1.4.3.7" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.4.3.7.1" class="ltx_text" style="background-color:#E6E6E6;">69.5</span></td>
</tr>
<tr id="A1.T3.1.1.5.4" class="ltx_tr">
<td id="A1.T3.1.1.5.4.1" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T3.1.1.5.4.2" class="ltx_td ltx_align_right">74.1</td>
<td id="A1.T3.1.1.5.4.3" class="ltx_td ltx_align_right">73.9</td>
<td id="A1.T3.1.1.5.4.4" class="ltx_td ltx_align_right">69.8</td>
<td id="A1.T3.1.1.5.4.5" class="ltx_td ltx_align_right">63.4</td>
<td id="A1.T3.1.1.5.4.6" class="ltx_td ltx_align_right">68.0</td>
<td id="A1.T3.1.1.5.4.7" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.5.4.7.1" class="ltx_text" style="background-color:#E6E6E6;">69.8</span></td>
</tr>
<tr id="A1.T3.1.1.6.5" class="ltx_tr">
<td id="A1.T3.1.1.6.5.1" class="ltx_td ltx_align_left ltx_border_r">W4-g</td>
<td id="A1.T3.1.1.6.5.2" class="ltx_td ltx_align_right">71.2</td>
<td id="A1.T3.1.1.6.5.3" class="ltx_td ltx_align_right">75.7</td>
<td id="A1.T3.1.1.6.5.4" class="ltx_td ltx_align_right">69.0</td>
<td id="A1.T3.1.1.6.5.5" class="ltx_td ltx_align_right">58.0</td>
<td id="A1.T3.1.1.6.5.6" class="ltx_td ltx_align_right">68.9</td>
<td id="A1.T3.1.1.6.5.7" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.6.5.7.1" class="ltx_text" style="background-color:#E6E6E6;">68.6</span></td>
</tr>
<tr id="A1.T3.1.1.7.6" class="ltx_tr">
<td id="A1.T3.1.1.7.6.1" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T3.1.1.7.6.2" class="ltx_td ltx_align_right">64.6</td>
<td id="A1.T3.1.1.7.6.3" class="ltx_td ltx_align_right">71.3</td>
<td id="A1.T3.1.1.7.6.4" class="ltx_td ltx_align_right">66.5</td>
<td id="A1.T3.1.1.7.6.5" class="ltx_td ltx_align_right">56.1</td>
<td id="A1.T3.1.1.7.6.6" class="ltx_td ltx_align_right">63.5</td>
<td id="A1.T3.1.1.7.6.7" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.7.6.7.1" class="ltx_text" style="background-color:#E6E6E6;">64.4</span></td>
</tr>
<tr id="A1.T3.1.1.8.7" class="ltx_tr">
<td id="A1.T3.1.1.8.7.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="4"><span id="A1.T3.1.1.8.7.1.1" class="ltx_text">35B</span></td>
<td id="A1.T3.1.1.8.7.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T3.1.1.8.7.3" class="ltx_td ltx_align_right ltx_border_t">56.6</td>
<td id="A1.T3.1.1.8.7.4" class="ltx_td ltx_align_right ltx_border_t">57.3</td>
<td id="A1.T3.1.1.8.7.5" class="ltx_td ltx_align_right ltx_border_t">51.8</td>
<td id="A1.T3.1.1.8.7.6" class="ltx_td ltx_align_right ltx_border_t">38.8</td>
<td id="A1.T3.1.1.8.7.7" class="ltx_td ltx_align_right ltx_border_t">44.4</td>
<td id="A1.T3.1.1.8.7.8" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.8.7.8.1" class="ltx_text" style="background-color:#E6E6E6;">49.8</span></td>
</tr>
<tr id="A1.T3.1.1.9.8" class="ltx_tr">
<td id="A1.T3.1.1.9.8.1" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T3.1.1.9.8.2" class="ltx_td ltx_align_right">55.9</td>
<td id="A1.T3.1.1.9.8.3" class="ltx_td ltx_align_right">56.6</td>
<td id="A1.T3.1.1.9.8.4" class="ltx_td ltx_align_right">52.1</td>
<td id="A1.T3.1.1.9.8.5" class="ltx_td ltx_align_right">37.4</td>
<td id="A1.T3.1.1.9.8.6" class="ltx_td ltx_align_right">45.1</td>
<td id="A1.T3.1.1.9.8.7" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.9.8.7.1" class="ltx_text" style="background-color:#E6E6E6;">49.4</span></td>
</tr>
<tr id="A1.T3.1.1.10.9" class="ltx_tr">
<td id="A1.T3.1.1.10.9.1" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T3.1.1.10.9.2" class="ltx_td ltx_align_right">54.2</td>
<td id="A1.T3.1.1.10.9.3" class="ltx_td ltx_align_right">53.4</td>
<td id="A1.T3.1.1.10.9.4" class="ltx_td ltx_align_right">49.9</td>
<td id="A1.T3.1.1.10.9.5" class="ltx_td ltx_align_right">35.8</td>
<td id="A1.T3.1.1.10.9.6" class="ltx_td ltx_align_right">42.0</td>
<td id="A1.T3.1.1.10.9.7" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.10.9.7.1" class="ltx_text" style="background-color:#E6E6E6;">47.1</span></td>
</tr>
<tr id="A1.T3.1.1.11.10" class="ltx_tr">
<td id="A1.T3.1.1.11.10.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4-g</td>
<td id="A1.T3.1.1.11.10.2" class="ltx_td ltx_align_right ltx_border_bb">47.2</td>
<td id="A1.T3.1.1.11.10.3" class="ltx_td ltx_align_right ltx_border_bb">51.0</td>
<td id="A1.T3.1.1.11.10.4" class="ltx_td ltx_align_right ltx_border_bb">47.1</td>
<td id="A1.T3.1.1.11.10.5" class="ltx_td ltx_align_right ltx_border_bb">34.3</td>
<td id="A1.T3.1.1.11.10.6" class="ltx_td ltx_align_right ltx_border_bb">36.7</td>
<td id="A1.T3.1.1.11.10.7" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;"><span id="A1.T3.1.1.11.10.7.1" class="ltx_text" style="background-color:#E6E6E6;">43.3</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A3: </span><span id="A1.T3.3.1" class="ltx_text ltx_font_bold">Command model MGSM results.</span> (Acc.)</figcaption>
</figure>
<figure id="A1.T4" class="ltx_table">
<div id="A1.T4.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:160.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-2.2pt,0.8pt) scale(0.99006121068977,0.99006121068977) ;">
<table id="A1.T4.1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T4.1.1.1" class="ltx_tr">
<td id="A1.T4.1.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="A1.T4.1.1.1.3" class="ltx_td ltx_border_r ltx_border_tt"></td>
<td id="A1.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T4.1.1.1.4.1" class="ltx_text ltx_font_bold">de</span></td>
<td id="A1.T4.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T4.1.1.1.5.1" class="ltx_text ltx_font_bold">es</span></td>
<td id="A1.T4.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T4.1.1.1.6.1" class="ltx_text ltx_font_bold">fr</span></td>
<td id="A1.T4.1.1.1.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T4.1.1.1.7.1" class="ltx_text ltx_font_bold">ja</span></td>
<td id="A1.T4.1.1.1.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="A1.T4.1.1.1.8.1" class="ltx_text ltx_font_bold">zh</span></td>
<td id="A1.T4.1.1.1.9" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T4.1.1.1.9.1" class="ltx_text ltx_font_bold">Avg</span></td>
<td id="A1.T4.1.1.1.10" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T4.1.1.1.10.1" class="ltx_text ltx_font_bold">Ltn/IE</span></td>
<td id="A1.T4.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt"><math id="A1.T4.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T4.1.1.1.1.m1.1a"><mo id="A1.T4.1.1.1.1.m1.1.1" xref="A1.T4.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T4.1.1.1.1.m1.1b"><not id="A1.T4.1.1.1.1.m1.1.1.cmml" xref="A1.T4.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.1.1.1.1.m1.1c">\neg</annotation></semantics></math></td>
</tr>
<tr id="A1.T4.1.1.2.1" class="ltx_tr">
<td id="A1.T4.1.1.2.1.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T4.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T4.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F7FCFA;"><span id="A1.T4.1.1.2.1.3.1" class="ltx_text" style="background-color:#F7FCFA;">0.3%</span></td>
<td id="A1.T4.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE6E4;"><span id="A1.T4.1.1.2.1.4.1" class="ltx_text" style="background-color:#FAE6E4;">-0.9%</span></td>
<td id="A1.T4.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F7D5D2;"><span id="A1.T4.1.1.2.1.5.1" class="ltx_text" style="background-color:#F7D5D2;">-1.6%</span></td>
<td id="A1.T4.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F1B9B4;"><span id="A1.T4.1.1.2.1.6.1" class="ltx_text" style="background-color:#F1B9B4;">-2.7%</span></td>
<td id="A1.T4.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FEFCFB;"><span id="A1.T4.1.1.2.1.7.1" class="ltx_text" style="background-color:#FEFCFB;">-0.1%</span></td>
<td id="A1.T4.1.1.2.1.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#FAE4E3;"><span id="A1.T4.1.1.2.1.8.1" class="ltx_text" style="background-color:#FAE4E3;">-1.0%</span></td>
<td id="A1.T4.1.1.2.1.9" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FBEBEA;"><span id="A1.T4.1.1.2.1.9.1" class="ltx_text" style="background-color:#FBEBEA;">-0.7%</span></td>
<td id="A1.T4.1.1.2.1.10" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#F8DAD8;"><span id="A1.T4.1.1.2.1.10.1" class="ltx_text" style="background-color:#F8DAD8;">-1.4%</span></td>
</tr>
<tr id="A1.T4.1.1.3.2" class="ltx_tr">
<td id="A1.T4.1.1.3.2.1" class="ltx_td"></td>
<td id="A1.T4.1.1.3.2.2" class="ltx_td ltx_align_left ltx_border_r">W8A8-sq</td>
<td id="A1.T4.1.1.3.2.3" class="ltx_td ltx_align_right" style="background-color:#E3F4EC;"><span id="A1.T4.1.1.3.2.3.1" class="ltx_text" style="background-color:#E3F4EC;">1.1%</span></td>
<td id="A1.T4.1.1.3.2.4" class="ltx_td ltx_align_right" style="background-color:#EC9F98;"><span id="A1.T4.1.1.3.2.4.1" class="ltx_text" style="background-color:#EC9F98;">-3.7%</span></td>
<td id="A1.T4.1.1.3.2.5" class="ltx_td ltx_align_right" style="background-color:#F7D8D5;"><span id="A1.T4.1.1.3.2.5.1" class="ltx_text" style="background-color:#F7D8D5;">-1.5%</span></td>
<td id="A1.T4.1.1.3.2.6" class="ltx_td ltx_align_right" style="background-color:#FEFBFB;"><span id="A1.T4.1.1.3.2.6.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="A1.T4.1.1.3.2.7" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#EC9F98;"><span id="A1.T4.1.1.3.2.7.1" class="ltx_text" style="background-color:#EC9F98;">-3.6%</span></td>
<td id="A1.T4.1.1.3.2.8" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#F6D4D1;"><span id="A1.T4.1.1.3.2.8.1" class="ltx_text" style="background-color:#F6D4D1;">-1.6%</span></td>
<td id="A1.T4.1.1.3.2.9" class="ltx_td ltx_align_center" style="background-color:#F8DBD9;"><span id="A1.T4.1.1.3.2.9.1" class="ltx_text" style="background-color:#F8DBD9;">-1.3%</span></td>
<td id="A1.T4.1.1.3.2.10" class="ltx_td ltx_align_center" style="background-color:#F5CDCA;"><span id="A1.T4.1.1.3.2.10.1" class="ltx_text" style="background-color:#F5CDCA;">-1.9%</span></td>
</tr>
<tr id="A1.T4.1.1.4.3" class="ltx_tr">
<td id="A1.T4.1.1.4.3.1" class="ltx_td"></td>
<td id="A1.T4.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T4.1.1.4.3.3" class="ltx_td ltx_align_right" style="background-color:#C9EADA;"><span id="A1.T4.1.1.4.3.3.1" class="ltx_text" style="background-color:#C9EADA;">2.1%</span></td>
<td id="A1.T4.1.1.4.3.4" class="ltx_td ltx_align_right" style="background-color:#EDA29B;"><span id="A1.T4.1.1.4.3.4.1" class="ltx_text" style="background-color:#EDA29B;">-3.5%</span></td>
<td id="A1.T4.1.1.4.3.5" class="ltx_td ltx_align_right" style="background-color:#F9E1DF;"><span id="A1.T4.1.1.4.3.5.1" class="ltx_text" style="background-color:#F9E1DF;">-1.1%</span></td>
<td id="A1.T4.1.1.4.3.6" class="ltx_td ltx_align_right" style="background-color:#EFF9F4;"><span id="A1.T4.1.1.4.3.6.1" class="ltx_text" style="background-color:#EFF9F4;">0.6%</span></td>
<td id="A1.T4.1.1.4.3.7" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#EFABA5;"><span id="A1.T4.1.1.4.3.7.1" class="ltx_text" style="background-color:#EFABA5;">-3.2%</span></td>
<td id="A1.T4.1.1.4.3.8" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#F9E4E2;"><span id="A1.T4.1.1.4.3.8.1" class="ltx_text" style="background-color:#F9E4E2;">-1.0%</span></td>
<td id="A1.T4.1.1.4.3.9" class="ltx_td ltx_align_center" style="background-color:#FAE8E6;"><span id="A1.T4.1.1.4.3.9.1" class="ltx_text" style="background-color:#FAE8E6;">-0.9%</span></td>
<td id="A1.T4.1.1.4.3.10" class="ltx_td ltx_align_center" style="background-color:#F8DDDB;"><span id="A1.T4.1.1.4.3.10.1" class="ltx_text" style="background-color:#F8DDDB;">-1.3%</span></td>
</tr>
<tr id="A1.T4.1.1.5.4" class="ltx_tr">
<td id="A1.T4.1.1.5.4.1" class="ltx_td"></td>
<td id="A1.T4.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_r">W4-g</td>
<td id="A1.T4.1.1.5.4.3" class="ltx_td ltx_align_right" style="background-color:#F5CDCA;"><span id="A1.T4.1.1.5.4.3.1" class="ltx_text" style="background-color:#F5CDCA;">-1.9%</span></td>
<td id="A1.T4.1.1.5.4.4" class="ltx_td ltx_align_right" style="background-color:#F8DEDB;"><span id="A1.T4.1.1.5.4.4.1" class="ltx_text" style="background-color:#F8DEDB;">-1.3%</span></td>
<td id="A1.T4.1.1.5.4.5" class="ltx_td ltx_align_right" style="background-color:#F3C3BF;"><span id="A1.T4.1.1.5.4.5.1" class="ltx_text" style="background-color:#F3C3BF;">-2.3%</span></td>
<td id="A1.T4.1.1.5.4.6" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T4.1.1.5.4.6.1" class="ltx_text" style="background-color:#E67C73;">-7.9%</span></td>
<td id="A1.T4.1.1.5.4.7" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F5CCC8;"><span id="A1.T4.1.1.5.4.7.1" class="ltx_text" style="background-color:#F5CCC8;">-1.9%</span></td>
<td id="A1.T4.1.1.5.4.8" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#EFAFA9;"><span id="A1.T4.1.1.5.4.8.1" class="ltx_text" style="background-color:#EFAFA9;">-3.0%</span></td>
<td id="A1.T4.1.1.5.4.9" class="ltx_td ltx_align_center" style="background-color:#F6CFCC;"><span id="A1.T4.1.1.5.4.9.1" class="ltx_text" style="background-color:#F6CFCC;">-1.8%</span></td>
<td id="A1.T4.1.1.5.4.10" class="ltx_td ltx_align_center" style="background-color:#E67E75;"><span id="A1.T4.1.1.5.4.10.1" class="ltx_text" style="background-color:#E67E75;">-4.9%</span></td>
</tr>
<tr id="A1.T4.1.1.6.5" class="ltx_tr">
<td id="A1.T4.1.1.6.5.1" class="ltx_td ltx_align_left"><span id="A1.T4.1.1.6.5.1.1" class="ltx_text">103B</span></td>
<td id="A1.T4.1.1.6.5.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T4.1.1.6.5.3" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T4.1.1.6.5.3.1" class="ltx_text" style="background-color:#E67C73;">-11.0%</span></td>
<td id="A1.T4.1.1.6.5.4" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T4.1.1.6.5.4.1" class="ltx_text" style="background-color:#E67C73;">-7.0%</span></td>
<td id="A1.T4.1.1.6.5.5" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T4.1.1.6.5.5.1" class="ltx_text" style="background-color:#E67C73;">-5.9%</span></td>
<td id="A1.T4.1.1.6.5.6" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T4.1.1.6.5.6.1" class="ltx_text" style="background-color:#E67C73;">-10.9%</span></td>
<td id="A1.T4.1.1.6.5.7" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#E67C73;"><span id="A1.T4.1.1.6.5.7.1" class="ltx_text" style="background-color:#E67C73;">-9.6%</span></td>
<td id="A1.T4.1.1.6.5.8" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E67C73;"><span id="A1.T4.1.1.6.5.8.1" class="ltx_text" style="background-color:#E67C73;">-8.8%</span></td>
<td id="A1.T4.1.1.6.5.9" class="ltx_td ltx_align_center" style="background-color:#E67C73;"><span id="A1.T4.1.1.6.5.9.1" class="ltx_text" style="background-color:#E67C73;">-8.0%</span></td>
<td id="A1.T4.1.1.6.5.10" class="ltx_td ltx_align_center" style="background-color:#E67C73;"><span id="A1.T4.1.1.6.5.10.1" class="ltx_text" style="background-color:#E67C73;">-10.2%</span></td>
</tr>
<tr id="A1.T4.1.1.7.6" class="ltx_tr">
<td id="A1.T4.1.1.7.6.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T4.1.1.7.6.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T4.1.1.7.6.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8DDDB;"><span id="A1.T4.1.1.7.6.3.1" class="ltx_text" style="background-color:#F8DDDB;">-1.3%</span></td>
<td id="A1.T4.1.1.7.6.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F9E1DF;"><span id="A1.T4.1.1.7.6.4.1" class="ltx_text" style="background-color:#F9E1DF;">-1.1%</span></td>
<td id="A1.T4.1.1.7.6.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F0F9F4;"><span id="A1.T4.1.1.7.6.5.1" class="ltx_text" style="background-color:#F0F9F4;">0.6%</span></td>
<td id="A1.T4.1.1.7.6.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#EC9D97;"><span id="A1.T4.1.1.7.6.6.1" class="ltx_text" style="background-color:#EC9D97;">-3.7%</span></td>
<td id="A1.T4.1.1.7.6.7" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#D6EFE2;"><span id="A1.T4.1.1.7.6.7.1" class="ltx_text" style="background-color:#D6EFE2;">1.6%</span></td>
<td id="A1.T4.1.1.7.6.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#FBEAE9;"><span id="A1.T4.1.1.7.6.8.1" class="ltx_text" style="background-color:#FBEAE9;">-0.8%</span></td>
<td id="A1.T4.1.1.7.6.9" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FCEFEE;"><span id="A1.T4.1.1.7.6.9.1" class="ltx_text" style="background-color:#FCEFEE;">-0.6%</span></td>
<td id="A1.T4.1.1.7.6.10" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#F9E3E1;"><span id="A1.T4.1.1.7.6.10.1" class="ltx_text" style="background-color:#F9E3E1;">-1.0%</span></td>
</tr>
<tr id="A1.T4.1.1.8.7" class="ltx_tr">
<td id="A1.T4.1.1.8.7.1" class="ltx_td"></td>
<td id="A1.T4.1.1.8.7.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T4.1.1.8.7.3" class="ltx_td ltx_align_right" style="background-color:#E98C84;"><span id="A1.T4.1.1.8.7.3.1" class="ltx_text" style="background-color:#E98C84;">-4.4%</span></td>
<td id="A1.T4.1.1.8.7.4" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T4.1.1.8.7.4.1" class="ltx_text" style="background-color:#E67C73;">-6.8%</span></td>
<td id="A1.T4.1.1.8.7.5" class="ltx_td ltx_align_right" style="background-color:#EDA19B;"><span id="A1.T4.1.1.8.7.5.1" class="ltx_text" style="background-color:#EDA19B;">-3.6%</span></td>
<td id="A1.T4.1.1.8.7.6" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T4.1.1.8.7.6.1" class="ltx_text" style="background-color:#E67C73;">-7.6%</span></td>
<td id="A1.T4.1.1.8.7.7" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#E67C73;"><span id="A1.T4.1.1.8.7.7.1" class="ltx_text" style="background-color:#E67C73;">-5.4%</span></td>
<td id="A1.T4.1.1.8.7.8" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E67C73;"><span id="A1.T4.1.1.8.7.8.1" class="ltx_text" style="background-color:#E67C73;">-5.6%</span></td>
<td id="A1.T4.1.1.8.7.9" class="ltx_td ltx_align_center" style="background-color:#E67D75;"><span id="A1.T4.1.1.8.7.9.1" class="ltx_text" style="background-color:#E67D75;">-4.9%</span></td>
<td id="A1.T4.1.1.8.7.10" class="ltx_td ltx_align_center" style="background-color:#E67C73;"><span id="A1.T4.1.1.8.7.10.1" class="ltx_text" style="background-color:#E67C73;">-6.5%</span></td>
</tr>
<tr id="A1.T4.1.1.9.8" class="ltx_tr">
<td id="A1.T4.1.1.9.8.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A1.T4.1.1.9.8.1.1" class="ltx_text">35B</span></td>
<td id="A1.T4.1.1.9.8.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4-g</td>
<td id="A1.T4.1.1.9.8.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T4.1.1.9.8.3.1" class="ltx_text" style="background-color:#E67C73;">-16.7%</span></td>
<td id="A1.T4.1.1.9.8.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T4.1.1.9.8.4.1" class="ltx_text" style="background-color:#E67C73;">-10.9%</span></td>
<td id="A1.T4.1.1.9.8.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T4.1.1.9.8.5.1" class="ltx_text" style="background-color:#E67C73;">-9.0%</span></td>
<td id="A1.T4.1.1.9.8.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T4.1.1.9.8.6.1" class="ltx_text" style="background-color:#E67C73;">-11.5%</span></td>
<td id="A1.T4.1.1.9.8.7" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#E67C73;"><span id="A1.T4.1.1.9.8.7.1" class="ltx_text" style="background-color:#E67C73;">-17.3%</span></td>
<td id="A1.T4.1.1.9.8.8" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="background-color:#E67C73;"><span id="A1.T4.1.1.9.8.8.1" class="ltx_text" style="background-color:#E67C73;">-13.1%</span></td>
<td id="A1.T4.1.1.9.8.9" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T4.1.1.9.8.9.1" class="ltx_text" style="background-color:#E67C73;">-12.2%</span></td>
<td id="A1.T4.1.1.9.8.10" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T4.1.1.9.8.10.1" class="ltx_text" style="background-color:#E67C73;">-14.4%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A4: </span><span id="A1.T4.4.1" class="ltx_text ltx_font_bold">Relative performance (%<math id="A1.T4.4.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="A1.T4.4.1.m1.1b"><mi mathvariant="normal" id="A1.T4.4.1.m1.1.1" xref="A1.T4.4.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="A1.T4.4.1.m1.1c"><ci id="A1.T4.4.1.m1.1.1.cmml" xref="A1.T4.4.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.4.1.m1.1d">\Delta</annotation></semantics></math>) vs. FP16 for Command Models on MGSM</span>. Ltn/IE: Latin-script/Indo-European languages: de, es, fr. <math id="A1.T4.5.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T4.5.m1.1b"><mo id="A1.T4.5.m1.1.1" xref="A1.T4.5.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T4.5.m1.1c"><not id="A1.T4.5.m1.1.1.cmml" xref="A1.T4.5.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.5.m1.1d">\neg</annotation></semantics></math>: ja, zh.
</figcaption>
</figure>
<figure id="A1.T5" class="ltx_table">
<table id="A1.T5.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T5.1.1.1" class="ltx_tr">
<th id="A1.T5.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="A1.T5.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="A1.T5.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T5.1.1.1.3.1" class="ltx_text ltx_font_bold">Avg (<span id="A1.T5.1.1.1.3.1.1" class="ltx_text ltx_font_italic">-en</span>)</span></th>
<th id="A1.T5.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt"><span id="A1.T5.1.1.1.4.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="A1.T5.1.1.1.5" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">de</th>
<th id="A1.T5.1.1.1.6" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">en</th>
<th id="A1.T5.1.1.1.7" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">es</th>
<th id="A1.T5.1.1.1.8" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">fr</th>
<th id="A1.T5.1.1.1.9" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">ja</th>
<th id="A1.T5.1.1.1.10" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">ru</th>
<th id="A1.T5.1.1.1.11" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">zh</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T5.1.2.1" class="ltx_tr">
<td id="A1.T5.1.2.1.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T5.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T5.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T5.1.2.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">51.2</span></td>
<td id="A1.T5.1.2.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T5.1.2.1.4.1" class="ltx_text" style="background-color:#E6E6E6;">53.7</span></td>
<td id="A1.T5.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.2.1.5.1" class="ltx_text" style="background-color:#FFFFFF;">61.6</span></td>
<td id="A1.T5.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.2.1.6.1" class="ltx_text" style="background-color:#FFFFFF;">68.4</span></td>
<td id="A1.T5.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.2.1.7.1" class="ltx_text" style="background-color:#FFFFFF;">58.4</span></td>
<td id="A1.T5.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.2.1.8.1" class="ltx_text" style="background-color:#FFFFFF;">55.6</span></td>
<td id="A1.T5.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.2.1.9.1" class="ltx_text" style="background-color:#FFFFFF;">22.8</span></td>
<td id="A1.T5.1.2.1.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.2.1.10.1" class="ltx_text" style="background-color:#FFFFFF;">58.0</span></td>
<td id="A1.T5.1.2.1.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.2.1.11.1" class="ltx_text" style="background-color:#FFFFFF;">50.8</span></td>
</tr>
<tr id="A1.T5.1.3.2" class="ltx_tr">
<td id="A1.T5.1.3.2.1" class="ltx_td ltx_align_left">Aya-23-35b</td>
<td id="A1.T5.1.3.2.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T5.1.3.2.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T5.1.3.2.3.1" class="ltx_text" style="background-color:#E6E6E6;">52.1</span></td>
<td id="A1.T5.1.3.2.4" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T5.1.3.2.4.1" class="ltx_text" style="background-color:#E6E6E6;">54.2</span></td>
<td id="A1.T5.1.3.2.5" class="ltx_td ltx_align_right">54.4</td>
<td id="A1.T5.1.3.2.6" class="ltx_td ltx_align_right">66.4</td>
<td id="A1.T5.1.3.2.7" class="ltx_td ltx_align_right">61.2</td>
<td id="A1.T5.1.3.2.8" class="ltx_td ltx_align_right">60.4</td>
<td id="A1.T5.1.3.2.9" class="ltx_td ltx_align_right">24.4</td>
<td id="A1.T5.1.3.2.10" class="ltx_td ltx_align_right">57.2</td>
<td id="A1.T5.1.3.2.11" class="ltx_td ltx_align_right">55.2</td>
</tr>
<tr id="A1.T5.1.4.3" class="ltx_tr">
<td id="A1.T5.1.4.3.1" class="ltx_td"></td>
<td id="A1.T5.1.4.3.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T5.1.4.3.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T5.1.4.3.3.1" class="ltx_text" style="background-color:#E6E6E6;">48.1</span></td>
<td id="A1.T5.1.4.3.4" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T5.1.4.3.4.1" class="ltx_text" style="background-color:#E6E6E6;">50.7</span></td>
<td id="A1.T5.1.4.3.5" class="ltx_td ltx_align_right">58.8</td>
<td id="A1.T5.1.4.3.6" class="ltx_td ltx_align_right">66.0</td>
<td id="A1.T5.1.4.3.7" class="ltx_td ltx_align_right">54.8</td>
<td id="A1.T5.1.4.3.8" class="ltx_td ltx_align_right">54.8</td>
<td id="A1.T5.1.4.3.9" class="ltx_td ltx_align_right">18.4</td>
<td id="A1.T5.1.4.3.10" class="ltx_td ltx_align_right">53.6</td>
<td id="A1.T5.1.4.3.11" class="ltx_td ltx_align_right">48.4</td>
</tr>
<tr id="A1.T5.1.5.4" class="ltx_tr">
<td id="A1.T5.1.5.4.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T5.1.5.4.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T5.1.5.4.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T5.1.5.4.3.1" class="ltx_text" style="background-color:#E6E6E6;">34.7</span></td>
<td id="A1.T5.1.5.4.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T5.1.5.4.4.1" class="ltx_text" style="background-color:#E6E6E6;">36.6</span></td>
<td id="A1.T5.1.5.4.5" class="ltx_td ltx_align_right ltx_border_t">40.4</td>
<td id="A1.T5.1.5.4.6" class="ltx_td ltx_align_right ltx_border_t">48.0</td>
<td id="A1.T5.1.5.4.7" class="ltx_td ltx_align_right ltx_border_t">45.2</td>
<td id="A1.T5.1.5.4.8" class="ltx_td ltx_align_right ltx_border_t">38.8</td>
<td id="A1.T5.1.5.4.9" class="ltx_td ltx_align_right ltx_border_t">12.8</td>
<td id="A1.T5.1.5.4.10" class="ltx_td ltx_align_right ltx_border_t">38.0</td>
<td id="A1.T5.1.5.4.11" class="ltx_td ltx_align_right ltx_border_t">32.8</td>
</tr>
<tr id="A1.T5.1.6.5" class="ltx_tr">
<td id="A1.T5.1.6.5.1" class="ltx_td ltx_align_left">Aya-23-8b</td>
<td id="A1.T5.1.6.5.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T5.1.6.5.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T5.1.6.5.3.1" class="ltx_text" style="background-color:#E6E6E6;">35.4</span></td>
<td id="A1.T5.1.6.5.4" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T5.1.6.5.4.1" class="ltx_text" style="background-color:#E6E6E6;">36.9</span></td>
<td id="A1.T5.1.6.5.5" class="ltx_td ltx_align_right">39.6</td>
<td id="A1.T5.1.6.5.6" class="ltx_td ltx_align_right">45.6</td>
<td id="A1.T5.1.6.5.7" class="ltx_td ltx_align_right">45.6</td>
<td id="A1.T5.1.6.5.8" class="ltx_td ltx_align_right">38.8</td>
<td id="A1.T5.1.6.5.9" class="ltx_td ltx_align_right">13.6</td>
<td id="A1.T5.1.6.5.10" class="ltx_td ltx_align_right">38.8</td>
<td id="A1.T5.1.6.5.11" class="ltx_td ltx_align_right">36.0</td>
</tr>
<tr id="A1.T5.1.7.6" class="ltx_tr">
<td id="A1.T5.1.7.6.1" class="ltx_td ltx_border_bb"></td>
<td id="A1.T5.1.7.6.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4</td>
<td id="A1.T5.1.7.6.3" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;"><span id="A1.T5.1.7.6.3.1" class="ltx_text" style="background-color:#E6E6E6;">32.1</span></td>
<td id="A1.T5.1.7.6.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T5.1.7.6.4.1" class="ltx_text" style="background-color:#E6E6E6;">33.5</span></td>
<td id="A1.T5.1.7.6.5" class="ltx_td ltx_align_right ltx_border_bb">39.6</td>
<td id="A1.T5.1.7.6.6" class="ltx_td ltx_align_right ltx_border_bb">42.4</td>
<td id="A1.T5.1.7.6.7" class="ltx_td ltx_align_right ltx_border_bb">42.0</td>
<td id="A1.T5.1.7.6.8" class="ltx_td ltx_align_right ltx_border_bb">34.0</td>
<td id="A1.T5.1.7.6.9" class="ltx_td ltx_align_right ltx_border_bb">7.2</td>
<td id="A1.T5.1.7.6.10" class="ltx_td ltx_align_right ltx_border_bb">33.6</td>
<td id="A1.T5.1.7.6.11" class="ltx_td ltx_align_right ltx_border_bb">36.0</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A5: </span><span id="A1.T5.3.1" class="ltx_text ltx_font_bold">Aya 23 language-specific results for MGSM (5-shot).</span></figcaption>
</figure>
<figure id="A1.T6" class="ltx_table">
<div id="A1.T6.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:66.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-77.9pt,11.9pt) scale(0.735779417070955,0.735779417070955) ;">
<table id="A1.T6.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T6.1.1.1" class="ltx_tr">
<th id="A1.T6.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="A1.T6.1.1.1.3" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="A1.T6.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T6.1.1.1.4.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T6.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T6.1.1.1.5.1" class="ltx_text ltx_font_bold">en</span></th>
<th id="A1.T6.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T6.1.1.1.6.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T6.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T6.1.1.1.7.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T6.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T6.1.1.1.8.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T6.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T6.1.1.1.9.1" class="ltx_text ltx_font_bold">ru</span></th>
<th id="A1.T6.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_rr ltx_border_tt"><span id="A1.T6.1.1.1.10.1" class="ltx_text ltx_font_bold">zh</span></th>
<th id="A1.T6.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T6.1.1.1.11.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="A1.T6.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T6.1.1.1.12.1" class="ltx_text ltx_font_bold">Avg (<span id="A1.T6.1.1.1.12.1.1" class="ltx_text ltx_font_italic">-en</span>)</span></th>
<th id="A1.T6.1.1.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T6.1.1.1.13.1" class="ltx_text ltx_font_bold">Ltn/IE</span></th>
<th id="A1.T6.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="A1.T6.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T6.1.1.1.1.m1.1a"><mo id="A1.T6.1.1.1.1.m1.1.1" xref="A1.T6.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T6.1.1.1.1.m1.1b"><not id="A1.T6.1.1.1.1.m1.1.1.cmml" xref="A1.T6.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.1.1.1.1.m1.1c">\neg</annotation></semantics></math></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T6.1.1.2.1" class="ltx_tr">
<td id="A1.T6.1.1.2.1.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T6.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_t">W8</td>
<td id="A1.T6.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#EB9891;"><span id="A1.T6.1.1.2.1.3.1" class="ltx_text" style="background-color:#EB9891;">-11.7%</span></td>
<td id="A1.T6.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE5E3;"><span id="A1.T6.1.1.2.1.4.1" class="ltx_text" style="background-color:#FAE5E3;">-2.9%</span></td>
<td id="A1.T6.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#ADDEC6;"><span id="A1.T6.1.1.2.1.5.1" class="ltx_text" style="background-color:#ADDEC6;">4.8%</span></td>
<td id="A1.T6.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#6BC398;"><span id="A1.T6.1.1.2.1.6.1" class="ltx_text" style="background-color:#6BC398;">8.6%</span></td>
<td id="A1.T6.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#87CFAB;"><span id="A1.T6.1.1.2.1.7.1" class="ltx_text" style="background-color:#87CFAB;">7.0%</span></td>
<td id="A1.T6.1.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF2F2;"><span id="A1.T6.1.1.2.1.8.1" class="ltx_text" style="background-color:#FCF2F2;">-1.4%</span></td>
<td id="A1.T6.1.1.2.1.9" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#6AC398;"><span id="A1.T6.1.1.2.1.9.1" class="ltx_text" style="background-color:#6AC398;">8.7%</span></td>
<td id="A1.T6.1.1.2.1.10" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#DFF2E9;"><span id="A1.T6.1.1.2.1.10.1" class="ltx_text" style="background-color:#DFF2E9;">1.9%</span></td>
<td id="A1.T6.1.1.2.1.11" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#D1EDDF;"><span id="A1.T6.1.1.2.1.11.1" class="ltx_text" style="background-color:#D1EDDF;">2.7%</span></td>
<td id="A1.T6.1.1.2.1.12" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T6.1.1.2.1.12.1" class="ltx_text" style="background-color:#FEFCFC;">-0.3%</span></td>
<td id="A1.T6.1.1.2.1.13" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#ADDEC6;"><span id="A1.T6.1.1.2.1.13.1" class="ltx_text" style="background-color:#ADDEC6;">4.8%</span></td>
</tr>
<tr id="A1.T6.1.1.3.2" class="ltx_tr">
<td id="A1.T6.1.1.3.2.1" class="ltx_td ltx_align_left"><span id="A1.T6.1.1.3.2.1.1" class="ltx_text">Aya-23-35b</span></td>
<td id="A1.T6.1.1.3.2.2" class="ltx_td ltx_align_left">W4</td>
<td id="A1.T6.1.1.3.2.3" class="ltx_td ltx_align_right" style="background-color:#F7D7D4;"><span id="A1.T6.1.1.3.2.3.1" class="ltx_text" style="background-color:#F7D7D4;">-4.5%</span></td>
<td id="A1.T6.1.1.3.2.4" class="ltx_td ltx_align_right" style="background-color:#F9E0DE;"><span id="A1.T6.1.1.3.2.4.1" class="ltx_text" style="background-color:#F9E0DE;">-3.5%</span></td>
<td id="A1.T6.1.1.3.2.5" class="ltx_td ltx_align_right" style="background-color:#F4C9C5;"><span id="A1.T6.1.1.3.2.5.1" class="ltx_text" style="background-color:#F4C9C5;">-6.2%</span></td>
<td id="A1.T6.1.1.3.2.6" class="ltx_td ltx_align_right" style="background-color:#FCF2F1;"><span id="A1.T6.1.1.3.2.6.1" class="ltx_text" style="background-color:#FCF2F1;">-1.4%</span></td>
<td id="A1.T6.1.1.3.2.7" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T6.1.1.3.2.7.1" class="ltx_text" style="background-color:#E67C73;">-19.3%</span></td>
<td id="A1.T6.1.1.3.2.8" class="ltx_td ltx_align_right" style="background-color:#F2BCB8;"><span id="A1.T6.1.1.3.2.8.1" class="ltx_text" style="background-color:#F2BCB8;">-7.6%</span></td>
<td id="A1.T6.1.1.3.2.9" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F7D5D2;"><span id="A1.T6.1.1.3.2.9.1" class="ltx_text" style="background-color:#F7D5D2;">-4.7%</span></td>
<td id="A1.T6.1.1.3.2.10" class="ltx_td ltx_align_center" style="background-color:#F3C4BF;"><span id="A1.T6.1.1.3.2.10.1" class="ltx_text" style="background-color:#F3C4BF;">-6.8%</span></td>
<td id="A1.T6.1.1.3.2.11" class="ltx_td ltx_align_center" style="background-color:#F2BFBA;"><span id="A1.T6.1.1.3.2.11.1" class="ltx_text" style="background-color:#F2BFBA;">-7.3%</span></td>
<td id="A1.T6.1.1.3.2.12" class="ltx_td ltx_align_center" style="background-color:#F8DCDA;"><span id="A1.T6.1.1.3.2.12.1" class="ltx_text" style="background-color:#F8DCDA;">-3.9%</span></td>
<td id="A1.T6.1.1.3.2.13" class="ltx_td ltx_align_center" style="background-color:#EDA29C;"><span id="A1.T6.1.1.3.2.13.1" class="ltx_text" style="background-color:#EDA29C;">-10.5%</span></td>
</tr>
<tr id="A1.T6.1.1.4.3" class="ltx_tr">
<td id="A1.T6.1.1.4.3.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T6.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_t">W8</td>
<td id="A1.T6.1.1.4.3.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEDEC;"><span id="A1.T6.1.1.4.3.3.1" class="ltx_text" style="background-color:#FBEDEC;">-2.0%</span></td>
<td id="A1.T6.1.1.4.3.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F6D3D0;"><span id="A1.T6.1.1.4.3.4.1" class="ltx_text" style="background-color:#F6D3D0;">-5.0%</span></td>
<td id="A1.T6.1.1.4.3.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F0F9F5;"><span id="A1.T6.1.1.4.3.5.1" class="ltx_text" style="background-color:#F0F9F5;">0.9%</span></td>
<td id="A1.T6.1.1.4.3.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T6.1.1.4.3.6.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T6.1.1.4.3.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#94D4B5;"><span id="A1.T6.1.1.4.3.7.1" class="ltx_text" style="background-color:#94D4B5;">6.2%</span></td>
<td id="A1.T6.1.1.4.3.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#DBF1E6;"><span id="A1.T6.1.1.4.3.8.1" class="ltx_text" style="background-color:#DBF1E6;">2.1%</span></td>
<td id="A1.T6.1.1.4.3.9" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#57BB8A;"><span id="A1.T6.1.1.4.3.9.1" class="ltx_text" style="background-color:#57BB8A;">9.8%</span></td>
<td id="A1.T6.1.1.4.3.10" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E2F4EB;"><span id="A1.T6.1.1.4.3.10.1" class="ltx_text" style="background-color:#E2F4EB;">1.7%</span></td>
<td id="A1.T6.1.1.4.3.11" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#CFECDD;"><span id="A1.T6.1.1.4.3.11.1" class="ltx_text" style="background-color:#CFECDD;">2.8%</span></td>
<td id="A1.T6.1.1.4.3.12" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FCF1F0;"><span id="A1.T6.1.1.4.3.12.1" class="ltx_text" style="background-color:#FCF1F0;">-1.5%</span></td>
<td id="A1.T6.1.1.4.3.13" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#98D5B7;"><span id="A1.T6.1.1.4.3.13.1" class="ltx_text" style="background-color:#98D5B7;">6.0%</span></td>
</tr>
<tr id="A1.T6.1.1.5.4" class="ltx_tr">
<td id="A1.T6.1.1.5.4.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A1.T6.1.1.5.4.1.1" class="ltx_text">Aya-23-8b</span></td>
<td id="A1.T6.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_bb">W4</td>
<td id="A1.T6.1.1.5.4.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FBEDEC;"><span id="A1.T6.1.1.5.4.3.1" class="ltx_text" style="background-color:#FBEDEC;">-2.0%</span></td>
<td id="A1.T6.1.1.5.4.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EB9992;"><span id="A1.T6.1.1.5.4.4.1" class="ltx_text" style="background-color:#EB9992;">-11.7%</span></td>
<td id="A1.T6.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C1BC;"><span id="A1.T6.1.1.5.4.5.1" class="ltx_text" style="background-color:#F3C1BC;">-7.1%</span></td>
<td id="A1.T6.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EA928B;"><span id="A1.T6.1.1.5.4.6.1" class="ltx_text" style="background-color:#EA928B;">-12.4%</span></td>
<td id="A1.T6.1.1.5.4.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T6.1.1.5.4.7.1" class="ltx_text" style="background-color:#E67C73;">-43.8%</span></td>
<td id="A1.T6.1.1.5.4.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EB9992;"><span id="A1.T6.1.1.5.4.8.1" class="ltx_text" style="background-color:#EB9992;">-11.6%</span></td>
<td id="A1.T6.1.1.5.4.9" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#57BB8A;"><span id="A1.T6.1.1.5.4.9.1" class="ltx_text" style="background-color:#57BB8A;">9.8%</span></td>
<td id="A1.T6.1.1.5.4.10" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#EC9C96;"><span id="A1.T6.1.1.5.4.10.1" class="ltx_text" style="background-color:#EC9C96;">-11.2%</span></td>
<td id="A1.T6.1.1.5.4.11" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#EC9C96;"><span id="A1.T6.1.1.5.4.11.1" class="ltx_text" style="background-color:#EC9C96;">-11.2%</span></td>
<td id="A1.T6.1.1.5.4.12" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#F1B6B1;"><span id="A1.T6.1.1.5.4.12.1" class="ltx_text" style="background-color:#F1B6B1;">-8.3%</span></td>
<td id="A1.T6.1.1.5.4.13" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T6.1.1.5.4.13.1" class="ltx_text" style="background-color:#E67C73;">-15.2%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A6: </span><span id="A1.T6.4.1" class="ltx_text ltx_font_bold">Relative performance (<math id="A1.T6.4.1.m1.1" class="ltx_math_unparsed" alttext="\%\Delta" display="inline"><semantics id="A1.T6.4.1.m1.1b"><mrow id="A1.T6.4.1.m1.1c"><mo id="A1.T6.4.1.m1.1.1">%</mo><mi mathvariant="normal" id="A1.T6.4.1.m1.1.2">Δ</mi></mrow><annotation encoding="application/x-tex" id="A1.T6.4.1.m1.1d">\%\Delta</annotation></semantics></math>) vs. FP16 for Aya Models on MGSM (5-shot).</span> Ltn/IE are Latin-script/Indo-European languages: de, en, es, fr. <math id="A1.T6.5.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T6.5.m1.1b"><mo id="A1.T6.5.m1.1.1" xref="A1.T6.5.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T6.5.m1.1c"><not id="A1.T6.5.m1.1.1.cmml" xref="A1.T6.5.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T6.5.m1.1d">\neg</annotation></semantics></math> are the rest: ja, ru, zh.</figcaption>
</figure>
<figure id="A1.T7" class="ltx_table">
<div id="A1.T7.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:62.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-222.5pt,31.9pt) scale(0.493496969820951,0.493496969820951) ;">
<table id="A1.T7.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T7.1.1.1.1" class="ltx_tr">
<th id="A1.T7.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="A1.T7.1.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="A1.T7.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Avg (<span id="A1.T7.1.1.1.1.3.1.1" class="ltx_text ltx_font_italic">-en</span>)</span></th>
<th id="A1.T7.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt"><span id="A1.T7.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="A1.T7.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.5.1" class="ltx_text ltx_font_bold">ar</span></th>
<th id="A1.T7.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.6.1" class="ltx_text ltx_font_bold">cs</span></th>
<th id="A1.T7.1.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.7.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T7.1.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.8.1" class="ltx_text ltx_font_bold">el</span></th>
<th id="A1.T7.1.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.9.1" class="ltx_text ltx_font_bold">en</span></th>
<th id="A1.T7.1.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.10.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T7.1.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.11.1" class="ltx_text ltx_font_bold">fa</span></th>
<th id="A1.T7.1.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.12.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T7.1.1.1.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.13.1" class="ltx_text ltx_font_bold">hi</span></th>
<th id="A1.T7.1.1.1.1.14" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.14.1" class="ltx_text ltx_font_bold">id</span></th>
<th id="A1.T7.1.1.1.1.15" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.15.1" class="ltx_text ltx_font_bold">it</span></th>
<th id="A1.T7.1.1.1.1.16" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.16.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T7.1.1.1.1.17" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.17.1" class="ltx_text ltx_font_bold">ko</span></th>
<th id="A1.T7.1.1.1.1.18" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.18.1" class="ltx_text ltx_font_bold">nl</span></th>
<th id="A1.T7.1.1.1.1.19" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.19.1" class="ltx_text ltx_font_bold">pl</span></th>
<th id="A1.T7.1.1.1.1.20" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.20.1" class="ltx_text ltx_font_bold">pt</span></th>
<th id="A1.T7.1.1.1.1.21" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.21.1" class="ltx_text ltx_font_bold">ro</span></th>
<th id="A1.T7.1.1.1.1.22" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.22.1" class="ltx_text ltx_font_bold">ru</span></th>
<th id="A1.T7.1.1.1.1.23" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.23.1" class="ltx_text ltx_font_bold">tr</span></th>
<th id="A1.T7.1.1.1.1.24" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.24.1" class="ltx_text ltx_font_bold">uk</span></th>
<th id="A1.T7.1.1.1.1.25" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.25.1" class="ltx_text ltx_font_bold">vi</span></th>
<th id="A1.T7.1.1.1.1.26" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T7.1.1.1.1.26.1" class="ltx_text ltx_font_bold">zh</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T7.1.1.2.1" class="ltx_tr">
<td id="A1.T7.1.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="3"><span id="A1.T7.1.1.2.1.1.1" class="ltx_text">35b</span></td>
<td id="A1.T7.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T7.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.2.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">77.6</span></td>
<td id="A1.T7.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.2.1.4.1" class="ltx_text" style="background-color:#E6E6E6;">77.9</span></td>
<td id="A1.T7.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t">78.9</td>
<td id="A1.T7.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t">78.2</td>
<td id="A1.T7.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t">77.1</td>
<td id="A1.T7.1.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t">76.4</td>
<td id="A1.T7.1.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t">84.7</td>
<td id="A1.T7.1.1.2.1.10" class="ltx_td ltx_align_right ltx_border_t">81.0</td>
<td id="A1.T7.1.1.2.1.11" class="ltx_td ltx_align_right ltx_border_t">75.8</td>
<td id="A1.T7.1.1.2.1.12" class="ltx_td ltx_align_right ltx_border_t">81.9</td>
<td id="A1.T7.1.1.2.1.13" class="ltx_td ltx_align_right ltx_border_t">65.6</td>
<td id="A1.T7.1.1.2.1.14" class="ltx_td ltx_align_right ltx_border_t">77.8</td>
<td id="A1.T7.1.1.2.1.15" class="ltx_td ltx_align_right ltx_border_t">79.8</td>
<td id="A1.T7.1.1.2.1.16" class="ltx_td ltx_align_right ltx_border_t">75.9</td>
<td id="A1.T7.1.1.2.1.17" class="ltx_td ltx_align_right ltx_border_t">73.3</td>
<td id="A1.T7.1.1.2.1.18" class="ltx_td ltx_align_right ltx_border_t">77.7</td>
<td id="A1.T7.1.1.2.1.19" class="ltx_td ltx_align_right ltx_border_t">75.8</td>
<td id="A1.T7.1.1.2.1.20" class="ltx_td ltx_align_right ltx_border_t">83.8</td>
<td id="A1.T7.1.1.2.1.21" class="ltx_td ltx_align_right ltx_border_t">78.9</td>
<td id="A1.T7.1.1.2.1.22" class="ltx_td ltx_align_right ltx_border_t">79.6</td>
<td id="A1.T7.1.1.2.1.23" class="ltx_td ltx_align_right ltx_border_t">74.1</td>
<td id="A1.T7.1.1.2.1.24" class="ltx_td ltx_align_right ltx_border_t">77.6</td>
<td id="A1.T7.1.1.2.1.25" class="ltx_td ltx_align_right ltx_border_t">78.3</td>
<td id="A1.T7.1.1.2.1.26" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">81.2</td>
</tr>
<tr id="A1.T7.1.1.3.2" class="ltx_tr">
<td id="A1.T7.1.1.3.2.1" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T7.1.1.3.2.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.3.2.2.1" class="ltx_text" style="background-color:#E6E6E6;">77.1</span></td>
<td id="A1.T7.1.1.3.2.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.3.2.3.1" class="ltx_text" style="background-color:#E6E6E6;">77.4</span></td>
<td id="A1.T7.1.1.3.2.4" class="ltx_td ltx_align_right">77.3</td>
<td id="A1.T7.1.1.3.2.5" class="ltx_td ltx_align_right">78.8</td>
<td id="A1.T7.1.1.3.2.6" class="ltx_td ltx_align_right">77.2</td>
<td id="A1.T7.1.1.3.2.7" class="ltx_td ltx_align_right">76.6</td>
<td id="A1.T7.1.1.3.2.8" class="ltx_td ltx_align_right">84.6</td>
<td id="A1.T7.1.1.3.2.9" class="ltx_td ltx_align_right">80.8</td>
<td id="A1.T7.1.1.3.2.10" class="ltx_td ltx_align_right">74.9</td>
<td id="A1.T7.1.1.3.2.11" class="ltx_td ltx_align_right">82.4</td>
<td id="A1.T7.1.1.3.2.12" class="ltx_td ltx_align_right">65.6</td>
<td id="A1.T7.1.1.3.2.13" class="ltx_td ltx_align_right">77.6</td>
<td id="A1.T7.1.1.3.2.14" class="ltx_td ltx_align_right">80.8</td>
<td id="A1.T7.1.1.3.2.15" class="ltx_td ltx_align_right">74.8</td>
<td id="A1.T7.1.1.3.2.16" class="ltx_td ltx_align_right">73.7</td>
<td id="A1.T7.1.1.3.2.17" class="ltx_td ltx_align_right">77.6</td>
<td id="A1.T7.1.1.3.2.18" class="ltx_td ltx_align_right">74.8</td>
<td id="A1.T7.1.1.3.2.19" class="ltx_td ltx_align_right">82.9</td>
<td id="A1.T7.1.1.3.2.20" class="ltx_td ltx_align_right">77.1</td>
<td id="A1.T7.1.1.3.2.21" class="ltx_td ltx_align_right">78.9</td>
<td id="A1.T7.1.1.3.2.22" class="ltx_td ltx_align_right">72.0</td>
<td id="A1.T7.1.1.3.2.23" class="ltx_td ltx_align_right">77.2</td>
<td id="A1.T7.1.1.3.2.24" class="ltx_td ltx_align_right">77.0</td>
<td id="A1.T7.1.1.3.2.25" class="ltx_td ltx_nopad_r ltx_align_right">80.3</td>
</tr>
<tr id="A1.T7.1.1.4.3" class="ltx_tr">
<td id="A1.T7.1.1.4.3.1" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T7.1.1.4.3.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.4.3.2.1" class="ltx_text" style="background-color:#E6E6E6;">73.0</span></td>
<td id="A1.T7.1.1.4.3.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.4.3.3.1" class="ltx_text" style="background-color:#E6E6E6;">73.4</span></td>
<td id="A1.T7.1.1.4.3.4" class="ltx_td ltx_align_right">73.8</td>
<td id="A1.T7.1.1.4.3.5" class="ltx_td ltx_align_right">74.9</td>
<td id="A1.T7.1.1.4.3.6" class="ltx_td ltx_align_right">73.2</td>
<td id="A1.T7.1.1.4.3.7" class="ltx_td ltx_align_right">70.8</td>
<td id="A1.T7.1.1.4.3.8" class="ltx_td ltx_align_right">83.2</td>
<td id="A1.T7.1.1.4.3.9" class="ltx_td ltx_align_right">77.0</td>
<td id="A1.T7.1.1.4.3.10" class="ltx_td ltx_align_right">71.4</td>
<td id="A1.T7.1.1.4.3.11" class="ltx_td ltx_align_right">78.1</td>
<td id="A1.T7.1.1.4.3.12" class="ltx_td ltx_align_right">61.0</td>
<td id="A1.T7.1.1.4.3.13" class="ltx_td ltx_align_right">73.9</td>
<td id="A1.T7.1.1.4.3.14" class="ltx_td ltx_align_right">76.2</td>
<td id="A1.T7.1.1.4.3.15" class="ltx_td ltx_align_right">71.7</td>
<td id="A1.T7.1.1.4.3.16" class="ltx_td ltx_align_right">67.4</td>
<td id="A1.T7.1.1.4.3.17" class="ltx_td ltx_align_right">73.0</td>
<td id="A1.T7.1.1.4.3.18" class="ltx_td ltx_align_right">70.4</td>
<td id="A1.T7.1.1.4.3.19" class="ltx_td ltx_align_right">80.1</td>
<td id="A1.T7.1.1.4.3.20" class="ltx_td ltx_align_right">74.3</td>
<td id="A1.T7.1.1.4.3.21" class="ltx_td ltx_align_right">73.3</td>
<td id="A1.T7.1.1.4.3.22" class="ltx_td ltx_align_right">68.2</td>
<td id="A1.T7.1.1.4.3.23" class="ltx_td ltx_align_right">73.0</td>
<td id="A1.T7.1.1.4.3.24" class="ltx_td ltx_align_right">71.4</td>
<td id="A1.T7.1.1.4.3.25" class="ltx_td ltx_nopad_r ltx_align_right">78.8</td>
</tr>
<tr id="A1.T7.1.1.5.4" class="ltx_tr">
<td id="A1.T7.1.1.5.4.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="3"><span id="A1.T7.1.1.5.4.1.1" class="ltx_text">8b</span></td>
<td id="A1.T7.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T7.1.1.5.4.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.5.4.3.1" class="ltx_text" style="background-color:#E6E6E6;">64.8</span></td>
<td id="A1.T7.1.1.5.4.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.5.4.4.1" class="ltx_text" style="background-color:#E6E6E6;">65.3</span></td>
<td id="A1.T7.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_t">65.6</td>
<td id="A1.T7.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_t">61.9</td>
<td id="A1.T7.1.1.5.4.7" class="ltx_td ltx_align_right ltx_border_t">65.6</td>
<td id="A1.T7.1.1.5.4.8" class="ltx_td ltx_align_right ltx_border_t">64.0</td>
<td id="A1.T7.1.1.5.4.9" class="ltx_td ltx_align_right ltx_border_t">77.0</td>
<td id="A1.T7.1.1.5.4.10" class="ltx_td ltx_align_right ltx_border_t">67.0</td>
<td id="A1.T7.1.1.5.4.11" class="ltx_td ltx_align_right ltx_border_t">63.6</td>
<td id="A1.T7.1.1.5.4.12" class="ltx_td ltx_align_right ltx_border_t">69.6</td>
<td id="A1.T7.1.1.5.4.13" class="ltx_td ltx_align_right ltx_border_t">54.3</td>
<td id="A1.T7.1.1.5.4.14" class="ltx_td ltx_align_right ltx_border_t">67.4</td>
<td id="A1.T7.1.1.5.4.15" class="ltx_td ltx_align_right ltx_border_t">65.7</td>
<td id="A1.T7.1.1.5.4.16" class="ltx_td ltx_align_right ltx_border_t">65.2</td>
<td id="A1.T7.1.1.5.4.17" class="ltx_td ltx_align_right ltx_border_t">61.7</td>
<td id="A1.T7.1.1.5.4.18" class="ltx_td ltx_align_right ltx_border_t">63.8</td>
<td id="A1.T7.1.1.5.4.19" class="ltx_td ltx_align_right ltx_border_t">61.3</td>
<td id="A1.T7.1.1.5.4.20" class="ltx_td ltx_align_right ltx_border_t">69.1</td>
<td id="A1.T7.1.1.5.4.21" class="ltx_td ltx_align_right ltx_border_t">65.7</td>
<td id="A1.T7.1.1.5.4.22" class="ltx_td ltx_align_right ltx_border_t">69.7</td>
<td id="A1.T7.1.1.5.4.23" class="ltx_td ltx_align_right ltx_border_t">58.1</td>
<td id="A1.T7.1.1.5.4.24" class="ltx_td ltx_align_right ltx_border_t">66.8</td>
<td id="A1.T7.1.1.5.4.25" class="ltx_td ltx_align_right ltx_border_t">62.3</td>
<td id="A1.T7.1.1.5.4.26" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">72.2</td>
</tr>
<tr id="A1.T7.1.1.6.5" class="ltx_tr">
<td id="A1.T7.1.1.6.5.1" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T7.1.1.6.5.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.6.5.2.1" class="ltx_text" style="background-color:#E6E6E6;">64.6</span></td>
<td id="A1.T7.1.1.6.5.3" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.6.5.3.1" class="ltx_text" style="background-color:#E6E6E6;">65.1</span></td>
<td id="A1.T7.1.1.6.5.4" class="ltx_td ltx_align_right">64.3</td>
<td id="A1.T7.1.1.6.5.5" class="ltx_td ltx_align_right">61.8</td>
<td id="A1.T7.1.1.6.5.6" class="ltx_td ltx_align_right">64.8</td>
<td id="A1.T7.1.1.6.5.7" class="ltx_td ltx_align_right">63.0</td>
<td id="A1.T7.1.1.6.5.8" class="ltx_td ltx_align_right">76.1</td>
<td id="A1.T7.1.1.6.5.9" class="ltx_td ltx_align_right">67.4</td>
<td id="A1.T7.1.1.6.5.10" class="ltx_td ltx_align_right">63.9</td>
<td id="A1.T7.1.1.6.5.11" class="ltx_td ltx_align_right">70.4</td>
<td id="A1.T7.1.1.6.5.12" class="ltx_td ltx_align_right">54.2</td>
<td id="A1.T7.1.1.6.5.13" class="ltx_td ltx_align_right">67.4</td>
<td id="A1.T7.1.1.6.5.14" class="ltx_td ltx_align_right">64.6</td>
<td id="A1.T7.1.1.6.5.15" class="ltx_td ltx_align_right">65.4</td>
<td id="A1.T7.1.1.6.5.16" class="ltx_td ltx_align_right">61.4</td>
<td id="A1.T7.1.1.6.5.17" class="ltx_td ltx_align_right">64.3</td>
<td id="A1.T7.1.1.6.5.18" class="ltx_td ltx_align_right">59.8</td>
<td id="A1.T7.1.1.6.5.19" class="ltx_td ltx_align_right">68.7</td>
<td id="A1.T7.1.1.6.5.20" class="ltx_td ltx_align_right">65.4</td>
<td id="A1.T7.1.1.6.5.21" class="ltx_td ltx_align_right">68.7</td>
<td id="A1.T7.1.1.6.5.22" class="ltx_td ltx_align_right">58.1</td>
<td id="A1.T7.1.1.6.5.23" class="ltx_td ltx_align_right">67.0</td>
<td id="A1.T7.1.1.6.5.24" class="ltx_td ltx_align_right">63.7</td>
<td id="A1.T7.1.1.6.5.25" class="ltx_td ltx_nopad_r ltx_align_right">71.8</td>
</tr>
<tr id="A1.T7.1.1.7.6" class="ltx_tr">
<td id="A1.T7.1.1.7.6.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4</td>
<td id="A1.T7.1.1.7.6.2" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.7.6.2.1" class="ltx_text" style="background-color:#E6E6E6;">59.3</span></td>
<td id="A1.T7.1.1.7.6.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T7.1.1.7.6.3.1" class="ltx_text" style="background-color:#E6E6E6;">59.9</span></td>
<td id="A1.T7.1.1.7.6.4" class="ltx_td ltx_align_right ltx_border_bb">61.9</td>
<td id="A1.T7.1.1.7.6.5" class="ltx_td ltx_align_right ltx_border_bb">57.0</td>
<td id="A1.T7.1.1.7.6.6" class="ltx_td ltx_align_right ltx_border_bb">61.6</td>
<td id="A1.T7.1.1.7.6.7" class="ltx_td ltx_align_right ltx_border_bb">57.7</td>
<td id="A1.T7.1.1.7.6.8" class="ltx_td ltx_align_right ltx_border_bb">73.8</td>
<td id="A1.T7.1.1.7.6.9" class="ltx_td ltx_align_right ltx_border_bb">61.1</td>
<td id="A1.T7.1.1.7.6.10" class="ltx_td ltx_align_right ltx_border_bb">58.2</td>
<td id="A1.T7.1.1.7.6.11" class="ltx_td ltx_align_right ltx_border_bb">65.7</td>
<td id="A1.T7.1.1.7.6.12" class="ltx_td ltx_align_right ltx_border_bb">49.8</td>
<td id="A1.T7.1.1.7.6.13" class="ltx_td ltx_align_right ltx_border_bb">64.7</td>
<td id="A1.T7.1.1.7.6.14" class="ltx_td ltx_align_right ltx_border_bb">58.3</td>
<td id="A1.T7.1.1.7.6.15" class="ltx_td ltx_align_right ltx_border_bb">60.7</td>
<td id="A1.T7.1.1.7.6.16" class="ltx_td ltx_align_right ltx_border_bb">51.1</td>
<td id="A1.T7.1.1.7.6.17" class="ltx_td ltx_align_right ltx_border_bb">60.7</td>
<td id="A1.T7.1.1.7.6.18" class="ltx_td ltx_align_right ltx_border_bb">54.9</td>
<td id="A1.T7.1.1.7.6.19" class="ltx_td ltx_align_right ltx_border_bb">62.0</td>
<td id="A1.T7.1.1.7.6.20" class="ltx_td ltx_align_right ltx_border_bb">59.8</td>
<td id="A1.T7.1.1.7.6.21" class="ltx_td ltx_align_right ltx_border_bb">63.9</td>
<td id="A1.T7.1.1.7.6.22" class="ltx_td ltx_align_right ltx_border_bb">50.1</td>
<td id="A1.T7.1.1.7.6.23" class="ltx_td ltx_align_right ltx_border_bb">61.0</td>
<td id="A1.T7.1.1.7.6.24" class="ltx_td ltx_align_right ltx_border_bb">58.8</td>
<td id="A1.T7.1.1.7.6.25" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb">66.2</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A7: </span><span id="A1.T7.3.1" class="ltx_text ltx_font_bold">Aya 23 language-specific results for Belebele.</span> (Accuracy)</figcaption>
</figure>
<figure id="A1.T8" class="ltx_table">
<div id="A1.T8.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:34.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-349.1pt,27.8pt) scale(0.383112398692546,0.383112398692546) ;">
<table id="A1.T8.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T8.1.1.1" class="ltx_tr">
<th id="A1.T8.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="A1.T8.1.1.1.3" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="A1.T8.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.4.1" class="ltx_text ltx_font_bold">ar</span></th>
<th id="A1.T8.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.5.1" class="ltx_text ltx_font_bold">cs</span></th>
<th id="A1.T8.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.6.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T8.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.7.1" class="ltx_text ltx_font_bold">el</span></th>
<th id="A1.T8.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.8.1" class="ltx_text ltx_font_bold">en</span></th>
<th id="A1.T8.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.9.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T8.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.10.1" class="ltx_text ltx_font_bold">fa</span></th>
<th id="A1.T8.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.11.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T8.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.12.1" class="ltx_text ltx_font_bold">hi</span></th>
<th id="A1.T8.1.1.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.13.1" class="ltx_text ltx_font_bold">id</span></th>
<th id="A1.T8.1.1.1.14" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.14.1" class="ltx_text ltx_font_bold">it</span></th>
<th id="A1.T8.1.1.1.15" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.15.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T8.1.1.1.16" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.16.1" class="ltx_text ltx_font_bold">ko</span></th>
<th id="A1.T8.1.1.1.17" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.17.1" class="ltx_text ltx_font_bold">nl</span></th>
<th id="A1.T8.1.1.1.18" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.18.1" class="ltx_text ltx_font_bold">pl</span></th>
<th id="A1.T8.1.1.1.19" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.19.1" class="ltx_text ltx_font_bold">pt</span></th>
<th id="A1.T8.1.1.1.20" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.20.1" class="ltx_text ltx_font_bold">ro</span></th>
<th id="A1.T8.1.1.1.21" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.21.1" class="ltx_text ltx_font_bold">ru</span></th>
<th id="A1.T8.1.1.1.22" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.22.1" class="ltx_text ltx_font_bold">tr</span></th>
<th id="A1.T8.1.1.1.23" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.23.1" class="ltx_text ltx_font_bold">uk</span></th>
<th id="A1.T8.1.1.1.24" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.24.1" class="ltx_text ltx_font_bold">vi</span></th>
<th id="A1.T8.1.1.1.25" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.25.1" class="ltx_text ltx_font_bold">zh</span></th>
<th id="A1.T8.1.1.1.26" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_l ltx_border_tt"><span id="A1.T8.1.1.1.26.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="A1.T8.1.1.1.27" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<span id="A1.T8.1.1.1.27.1" class="ltx_text ltx_font_bold">Avg</span> (<span id="A1.T8.1.1.1.27.2" class="ltx_text ltx_font_italic">-en</span>)</th>
<th id="A1.T8.1.1.1.28" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T8.1.1.1.28.1" class="ltx_text ltx_font_bold">Ltn</span></th>
<th id="A1.T8.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="A1.T8.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T8.1.1.1.1.m1.1a"><mo id="A1.T8.1.1.1.1.m1.1.1" xref="A1.T8.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T8.1.1.1.1.m1.1b"><not id="A1.T8.1.1.1.1.m1.1.1.cmml" xref="A1.T8.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T8.1.1.1.1.m1.1c">\neg</annotation></semantics></math></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T8.1.1.2.1" class="ltx_tr">
<td id="A1.T8.1.1.2.1.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T8.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T8.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F7D9D7;"><span id="A1.T8.1.1.2.1.3.1" class="ltx_text" style="background-color:#F7D9D7;">-2.0%</span></td>
<td id="A1.T8.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#C3E7D6;"><span id="A1.T8.1.1.2.1.4.1" class="ltx_text" style="background-color:#C3E7D6;">0.7%</span></td>
<td id="A1.T8.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F4FBF7;"><span id="A1.T8.1.1.2.1.5.1" class="ltx_text" style="background-color:#F4FBF7;">0.1%</span></td>
<td id="A1.T8.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F2FAF6;"><span id="A1.T8.1.1.2.1.6.1" class="ltx_text" style="background-color:#F2FAF6;">0.2%</span></td>
<td id="A1.T8.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T8.1.1.2.1.7.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T8.1.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEF9F9;"><span id="A1.T8.1.1.2.1.8.1" class="ltx_text" style="background-color:#FEF9F9;">-0.3%</span></td>
<td id="A1.T8.1.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE9E7;"><span id="A1.T8.1.1.2.1.9.1" class="ltx_text" style="background-color:#FAE9E7;">-1.2%</span></td>
<td id="A1.T8.1.1.2.1.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#C7E9D8;"><span id="A1.T8.1.1.2.1.10.1" class="ltx_text" style="background-color:#C7E9D8;">0.7%</span></td>
<td id="A1.T8.1.1.2.1.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T8.1.1.2.1.11.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T8.1.1.2.1.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F9;"><span id="A1.T8.1.1.2.1.12.1" class="ltx_text" style="background-color:#FDF9F9;">-0.3%</span></td>
<td id="A1.T8.1.1.2.1.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#96D5B6;"><span id="A1.T8.1.1.2.1.13.1" class="ltx_text" style="background-color:#96D5B6;">1.3%</span></td>
<td id="A1.T8.1.1.2.1.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F9E3E1;"><span id="A1.T8.1.1.2.1.14.1" class="ltx_text" style="background-color:#F9E3E1;">-1.5%</span></td>
<td id="A1.T8.1.1.2.1.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D9F0E4;"><span id="A1.T8.1.1.2.1.15.1" class="ltx_text" style="background-color:#D9F0E4;">0.5%</span></td>
<td id="A1.T8.1.1.2.1.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T8.1.1.2.1.16.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T8.1.1.2.1.17" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE6E4;"><span id="A1.T8.1.1.2.1.17.1" class="ltx_text" style="background-color:#FAE6E4;">-1.3%</span></td>
<td id="A1.T8.1.1.2.1.18" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEBE9;"><span id="A1.T8.1.1.2.1.18.1" class="ltx_text" style="background-color:#FBEBE9;">-1.1%</span></td>
<td id="A1.T8.1.1.2.1.19" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F6D4D1;"><span id="A1.T8.1.1.2.1.19.1" class="ltx_text" style="background-color:#F6D4D1;">-2.3%</span></td>
<td id="A1.T8.1.1.2.1.20" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEFEE;"><span id="A1.T8.1.1.2.1.20.1" class="ltx_text" style="background-color:#FBEFEE;">-0.8%</span></td>
<td id="A1.T8.1.1.2.1.21" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F4C9C6;"><span id="A1.T8.1.1.2.1.21.1" class="ltx_text" style="background-color:#F4C9C6;">-2.8%</span></td>
<td id="A1.T8.1.1.2.1.22" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F6;"><span id="A1.T8.1.1.2.1.22.1" class="ltx_text" style="background-color:#FDF6F6;">-0.4%</span></td>
<td id="A1.T8.1.1.2.1.23" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8DFDD;"><span id="A1.T8.1.1.2.1.23.1" class="ltx_text" style="background-color:#F8DFDD;">-1.7%</span></td>
<td id="A1.T8.1.1.2.1.24" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FBEAE9;"><span id="A1.T8.1.1.2.1.24.1" class="ltx_text" style="background-color:#FBEAE9;">-1.1%</span></td>
<td id="A1.T8.1.1.2.1.25" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FCF3F2;"><span id="A1.T8.1.1.2.1.25.1" class="ltx_text" style="background-color:#FCF3F2;">-0.6%</span></td>
<td id="A1.T8.1.1.2.1.26" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FCF3F2;"><span id="A1.T8.1.1.2.1.26.1" class="ltx_text" style="background-color:#FCF3F2;">-0.6%</span></td>
<td id="A1.T8.1.1.2.1.27" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FDF4F3;"><span id="A1.T8.1.1.2.1.27.1" class="ltx_text" style="background-color:#FDF4F3;">-0.6%</span></td>
<td id="A1.T8.1.1.2.1.28" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FBEDEC;"><span id="A1.T8.1.1.2.1.28.1" class="ltx_text" style="background-color:#FBEDEC;">-0.7%</span></td>
</tr>
<tr id="A1.T8.1.1.3.2" class="ltx_tr">
<td id="A1.T8.1.1.3.2.1" class="ltx_td ltx_align_left"><span id="A1.T8.1.1.3.2.1.1" class="ltx_text">35b</span></td>
<td id="A1.T8.1.1.3.2.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T8.1.1.3.2.3" class="ltx_td ltx_align_right" style="background-color:#E7857D;"><span id="A1.T8.1.1.3.2.3.1" class="ltx_text" style="background-color:#E7857D;">-6.5%</span></td>
<td id="A1.T8.1.1.3.2.4" class="ltx_td ltx_align_right" style="background-color:#EFAFA9;"><span id="A1.T8.1.1.3.2.4.1" class="ltx_text" style="background-color:#EFAFA9;">-4.3%</span></td>
<td id="A1.T8.1.1.3.2.5" class="ltx_td ltx_align_right" style="background-color:#ECA09A;"><span id="A1.T8.1.1.3.2.5.1" class="ltx_text" style="background-color:#ECA09A;">-5.0%</span></td>
<td id="A1.T8.1.1.3.2.6" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T8.1.1.3.2.6.1" class="ltx_text" style="background-color:#E67C73;">-7.4%</span></td>
<td id="A1.T8.1.1.3.2.7" class="ltx_td ltx_align_right" style="background-color:#F8DEDC;"><span id="A1.T8.1.1.3.2.7.1" class="ltx_text" style="background-color:#F8DEDC;">-1.7%</span></td>
<td id="A1.T8.1.1.3.2.8" class="ltx_td ltx_align_right" style="background-color:#EDA29C;"><span id="A1.T8.1.1.3.2.8.1" class="ltx_text" style="background-color:#EDA29C;">-4.9%</span></td>
<td id="A1.T8.1.1.3.2.9" class="ltx_td ltx_align_right" style="background-color:#EA938C;"><span id="A1.T8.1.1.3.2.9.1" class="ltx_text" style="background-color:#EA938C;">-5.7%</span></td>
<td id="A1.T8.1.1.3.2.10" class="ltx_td ltx_align_right" style="background-color:#EEA8A2;"><span id="A1.T8.1.1.3.2.10.1" class="ltx_text" style="background-color:#EEA8A2;">-4.6%</span></td>
<td id="A1.T8.1.1.3.2.11" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T8.1.1.3.2.11.1" class="ltx_text" style="background-color:#E67C73;">-7.0%</span></td>
<td id="A1.T8.1.1.3.2.12" class="ltx_td ltx_align_right" style="background-color:#EDA19A;"><span id="A1.T8.1.1.3.2.12.1" class="ltx_text" style="background-color:#EDA19A;">-5.0%</span></td>
<td id="A1.T8.1.1.3.2.13" class="ltx_td ltx_align_right" style="background-color:#EFABA5;"><span id="A1.T8.1.1.3.2.13.1" class="ltx_text" style="background-color:#EFABA5;">-4.5%</span></td>
<td id="A1.T8.1.1.3.2.14" class="ltx_td ltx_align_right" style="background-color:#EB968F;"><span id="A1.T8.1.1.3.2.14.1" class="ltx_text" style="background-color:#EB968F;">-5.6%</span></td>
<td id="A1.T8.1.1.3.2.15" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T8.1.1.3.2.15.1" class="ltx_text" style="background-color:#E67C73;">-8.0%</span></td>
<td id="A1.T8.1.1.3.2.16" class="ltx_td ltx_align_right" style="background-color:#E98E86;"><span id="A1.T8.1.1.3.2.16.1" class="ltx_text" style="background-color:#E98E86;">-6.0%</span></td>
<td id="A1.T8.1.1.3.2.17" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T8.1.1.3.2.17.1" class="ltx_text" style="background-color:#E67C73;">-7.0%</span></td>
<td id="A1.T8.1.1.3.2.18" class="ltx_td ltx_align_right" style="background-color:#EFADA7;"><span id="A1.T8.1.1.3.2.18.1" class="ltx_text" style="background-color:#EFADA7;">-4.4%</span></td>
<td id="A1.T8.1.1.3.2.19" class="ltx_td ltx_align_right" style="background-color:#EA928B;"><span id="A1.T8.1.1.3.2.19.1" class="ltx_text" style="background-color:#EA928B;">-5.8%</span></td>
<td id="A1.T8.1.1.3.2.20" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T8.1.1.3.2.20.1" class="ltx_text" style="background-color:#E67C73;">-7.8%</span></td>
<td id="A1.T8.1.1.3.2.21" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T8.1.1.3.2.21.1" class="ltx_text" style="background-color:#E67C73;">-7.9%</span></td>
<td id="A1.T8.1.1.3.2.22" class="ltx_td ltx_align_right" style="background-color:#EA9089;"><span id="A1.T8.1.1.3.2.22.1" class="ltx_text" style="background-color:#EA9089;">-5.9%</span></td>
<td id="A1.T8.1.1.3.2.23" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T8.1.1.3.2.23.1" class="ltx_text" style="background-color:#E67C73;">-8.8%</span></td>
<td id="A1.T8.1.1.3.2.24" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F4C6C2;"><span id="A1.T8.1.1.3.2.24.1" class="ltx_text" style="background-color:#F4C6C2;">-3.0%</span></td>
<td id="A1.T8.1.1.3.2.25" class="ltx_td ltx_align_center" style="background-color:#EA938B;"><span id="A1.T8.1.1.3.2.25.1" class="ltx_text" style="background-color:#EA938B;">-5.8%</span></td>
<td id="A1.T8.1.1.3.2.26" class="ltx_td ltx_align_center" style="background-color:#EA938B;"><span id="A1.T8.1.1.3.2.26.1" class="ltx_text" style="background-color:#EA938B;">-6.0%</span></td>
<td id="A1.T8.1.1.3.2.27" class="ltx_td ltx_align_center" style="background-color:#EB9A93;"><span id="A1.T8.1.1.3.2.27.1" class="ltx_text" style="background-color:#EB9A93;">-5.4%</span></td>
<td id="A1.T8.1.1.3.2.28" class="ltx_td ltx_align_center" style="background-color:#E7857D;"><span id="A1.T8.1.1.3.2.28.1" class="ltx_text" style="background-color:#E7857D;">-6.3%</span></td>
</tr>
<tr id="A1.T8.1.1.4.3" class="ltx_tr">
<td id="A1.T8.1.1.4.3.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T8.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T8.1.1.4.3.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8DBD9;"><span id="A1.T8.1.1.4.3.3.1" class="ltx_text" style="background-color:#F8DBD9;">-1.9%</span></td>
<td id="A1.T8.1.1.4.3.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="A1.T8.1.1.4.3.4.1" class="ltx_text" style="background-color:#FEFBFB;">-0.2%</span></td>
<td id="A1.T8.1.1.4.3.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE8E7;"><span id="A1.T8.1.1.4.3.5.1" class="ltx_text" style="background-color:#FAE8E7;">-1.2%</span></td>
<td id="A1.T8.1.1.4.3.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F9E1DF;"><span id="A1.T8.1.1.4.3.6.1" class="ltx_text" style="background-color:#F9E1DF;">-1.6%</span></td>
<td id="A1.T8.1.1.4.3.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE9E7;"><span id="A1.T8.1.1.4.3.7.1" class="ltx_text" style="background-color:#FAE9E7;">-1.2%</span></td>
<td id="A1.T8.1.1.4.3.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#C8E9D9;"><span id="A1.T8.1.1.4.3.8.1" class="ltx_text" style="background-color:#C8E9D9;">0.7%</span></td>
<td id="A1.T8.1.1.4.3.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D4EEE1;"><span id="A1.T8.1.1.4.3.9.1" class="ltx_text" style="background-color:#D4EEE1;">0.5%</span></td>
<td id="A1.T8.1.1.4.3.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#95D4B5;"><span id="A1.T8.1.1.4.3.10.1" class="ltx_text" style="background-color:#95D4B5;">1.3%</span></td>
<td id="A1.T8.1.1.4.3.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFA;"><span id="A1.T8.1.1.4.3.11.1" class="ltx_text" style="background-color:#FEFBFA;">-0.2%</span></td>
<td id="A1.T8.1.1.4.3.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T8.1.1.4.3.12.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T8.1.1.4.3.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8DFDD;"><span id="A1.T8.1.1.4.3.13.1" class="ltx_text" style="background-color:#F8DFDD;">-1.7%</span></td>
<td id="A1.T8.1.1.4.3.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E3F4EC;"><span id="A1.T8.1.1.4.3.14.1" class="ltx_text" style="background-color:#E3F4EC;">0.3%</span></td>
<td id="A1.T8.1.1.4.3.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF8F7;"><span id="A1.T8.1.1.4.3.15.1" class="ltx_text" style="background-color:#FDF8F7;">-0.4%</span></td>
<td id="A1.T8.1.1.4.3.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#B7E2CD;"><span id="A1.T8.1.1.4.3.16.1" class="ltx_text" style="background-color:#B7E2CD;">0.9%</span></td>
<td id="A1.T8.1.1.4.3.17" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F5CFCC;"><span id="A1.T8.1.1.4.3.17.1" class="ltx_text" style="background-color:#F5CFCC;">-2.5%</span></td>
<td id="A1.T8.1.1.4.3.18" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F2;"><span id="A1.T8.1.1.4.3.18.1" class="ltx_text" style="background-color:#FCF3F2;">-0.6%</span></td>
<td id="A1.T8.1.1.4.3.19" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF8F7;"><span id="A1.T8.1.1.4.3.19.1" class="ltx_text" style="background-color:#FDF8F7;">-0.4%</span></td>
<td id="A1.T8.1.1.4.3.20" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F9E4E2;"><span id="A1.T8.1.1.4.3.20.1" class="ltx_text" style="background-color:#F9E4E2;">-1.4%</span></td>
<td id="A1.T8.1.1.4.3.21" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T8.1.1.4.3.21.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T8.1.1.4.3.22" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E4F4EC;"><span id="A1.T8.1.1.4.3.22.1" class="ltx_text" style="background-color:#E4F4EC;">0.3%</span></td>
<td id="A1.T8.1.1.4.3.23" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#57BB8A;"><span id="A1.T8.1.1.4.3.23.1" class="ltx_text" style="background-color:#57BB8A;">2.1%</span></td>
<td id="A1.T8.1.1.4.3.24" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FCF3F2;"><span id="A1.T8.1.1.4.3.24.1" class="ltx_text" style="background-color:#FCF3F2;">-0.6%</span></td>
<td id="A1.T8.1.1.4.3.25" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FDF8F8;"><span id="A1.T8.1.1.4.3.25.1" class="ltx_text" style="background-color:#FDF8F8;">-0.3%</span></td>
<td id="A1.T8.1.1.4.3.26" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FDF8F8;"><span id="A1.T8.1.1.4.3.26.1" class="ltx_text" style="background-color:#FDF8F8;">-0.3%</span></td>
<td id="A1.T8.1.1.4.3.27" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FEFAFA;"><span id="A1.T8.1.1.4.3.27.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="A1.T8.1.1.4.3.28" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FDF5F5;"><span id="A1.T8.1.1.4.3.28.1" class="ltx_text" style="background-color:#FDF5F5;">-0.5%</span></td>
</tr>
<tr id="A1.T8.1.1.5.4" class="ltx_tr">
<td id="A1.T8.1.1.5.4.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A1.T8.1.1.5.4.1.1" class="ltx_text">8b</span></td>
<td id="A1.T8.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4</td>
<td id="A1.T8.1.1.5.4.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EB968F;"><span id="A1.T8.1.1.5.4.3.1" class="ltx_text" style="background-color:#EB968F;">-5.6%</span></td>
<td id="A1.T8.1.1.5.4.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.4.1" class="ltx_text" style="background-color:#E67C73;">-7.9%</span></td>
<td id="A1.T8.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E98C84;"><span id="A1.T8.1.1.5.4.5.1" class="ltx_text" style="background-color:#E98C84;">-6.1%</span></td>
<td id="A1.T8.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.6.1" class="ltx_text" style="background-color:#E67C73;">-9.9%</span></td>
<td id="A1.T8.1.1.5.4.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F0B0AB;"><span id="A1.T8.1.1.5.4.7.1" class="ltx_text" style="background-color:#F0B0AB;">-4.2%</span></td>
<td id="A1.T8.1.1.5.4.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.8.1" class="ltx_text" style="background-color:#E67C73;">-8.8%</span></td>
<td id="A1.T8.1.1.5.4.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.9.1" class="ltx_text" style="background-color:#E67C73;">-8.4%</span></td>
<td id="A1.T8.1.1.5.4.10" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EB968F;"><span id="A1.T8.1.1.5.4.10.1" class="ltx_text" style="background-color:#EB968F;">-5.6%</span></td>
<td id="A1.T8.1.1.5.4.11" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.11.1" class="ltx_text" style="background-color:#E67C73;">-8.4%</span></td>
<td id="A1.T8.1.1.5.4.12" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F0B2AC;"><span id="A1.T8.1.1.5.4.12.1" class="ltx_text" style="background-color:#F0B2AC;">-4.1%</span></td>
<td id="A1.T8.1.1.5.4.13" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.13.1" class="ltx_text" style="background-color:#E67C73;">-11.2%</span></td>
<td id="A1.T8.1.1.5.4.14" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.14.1" class="ltx_text" style="background-color:#E67C73;">-7.0%</span></td>
<td id="A1.T8.1.1.5.4.15" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.15.1" class="ltx_text" style="background-color:#E67C73;">-17.1%</span></td>
<td id="A1.T8.1.1.5.4.16" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EDA39D;"><span id="A1.T8.1.1.5.4.16.1" class="ltx_text" style="background-color:#EDA39D;">-4.9%</span></td>
<td id="A1.T8.1.1.5.4.17" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.17.1" class="ltx_text" style="background-color:#E67C73;">-10.5%</span></td>
<td id="A1.T8.1.1.5.4.18" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.18.1" class="ltx_text" style="background-color:#E67C73;">-10.3%</span></td>
<td id="A1.T8.1.1.5.4.19" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.19.1" class="ltx_text" style="background-color:#E67C73;">-9.0%</span></td>
<td id="A1.T8.1.1.5.4.20" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.20.1" class="ltx_text" style="background-color:#E67C73;">-8.3%</span></td>
<td id="A1.T8.1.1.5.4.21" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.21.1" class="ltx_text" style="background-color:#E67C73;">-13.8%</span></td>
<td id="A1.T8.1.1.5.4.22" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.22.1" class="ltx_text" style="background-color:#E67C73;">-8.7%</span></td>
<td id="A1.T8.1.1.5.4.23" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EA948D;"><span id="A1.T8.1.1.5.4.23.1" class="ltx_text" style="background-color:#EA948D;">-5.7%</span></td>
<td id="A1.T8.1.1.5.4.24" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.24.1" class="ltx_text" style="background-color:#E67C73;">-8.3%</span></td>
<td id="A1.T8.1.1.5.4.25" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.25.1" class="ltx_text" style="background-color:#E67C73;">-8.3%</span></td>
<td id="A1.T8.1.1.5.4.26" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.26.1" class="ltx_text" style="background-color:#E67C73;">-8.5%</span></td>
<td id="A1.T8.1.1.5.4.27" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.27.1" class="ltx_text" style="background-color:#E67C73;">-7.8%</span></td>
<td id="A1.T8.1.1.5.4.28" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T8.1.1.5.4.28.1" class="ltx_text" style="background-color:#E67C73;">-9.1%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A8: </span><span id="A1.T8.4.1" class="ltx_text ltx_font_bold">Relative performance (<math id="A1.T8.4.1.m1.1" class="ltx_math_unparsed" alttext="\%\Delta" display="inline"><semantics id="A1.T8.4.1.m1.1b"><mrow id="A1.T8.4.1.m1.1c"><mo id="A1.T8.4.1.m1.1.1">%</mo><mi mathvariant="normal" id="A1.T8.4.1.m1.1.2">Δ</mi></mrow><annotation encoding="application/x-tex" id="A1.T8.4.1.m1.1d">\%\Delta</annotation></semantics></math>) vs. FP16 for Aya Models on Belebele.</span> Ltn are Latin-script languages: cs, de, es, en, es, fr, id, it, nl, pl, pt, ro, tr, vi. <math id="A1.T8.5.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T8.5.m1.1b"><mo id="A1.T8.5.m1.1.1" xref="A1.T8.5.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T8.5.m1.1c"><not id="A1.T8.5.m1.1.1.cmml" xref="A1.T8.5.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T8.5.m1.1d">\neg</annotation></semantics></math> are the rest.</figcaption>
</figure>
<figure id="A1.T9" class="ltx_table">
<table id="A1.T9.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T9.1.1.1" class="ltx_tr">
<th id="A1.T9.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt"></th>
<th id="A1.T9.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt"></th>
<th id="A1.T9.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T9.1.1.1.3.1" class="ltx_text ltx_font_bold">ar</span></th>
<th id="A1.T9.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T9.1.1.1.4.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T9.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T9.1.1.1.5.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T9.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T9.1.1.1.6.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T9.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T9.1.1.1.7.1" class="ltx_text ltx_font_bold">it</span></th>
<th id="A1.T9.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T9.1.1.1.8.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T9.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T9.1.1.1.9.1" class="ltx_text ltx_font_bold">ko</span></th>
<th id="A1.T9.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T9.1.1.1.10.1" class="ltx_text ltx_font_bold">pt</span></th>
<th id="A1.T9.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt"><span id="A1.T9.1.1.1.11.1" class="ltx_text ltx_font_bold">zh</span></th>
<th id="A1.T9.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T9.1.1.1.12.1" class="ltx_text ltx_font_bold">Avg</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T9.1.2.1" class="ltx_tr">
<th id="A1.T9.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="6"><span id="A1.T9.1.2.1.1.1" class="ltx_text">103B</span></th>
<th id="A1.T9.1.2.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="A1.T9.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t">64.0</td>
<td id="A1.T9.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t">68.3</td>
<td id="A1.T9.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t">68.7</td>
<td id="A1.T9.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t">68.0</td>
<td id="A1.T9.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t">69.3</td>
<td id="A1.T9.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t">64.4</td>
<td id="A1.T9.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t">62.3</td>
<td id="A1.T9.1.2.1.10" class="ltx_td ltx_align_right ltx_border_t">70.0</td>
<td id="A1.T9.1.2.1.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">65.0</td>
<td id="A1.T9.1.2.1.12" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">66.7</td>
</tr>
<tr id="A1.T9.1.3.2" class="ltx_tr">
<th id="A1.T9.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8</th>
<td id="A1.T9.1.3.2.2" class="ltx_td ltx_align_right">64.1</td>
<td id="A1.T9.1.3.2.3" class="ltx_td ltx_align_right">68.3</td>
<td id="A1.T9.1.3.2.4" class="ltx_td ltx_align_right">68.7</td>
<td id="A1.T9.1.3.2.5" class="ltx_td ltx_align_right">68.1</td>
<td id="A1.T9.1.3.2.6" class="ltx_td ltx_align_right">69.4</td>
<td id="A1.T9.1.3.2.7" class="ltx_td ltx_align_right">64.3</td>
<td id="A1.T9.1.3.2.8" class="ltx_td ltx_align_right">62.3</td>
<td id="A1.T9.1.3.2.9" class="ltx_td ltx_align_right">69.9</td>
<td id="A1.T9.1.3.2.10" class="ltx_td ltx_align_right ltx_border_r">65.0</td>
<td id="A1.T9.1.3.2.11" class="ltx_td ltx_nopad_r ltx_align_right">66.7</td>
</tr>
<tr id="A1.T9.1.4.3" class="ltx_tr">
<th id="A1.T9.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="A1.T9.1.4.3.2" class="ltx_td ltx_align_right">63.5</td>
<td id="A1.T9.1.4.3.3" class="ltx_td ltx_align_right">67.9</td>
<td id="A1.T9.1.4.3.4" class="ltx_td ltx_align_right">68.8</td>
<td id="A1.T9.1.4.3.5" class="ltx_td ltx_align_right">68.0</td>
<td id="A1.T9.1.4.3.6" class="ltx_td ltx_align_right">69.1</td>
<td id="A1.T9.1.4.3.7" class="ltx_td ltx_align_right">63.6</td>
<td id="A1.T9.1.4.3.8" class="ltx_td ltx_align_right">61.8</td>
<td id="A1.T9.1.4.3.9" class="ltx_td ltx_align_right">69.2</td>
<td id="A1.T9.1.4.3.10" class="ltx_td ltx_align_right ltx_border_r">64.9</td>
<td id="A1.T9.1.4.3.11" class="ltx_td ltx_nopad_r ltx_align_right">66.3</td>
</tr>
<tr id="A1.T9.1.5.4" class="ltx_tr">
<th id="A1.T9.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8</th>
<td id="A1.T9.1.5.4.2" class="ltx_td ltx_align_right">62.6</td>
<td id="A1.T9.1.5.4.3" class="ltx_td ltx_align_right">67.1</td>
<td id="A1.T9.1.5.4.4" class="ltx_td ltx_align_right">68.2</td>
<td id="A1.T9.1.5.4.5" class="ltx_td ltx_align_right">67.4</td>
<td id="A1.T9.1.5.4.6" class="ltx_td ltx_align_right">68.3</td>
<td id="A1.T9.1.5.4.7" class="ltx_td ltx_align_right">62.9</td>
<td id="A1.T9.1.5.4.8" class="ltx_td ltx_align_right">60.8</td>
<td id="A1.T9.1.5.4.9" class="ltx_td ltx_align_right">68.7</td>
<td id="A1.T9.1.5.4.10" class="ltx_td ltx_align_right ltx_border_r">64.1</td>
<td id="A1.T9.1.5.4.11" class="ltx_td ltx_nopad_r ltx_align_right">65.6</td>
</tr>
<tr id="A1.T9.1.6.5" class="ltx_tr">
<th id="A1.T9.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4-g</th>
<td id="A1.T9.1.6.5.2" class="ltx_td ltx_align_right">62.9</td>
<td id="A1.T9.1.6.5.3" class="ltx_td ltx_align_right">67.5</td>
<td id="A1.T9.1.6.5.4" class="ltx_td ltx_align_right">68.2</td>
<td id="A1.T9.1.6.5.5" class="ltx_td ltx_align_right">67.6</td>
<td id="A1.T9.1.6.5.6" class="ltx_td ltx_align_right">68.6</td>
<td id="A1.T9.1.6.5.7" class="ltx_td ltx_align_right">62.8</td>
<td id="A1.T9.1.6.5.8" class="ltx_td ltx_align_right">61.1</td>
<td id="A1.T9.1.6.5.9" class="ltx_td ltx_align_right">68.6</td>
<td id="A1.T9.1.6.5.10" class="ltx_td ltx_align_right ltx_border_r">64.0</td>
<td id="A1.T9.1.6.5.11" class="ltx_td ltx_nopad_r ltx_align_right">65.7</td>
</tr>
<tr id="A1.T9.1.7.6" class="ltx_tr">
<th id="A1.T9.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4</th>
<td id="A1.T9.1.7.6.2" class="ltx_td ltx_align_right">60.5</td>
<td id="A1.T9.1.7.6.3" class="ltx_td ltx_align_right">65.7</td>
<td id="A1.T9.1.7.6.4" class="ltx_td ltx_align_right">66.5</td>
<td id="A1.T9.1.7.6.5" class="ltx_td ltx_align_right">65.4</td>
<td id="A1.T9.1.7.6.6" class="ltx_td ltx_align_right">66.6</td>
<td id="A1.T9.1.7.6.7" class="ltx_td ltx_align_right">61.1</td>
<td id="A1.T9.1.7.6.8" class="ltx_td ltx_align_right">59.3</td>
<td id="A1.T9.1.7.6.9" class="ltx_td ltx_align_right">66.7</td>
<td id="A1.T9.1.7.6.10" class="ltx_td ltx_align_right ltx_border_r">62.1</td>
<td id="A1.T9.1.7.6.11" class="ltx_td ltx_nopad_r ltx_align_right">63.8</td>
</tr>
<tr id="A1.T9.1.8.7" class="ltx_tr">
<th id="A1.T9.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t" rowspan="4"><span id="A1.T9.1.8.7.1.1" class="ltx_text">35B</span></th>
<th id="A1.T9.1.8.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="A1.T9.1.8.7.3" class="ltx_td ltx_align_right ltx_border_t">56.5</td>
<td id="A1.T9.1.8.7.4" class="ltx_td ltx_align_right ltx_border_t">60.7</td>
<td id="A1.T9.1.8.7.5" class="ltx_td ltx_align_right ltx_border_t">62.3</td>
<td id="A1.T9.1.8.7.6" class="ltx_td ltx_align_right ltx_border_t">61.8</td>
<td id="A1.T9.1.8.7.7" class="ltx_td ltx_align_right ltx_border_t">62.0</td>
<td id="A1.T9.1.8.7.8" class="ltx_td ltx_align_right ltx_border_t">56.4</td>
<td id="A1.T9.1.8.7.9" class="ltx_td ltx_align_right ltx_border_t">54.8</td>
<td id="A1.T9.1.8.7.10" class="ltx_td ltx_align_right ltx_border_t">62.0</td>
<td id="A1.T9.1.8.7.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">57.9</td>
<td id="A1.T9.1.8.7.12" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">59.4</td>
</tr>
<tr id="A1.T9.1.9.8" class="ltx_tr">
<th id="A1.T9.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8</th>
<td id="A1.T9.1.9.8.2" class="ltx_td ltx_align_right">56.5</td>
<td id="A1.T9.1.9.8.3" class="ltx_td ltx_align_right">60.6</td>
<td id="A1.T9.1.9.8.4" class="ltx_td ltx_align_right">62.2</td>
<td id="A1.T9.1.9.8.5" class="ltx_td ltx_align_right">61.8</td>
<td id="A1.T9.1.9.8.6" class="ltx_td ltx_align_right">61.9</td>
<td id="A1.T9.1.9.8.7" class="ltx_td ltx_align_right">56.4</td>
<td id="A1.T9.1.9.8.8" class="ltx_td ltx_align_right">54.7</td>
<td id="A1.T9.1.9.8.9" class="ltx_td ltx_align_right">62.1</td>
<td id="A1.T9.1.9.8.10" class="ltx_td ltx_align_right ltx_border_r">57.9</td>
<td id="A1.T9.1.9.8.11" class="ltx_td ltx_nopad_r ltx_align_right">59.3</td>
</tr>
<tr id="A1.T9.1.10.9" class="ltx_tr">
<th id="A1.T9.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8</th>
<td id="A1.T9.1.10.9.2" class="ltx_td ltx_align_right">56.4</td>
<td id="A1.T9.1.10.9.3" class="ltx_td ltx_align_right">60.5</td>
<td id="A1.T9.1.10.9.4" class="ltx_td ltx_align_right">62.5</td>
<td id="A1.T9.1.10.9.5" class="ltx_td ltx_align_right">61.9</td>
<td id="A1.T9.1.10.9.6" class="ltx_td ltx_align_right">62.0</td>
<td id="A1.T9.1.10.9.7" class="ltx_td ltx_align_right">55.8</td>
<td id="A1.T9.1.10.9.8" class="ltx_td ltx_align_right">54.5</td>
<td id="A1.T9.1.10.9.9" class="ltx_td ltx_align_right">61.8</td>
<td id="A1.T9.1.10.9.10" class="ltx_td ltx_align_right ltx_border_r">58.1</td>
<td id="A1.T9.1.10.9.11" class="ltx_td ltx_nopad_r ltx_align_right">59.3</td>
</tr>
<tr id="A1.T9.1.11.10" class="ltx_tr">
<th id="A1.T9.1.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">W4-g</th>
<td id="A1.T9.1.11.10.2" class="ltx_td ltx_align_right ltx_border_bb">55.4</td>
<td id="A1.T9.1.11.10.3" class="ltx_td ltx_align_right ltx_border_bb">59.7</td>
<td id="A1.T9.1.11.10.4" class="ltx_td ltx_align_right ltx_border_bb">62.0</td>
<td id="A1.T9.1.11.10.5" class="ltx_td ltx_align_right ltx_border_bb">61.0</td>
<td id="A1.T9.1.11.10.6" class="ltx_td ltx_align_right ltx_border_bb">60.7</td>
<td id="A1.T9.1.11.10.7" class="ltx_td ltx_align_right ltx_border_bb">54.4</td>
<td id="A1.T9.1.11.10.8" class="ltx_td ltx_align_right ltx_border_bb">53.2</td>
<td id="A1.T9.1.11.10.9" class="ltx_td ltx_align_right ltx_border_bb">60.8</td>
<td id="A1.T9.1.11.10.10" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r">56.6</td>
<td id="A1.T9.1.11.10.11" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb">58.2</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A9: </span><span id="A1.T9.3.1" class="ltx_text ltx_font_bold">mMMLU scores for Command Models.</span> (Accuracy)</figcaption>
</figure>
<figure id="A1.T10" class="ltx_table">
<div id="A1.T10.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:123.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-66.7pt,19.1pt) scale(0.764648409620307,0.764648409620307) ;">
<table id="A1.T10.1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T10.1.1.1" class="ltx_tr">
<td id="A1.T10.1.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="A1.T10.1.1.1.3" class="ltx_td ltx_border_r ltx_border_tt"></td>
<td id="A1.T10.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.4.1" class="ltx_text ltx_font_bold">ar</span></td>
<td id="A1.T10.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.5.1" class="ltx_text ltx_font_bold">de</span></td>
<td id="A1.T10.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.6.1" class="ltx_text ltx_font_bold">es</span></td>
<td id="A1.T10.1.1.1.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.7.1" class="ltx_text ltx_font_bold">fr</span></td>
<td id="A1.T10.1.1.1.8" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.8.1" class="ltx_text ltx_font_bold">it</span></td>
<td id="A1.T10.1.1.1.9" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.9.1" class="ltx_text ltx_font_bold">ja</span></td>
<td id="A1.T10.1.1.1.10" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.10.1" class="ltx_text ltx_font_bold">ko</span></td>
<td id="A1.T10.1.1.1.11" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.11.1" class="ltx_text ltx_font_bold">pt</span></td>
<td id="A1.T10.1.1.1.12" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="A1.T10.1.1.1.12.1" class="ltx_text ltx_font_bold">zh</span></td>
<td id="A1.T10.1.1.1.13" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.13.1" class="ltx_text ltx_font_bold">Avg</span></td>
<td id="A1.T10.1.1.1.14" class="ltx_td ltx_align_center ltx_border_tt"><span id="A1.T10.1.1.1.14.1" class="ltx_text ltx_font_bold">Ltn/IE</span></td>
<td id="A1.T10.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt"><math id="A1.T10.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T10.1.1.1.1.m1.1a"><mo id="A1.T10.1.1.1.1.m1.1.1" xref="A1.T10.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T10.1.1.1.1.m1.1b"><not id="A1.T10.1.1.1.1.m1.1.1.cmml" xref="A1.T10.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T10.1.1.1.1.m1.1c">\neg</annotation></semantics></math></td>
</tr>
<tr id="A1.T10.1.1.2.1" class="ltx_tr">
<td id="A1.T10.1.1.2.1.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T10.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T10.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBFEFD;"><span id="A1.T10.1.1.2.1.3.1" class="ltx_text" style="background-color:#FBFEFD;">0.2%</span></td>
<td id="A1.T10.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.2.1.4.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.2.1.5.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCFEFD;"><span id="A1.T10.1.1.2.1.6.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T10.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCFEFD;"><span id="A1.T10.1.1.2.1.7.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T10.1.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAFA;"><span id="A1.T10.1.1.2.1.8.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="A1.T10.1.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.2.1.9.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.2.1.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFA;"><span id="A1.T10.1.1.2.1.10.1" class="ltx_text" style="background-color:#FEFBFA;">-0.1%</span></td>
<td id="A1.T10.1.1.2.1.11" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.2.1.11.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.2.1.12" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.2.1.12.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.2.1.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.2.1.13.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.2.1.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.2.1.14.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
</tr>
<tr id="A1.T10.1.1.3.2" class="ltx_tr">
<td id="A1.T10.1.1.3.2.1" class="ltx_td"></td>
<td id="A1.T10.1.1.3.2.2" class="ltx_td ltx_align_left ltx_border_r">W8A8-sq</td>
<td id="A1.T10.1.1.3.2.3" class="ltx_td ltx_align_right" style="background-color:#FAE8E7;"><span id="A1.T10.1.1.3.2.3.1" class="ltx_text" style="background-color:#FAE8E7;">-0.8%</span></td>
<td id="A1.T10.1.1.3.2.4" class="ltx_td ltx_align_right" style="background-color:#FCEFEE;"><span id="A1.T10.1.1.3.2.4.1" class="ltx_text" style="background-color:#FCEFEE;">-0.6%</span></td>
<td id="A1.T10.1.1.3.2.5" class="ltx_td ltx_align_right" style="background-color:#FCFEFD;"><span id="A1.T10.1.1.3.2.5.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T10.1.1.3.2.6" class="ltx_td ltx_align_right" style="background-color:#FEFFFE;"><span id="A1.T10.1.1.3.2.6.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="A1.T10.1.1.3.2.7" class="ltx_td ltx_align_right" style="background-color:#FDF7F6;"><span id="A1.T10.1.1.3.2.7.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
<td id="A1.T10.1.1.3.2.8" class="ltx_td ltx_align_right" style="background-color:#F8DCDA;"><span id="A1.T10.1.1.3.2.8.1" class="ltx_text" style="background-color:#F8DCDA;">-1.3%</span></td>
<td id="A1.T10.1.1.3.2.9" class="ltx_td ltx_align_right" style="background-color:#FAE9E8;"><span id="A1.T10.1.1.3.2.9.1" class="ltx_text" style="background-color:#FAE9E8;">-0.8%</span></td>
<td id="A1.T10.1.1.3.2.10" class="ltx_td ltx_align_right" style="background-color:#F9E2E0;"><span id="A1.T10.1.1.3.2.10.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="A1.T10.1.1.3.2.11" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#FEFAFA;"><span id="A1.T10.1.1.3.2.11.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="A1.T10.1.1.3.2.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FCF0EF;"><span id="A1.T10.1.1.3.2.12.1" class="ltx_text" style="background-color:#FCF0EF;">-0.5%</span></td>
<td id="A1.T10.1.1.3.2.13" class="ltx_td ltx_align_right" style="background-color:#FDF5F5;"><span id="A1.T10.1.1.3.2.13.1" class="ltx_text" style="background-color:#FDF5F5;">-0.4%</span></td>
<td id="A1.T10.1.1.3.2.14" class="ltx_td ltx_align_right" style="background-color:#FBEAE9;"><span id="A1.T10.1.1.3.2.14.1" class="ltx_text" style="background-color:#FBEAE9;">-0.8%</span></td>
</tr>
<tr id="A1.T10.1.1.4.3" class="ltx_tr">
<td id="A1.T10.1.1.4.3.1" class="ltx_td"></td>
<td id="A1.T10.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T10.1.1.4.3.3" class="ltx_td ltx_align_right" style="background-color:#F4C5C1;"><span id="A1.T10.1.1.4.3.3.1" class="ltx_text" style="background-color:#F4C5C1;">-2.2%</span></td>
<td id="A1.T10.1.1.4.3.4" class="ltx_td ltx_align_right" style="background-color:#F6D0CD;"><span id="A1.T10.1.1.4.3.4.1" class="ltx_text" style="background-color:#F6D0CD;">-1.8%</span></td>
<td id="A1.T10.1.1.4.3.5" class="ltx_td ltx_align_right" style="background-color:#FBECEB;"><span id="A1.T10.1.1.4.3.5.1" class="ltx_text" style="background-color:#FBECEB;">-0.7%</span></td>
<td id="A1.T10.1.1.4.3.6" class="ltx_td ltx_align_right" style="background-color:#FAE5E4;"><span id="A1.T10.1.1.4.3.6.1" class="ltx_text" style="background-color:#FAE5E4;">-1.0%</span></td>
<td id="A1.T10.1.1.4.3.7" class="ltx_td ltx_align_right" style="background-color:#F7D8D6;"><span id="A1.T10.1.1.4.3.7.1" class="ltx_text" style="background-color:#F7D8D6;">-1.5%</span></td>
<td id="A1.T10.1.1.4.3.8" class="ltx_td ltx_align_right" style="background-color:#F3C2BE;"><span id="A1.T10.1.1.4.3.8.1" class="ltx_text" style="background-color:#F3C2BE;">-2.3%</span></td>
<td id="A1.T10.1.1.4.3.9" class="ltx_td ltx_align_right" style="background-color:#F2BFBA;"><span id="A1.T10.1.1.4.3.9.1" class="ltx_text" style="background-color:#F2BFBA;">-2.4%</span></td>
<td id="A1.T10.1.1.4.3.10" class="ltx_td ltx_align_right" style="background-color:#F5CFCC;"><span id="A1.T10.1.1.4.3.10.1" class="ltx_text" style="background-color:#F5CFCC;">-1.8%</span></td>
<td id="A1.T10.1.1.4.3.11" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F8DAD7;"><span id="A1.T10.1.1.4.3.11.1" class="ltx_text" style="background-color:#F8DAD7;">-1.4%</span></td>
<td id="A1.T10.1.1.4.3.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F6D3D0;"><span id="A1.T10.1.1.4.3.12.1" class="ltx_text" style="background-color:#F6D3D0;">-1.7%</span></td>
<td id="A1.T10.1.1.4.3.13" class="ltx_td ltx_align_right" style="background-color:#F8DBD9;"><span id="A1.T10.1.1.4.3.13.1" class="ltx_text" style="background-color:#F8DBD9;">-1.3%</span></td>
<td id="A1.T10.1.1.4.3.14" class="ltx_td ltx_align_right" style="background-color:#F4C8C4;"><span id="A1.T10.1.1.4.3.14.1" class="ltx_text" style="background-color:#F4C8C4;">-2.1%</span></td>
</tr>
<tr id="A1.T10.1.1.5.4" class="ltx_tr">
<td id="A1.T10.1.1.5.4.1" class="ltx_td"></td>
<td id="A1.T10.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_r">W4-g</td>
<td id="A1.T10.1.1.5.4.3" class="ltx_td ltx_align_right" style="background-color:#F6D1CE;"><span id="A1.T10.1.1.5.4.3.1" class="ltx_text" style="background-color:#F6D1CE;">-1.7%</span></td>
<td id="A1.T10.1.1.5.4.4" class="ltx_td ltx_align_right" style="background-color:#F9E0DE;"><span id="A1.T10.1.1.5.4.4.1" class="ltx_text" style="background-color:#F9E0DE;">-1.2%</span></td>
<td id="A1.T10.1.1.5.4.5" class="ltx_td ltx_align_right" style="background-color:#FBECEB;"><span id="A1.T10.1.1.5.4.5.1" class="ltx_text" style="background-color:#FBECEB;">-0.7%</span></td>
<td id="A1.T10.1.1.5.4.6" class="ltx_td ltx_align_right" style="background-color:#FCEFEE;"><span id="A1.T10.1.1.5.4.6.1" class="ltx_text" style="background-color:#FCEFEE;">-0.6%</span></td>
<td id="A1.T10.1.1.5.4.7" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="A1.T10.1.1.5.4.7.1" class="ltx_text" style="background-color:#F9E4E2;">-1.0%</span></td>
<td id="A1.T10.1.1.5.4.8" class="ltx_td ltx_align_right" style="background-color:#F2BDB9;"><span id="A1.T10.1.1.5.4.8.1" class="ltx_text" style="background-color:#F2BDB9;">-2.5%</span></td>
<td id="A1.T10.1.1.5.4.9" class="ltx_td ltx_align_right" style="background-color:#F5CCC9;"><span id="A1.T10.1.1.5.4.9.1" class="ltx_text" style="background-color:#F5CCC9;">-1.9%</span></td>
<td id="A1.T10.1.1.5.4.10" class="ltx_td ltx_align_right" style="background-color:#F5CAC7;"><span id="A1.T10.1.1.5.4.10.1" class="ltx_text" style="background-color:#F5CAC7;">-2.0%</span></td>
<td id="A1.T10.1.1.5.4.11" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F7D6D3;"><span id="A1.T10.1.1.5.4.11.1" class="ltx_text" style="background-color:#F7D6D3;">-1.5%</span></td>
<td id="A1.T10.1.1.5.4.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F7D8D6;"><span id="A1.T10.1.1.5.4.12.1" class="ltx_text" style="background-color:#F7D8D6;">-1.5%</span></td>
<td id="A1.T10.1.1.5.4.13" class="ltx_td ltx_align_right" style="background-color:#F9E2E0;"><span id="A1.T10.1.1.5.4.13.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="A1.T10.1.1.5.4.14" class="ltx_td ltx_align_right" style="background-color:#F5CCC9;"><span id="A1.T10.1.1.5.4.14.1" class="ltx_text" style="background-color:#F5CCC9;">-1.9%</span></td>
</tr>
<tr id="A1.T10.1.1.6.5" class="ltx_tr">
<td id="A1.T10.1.1.6.5.1" class="ltx_td ltx_align_left"><span id="A1.T10.1.1.6.5.1.1" class="ltx_text">103B</span></td>
<td id="A1.T10.1.1.6.5.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T10.1.1.6.5.3" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T10.1.1.6.5.3.1" class="ltx_text" style="background-color:#E67C73;">-5.5%</span></td>
<td id="A1.T10.1.1.6.5.4" class="ltx_td ltx_align_right" style="background-color:#EB9B94;"><span id="A1.T10.1.1.6.5.4.1" class="ltx_text" style="background-color:#EB9B94;">-3.8%</span></td>
<td id="A1.T10.1.1.6.5.5" class="ltx_td ltx_align_right" style="background-color:#EFACA6;"><span id="A1.T10.1.1.6.5.5.1" class="ltx_text" style="background-color:#EFACA6;">-3.1%</span></td>
<td id="A1.T10.1.1.6.5.6" class="ltx_td ltx_align_right" style="background-color:#EB9992;"><span id="A1.T10.1.1.6.5.6.1" class="ltx_text" style="background-color:#EB9992;">-3.9%</span></td>
<td id="A1.T10.1.1.6.5.7" class="ltx_td ltx_align_right" style="background-color:#EB9A93;"><span id="A1.T10.1.1.6.5.7.1" class="ltx_text" style="background-color:#EB9A93;">-3.8%</span></td>
<td id="A1.T10.1.1.6.5.8" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T10.1.1.6.5.8.1" class="ltx_text" style="background-color:#E67C73;">-5.1%</span></td>
<td id="A1.T10.1.1.6.5.9" class="ltx_td ltx_align_right" style="background-color:#E78279;"><span id="A1.T10.1.1.6.5.9.1" class="ltx_text" style="background-color:#E78279;">-4.8%</span></td>
<td id="A1.T10.1.1.6.5.10" class="ltx_td ltx_align_right" style="background-color:#E78179;"><span id="A1.T10.1.1.6.5.10.1" class="ltx_text" style="background-color:#E78179;">-4.8%</span></td>
<td id="A1.T10.1.1.6.5.11" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#E88B83;"><span id="A1.T10.1.1.6.5.11.1" class="ltx_text" style="background-color:#E88B83;">-4.4%</span></td>
<td id="A1.T10.1.1.6.5.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E98C85;"><span id="A1.T10.1.1.6.5.12.1" class="ltx_text" style="background-color:#E98C85;">-4.4%</span></td>
<td id="A1.T10.1.1.6.5.13" class="ltx_td ltx_align_right" style="background-color:#EB9992;"><span id="A1.T10.1.1.6.5.13.1" class="ltx_text" style="background-color:#EB9992;">-3.9%</span></td>
<td id="A1.T10.1.1.6.5.14" class="ltx_td ltx_align_right" style="background-color:#E67D74;"><span id="A1.T10.1.1.6.5.14.1" class="ltx_text" style="background-color:#E67D74;">-4.9%</span></td>
</tr>
<tr id="A1.T10.1.1.7.6" class="ltx_tr">
<td id="A1.T10.1.1.7.6.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T10.1.1.7.6.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T10.1.1.7.6.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.7.6.3.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.7.6.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAFA;"><span id="A1.T10.1.1.7.6.4.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="A1.T10.1.1.7.6.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAFA;"><span id="A1.T10.1.1.7.6.5.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="A1.T10.1.1.7.6.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.7.6.6.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.7.6.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAFA;"><span id="A1.T10.1.1.7.6.7.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="A1.T10.1.1.7.6.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.7.6.8.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.7.6.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAF9;"><span id="A1.T10.1.1.7.6.9.1" class="ltx_text" style="background-color:#FEFAF9;">-0.2%</span></td>
<td id="A1.T10.1.1.7.6.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBFEFD;"><span id="A1.T10.1.1.7.6.10.1" class="ltx_text" style="background-color:#FBFEFD;">0.2%</span></td>
<td id="A1.T10.1.1.7.6.11" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.7.6.11.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.7.6.12" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FEFDFD;"><span id="A1.T10.1.1.7.6.12.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T10.1.1.7.6.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="A1.T10.1.1.7.6.13.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T10.1.1.7.6.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="A1.T10.1.1.7.6.14.1" class="ltx_text" style="background-color:#FEFDFD;">0.0%</span></td>
</tr>
<tr id="A1.T10.1.1.8.7" class="ltx_tr">
<td id="A1.T10.1.1.8.7.1" class="ltx_td"></td>
<td id="A1.T10.1.1.8.7.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T10.1.1.8.7.3" class="ltx_td ltx_align_right" style="background-color:#FEFAFA;"><span id="A1.T10.1.1.8.7.3.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="A1.T10.1.1.8.7.4" class="ltx_td ltx_align_right" style="background-color:#FDF6F5;"><span id="A1.T10.1.1.8.7.4.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="A1.T10.1.1.8.7.5" class="ltx_td ltx_align_right" style="background-color:#F7FCFA;"><span id="A1.T10.1.1.8.7.5.1" class="ltx_text" style="background-color:#F7FCFA;">0.3%</span></td>
<td id="A1.T10.1.1.8.7.6" class="ltx_td ltx_align_right" style="background-color:#FBFEFD;"><span id="A1.T10.1.1.8.7.6.1" class="ltx_text" style="background-color:#FBFEFD;">0.2%</span></td>
<td id="A1.T10.1.1.8.7.7" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T10.1.1.8.7.7.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T10.1.1.8.7.8" class="ltx_td ltx_align_right" style="background-color:#F9E3E1;"><span id="A1.T10.1.1.8.7.8.1" class="ltx_text" style="background-color:#F9E3E1;">-1.1%</span></td>
<td id="A1.T10.1.1.8.7.9" class="ltx_td ltx_align_right" style="background-color:#FCF0EF;"><span id="A1.T10.1.1.8.7.9.1" class="ltx_text" style="background-color:#FCF0EF;">-0.5%</span></td>
<td id="A1.T10.1.1.8.7.10" class="ltx_td ltx_align_right" style="background-color:#FDF6F5;"><span id="A1.T10.1.1.8.7.10.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="A1.T10.1.1.8.7.11" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F7FCF9;"><span id="A1.T10.1.1.8.7.11.1" class="ltx_text" style="background-color:#F7FCF9;">0.3%</span></td>
<td id="A1.T10.1.1.8.7.12" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFAF9;"><span id="A1.T10.1.1.8.7.12.1" class="ltx_text" style="background-color:#FEFAF9;">-0.2%</span></td>
<td id="A1.T10.1.1.8.7.13" class="ltx_td ltx_align_right" style="background-color:#FEFEFE;"><span id="A1.T10.1.1.8.7.13.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="A1.T10.1.1.8.7.14" class="ltx_td ltx_align_right" style="background-color:#FDF5F4;"><span id="A1.T10.1.1.8.7.14.1" class="ltx_text" style="background-color:#FDF5F4;">-0.4%</span></td>
</tr>
<tr id="A1.T10.1.1.9.8" class="ltx_tr">
<td id="A1.T10.1.1.9.8.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A1.T10.1.1.9.8.1.1" class="ltx_text">35B</span></td>
<td id="A1.T10.1.1.9.8.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4-g</td>
<td id="A1.T10.1.1.9.8.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F5CBC8;"><span id="A1.T10.1.1.9.8.3.1" class="ltx_text" style="background-color:#F5CBC8;">-1.9%</span></td>
<td id="A1.T10.1.1.9.8.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F6D3D0;"><span id="A1.T10.1.1.9.8.4.1" class="ltx_text" style="background-color:#F6D3D0;">-1.6%</span></td>
<td id="A1.T10.1.1.9.8.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FCF2F1;"><span id="A1.T10.1.1.9.8.5.1" class="ltx_text" style="background-color:#FCF2F1;">-0.5%</span></td>
<td id="A1.T10.1.1.9.8.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F8DDDA;"><span id="A1.T10.1.1.9.8.6.1" class="ltx_text" style="background-color:#F8DDDA;">-1.3%</span></td>
<td id="A1.T10.1.1.9.8.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F4C8C4;"><span id="A1.T10.1.1.9.8.7.1" class="ltx_text" style="background-color:#F4C8C4;">-2.1%</span></td>
<td id="A1.T10.1.1.9.8.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EDA29B;"><span id="A1.T10.1.1.9.8.8.1" class="ltx_text" style="background-color:#EDA29B;">-3.5%</span></td>
<td id="A1.T10.1.1.9.8.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F0B2AD;"><span id="A1.T10.1.1.9.8.9.1" class="ltx_text" style="background-color:#F0B2AD;">-2.9%</span></td>
<td id="A1.T10.1.1.9.8.10" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F5CCC8;"><span id="A1.T10.1.1.9.8.10.1" class="ltx_text" style="background-color:#F5CCC8;">-1.9%</span></td>
<td id="A1.T10.1.1.9.8.11" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#F3C4C0;"><span id="A1.T10.1.1.9.8.11.1" class="ltx_text" style="background-color:#F3C4C0;">-2.2%</span></td>
<td id="A1.T10.1.1.9.8.12" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F4CAC6;"><span id="A1.T10.1.1.9.8.12.1" class="ltx_text" style="background-color:#F4CAC6;">-2.0%</span></td>
<td id="A1.T10.1.1.9.8.13" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F7D7D5;"><span id="A1.T10.1.1.9.8.13.1" class="ltx_text" style="background-color:#F7D7D5;">-1.5%</span></td>
<td id="A1.T10.1.1.9.8.14" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F1B9B4;"><span id="A1.T10.1.1.9.8.14.1" class="ltx_text" style="background-color:#F1B9B4;">-2.7%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A10: </span><span id="A1.T10.4.1" class="ltx_text ltx_font_bold">Relative performance (%<math id="A1.T10.4.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="A1.T10.4.1.m1.1b"><mi mathvariant="normal" id="A1.T10.4.1.m1.1.1" xref="A1.T10.4.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="A1.T10.4.1.m1.1c"><ci id="A1.T10.4.1.m1.1.1.cmml" xref="A1.T10.4.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T10.4.1.m1.1d">\Delta</annotation></semantics></math>) vs. FP16 for Command Models on mMMLU</span>. Ltn/IE are Latin-script/Indo-European languages: de, es, fr, it, pt. <math id="A1.T10.5.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T10.5.m1.1b"><mo id="A1.T10.5.m1.1.1" xref="A1.T10.5.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T10.5.m1.1c"><not id="A1.T10.5.m1.1.1.cmml" xref="A1.T10.5.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T10.5.m1.1d">\neg</annotation></semantics></math> are the rest: ar, ja, ko, zh.</figcaption>
</figure>
<figure id="A1.T11" class="ltx_table">
<div id="A1.T11.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:79.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-125.4pt,23.1pt) scale(0.633586163115955,0.633586163115955) ;">
<table id="A1.T11.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T11.1.1.1.1" class="ltx_tr">
<th id="A1.T11.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="A1.T11.1.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="A1.T11.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Avg (<span id="A1.T11.1.1.1.1.3.1.1" class="ltx_text ltx_font_italic">-en</span>)</span></th>
<th id="A1.T11.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="A1.T11.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.5.1" class="ltx_text ltx_font_bold">ar</span></th>
<th id="A1.T11.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.6.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T11.1.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.7.1" class="ltx_text ltx_font_bold">en</span></th>
<th id="A1.T11.1.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.8.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T11.1.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.9.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T11.1.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.10.1" class="ltx_text ltx_font_bold">hi</span></th>
<th id="A1.T11.1.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.11.1" class="ltx_text ltx_font_bold">id</span></th>
<th id="A1.T11.1.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.12.1" class="ltx_text ltx_font_bold">it</span></th>
<th id="A1.T11.1.1.1.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.13.1" class="ltx_text ltx_font_bold">nl</span></th>
<th id="A1.T11.1.1.1.1.14" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.14.1" class="ltx_text ltx_font_bold">pt</span></th>
<th id="A1.T11.1.1.1.1.15" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.15.1" class="ltx_text ltx_font_bold">ro</span></th>
<th id="A1.T11.1.1.1.1.16" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.16.1" class="ltx_text ltx_font_bold">ru</span></th>
<th id="A1.T11.1.1.1.1.17" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.17.1" class="ltx_text ltx_font_bold">uk</span></th>
<th id="A1.T11.1.1.1.1.18" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.18.1" class="ltx_text ltx_font_bold">vi</span></th>
<th id="A1.T11.1.1.1.1.19" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T11.1.1.1.1.19.1" class="ltx_text ltx_font_bold">zh</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T11.1.1.2.1" class="ltx_tr">
<td id="A1.T11.1.1.2.1.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T11.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T11.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.2.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">58.2</span></td>
<td id="A1.T11.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.2.1.4.1" class="ltx_text" style="background-color:#E6E6E6;">58.8</span></td>
<td id="A1.T11.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t">53.9</td>
<td id="A1.T11.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t">60.4</td>
<td id="A1.T11.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t">66.7</td>
<td id="A1.T11.1.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t">61.6</td>
<td id="A1.T11.1.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t">62.0</td>
<td id="A1.T11.1.1.2.1.10" class="ltx_td ltx_align_right ltx_border_t">47.8</td>
<td id="A1.T11.1.1.2.1.11" class="ltx_td ltx_align_right ltx_border_t">58.9</td>
<td id="A1.T11.1.1.2.1.12" class="ltx_td ltx_align_right ltx_border_t">61.5</td>
<td id="A1.T11.1.1.2.1.13" class="ltx_td ltx_align_right ltx_border_t">60.3</td>
<td id="A1.T11.1.1.2.1.14" class="ltx_td ltx_align_right ltx_border_t">62.0</td>
<td id="A1.T11.1.1.2.1.15" class="ltx_td ltx_align_right ltx_border_t">59.7</td>
<td id="A1.T11.1.1.2.1.16" class="ltx_td ltx_align_right ltx_border_t">57.8</td>
<td id="A1.T11.1.1.2.1.17" class="ltx_td ltx_align_right ltx_border_t">56.3</td>
<td id="A1.T11.1.1.2.1.18" class="ltx_td ltx_align_right ltx_border_t">55.3</td>
<td id="A1.T11.1.1.2.1.19" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">57.5</td>
</tr>
<tr id="A1.T11.1.1.3.2" class="ltx_tr">
<td id="A1.T11.1.1.3.2.1" class="ltx_td ltx_align_center">Aya-23-35b</td>
<td id="A1.T11.1.1.3.2.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T11.1.1.3.2.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.3.2.3.1" class="ltx_text" style="background-color:#E6E6E6;">57.9</span></td>
<td id="A1.T11.1.1.3.2.4" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.3.2.4.1" class="ltx_text" style="background-color:#E6E6E6;">58.5</span></td>
<td id="A1.T11.1.1.3.2.5" class="ltx_td ltx_align_right">53.8</td>
<td id="A1.T11.1.1.3.2.6" class="ltx_td ltx_align_right">60.0</td>
<td id="A1.T11.1.1.3.2.7" class="ltx_td ltx_align_right">66.2</td>
<td id="A1.T11.1.1.3.2.8" class="ltx_td ltx_align_right">61.7</td>
<td id="A1.T11.1.1.3.2.9" class="ltx_td ltx_align_right">61.7</td>
<td id="A1.T11.1.1.3.2.10" class="ltx_td ltx_align_right">47.4</td>
<td id="A1.T11.1.1.3.2.11" class="ltx_td ltx_align_right">58.7</td>
<td id="A1.T11.1.1.3.2.12" class="ltx_td ltx_align_right">61.1</td>
<td id="A1.T11.1.1.3.2.13" class="ltx_td ltx_align_right">60.0</td>
<td id="A1.T11.1.1.3.2.14" class="ltx_td ltx_align_right">61.6</td>
<td id="A1.T11.1.1.3.2.15" class="ltx_td ltx_align_right">59.1</td>
<td id="A1.T11.1.1.3.2.16" class="ltx_td ltx_align_right">57.5</td>
<td id="A1.T11.1.1.3.2.17" class="ltx_td ltx_align_right">56.1</td>
<td id="A1.T11.1.1.3.2.18" class="ltx_td ltx_align_right">54.9</td>
<td id="A1.T11.1.1.3.2.19" class="ltx_td ltx_nopad_r ltx_align_right">57.5</td>
</tr>
<tr id="A1.T11.1.1.4.3" class="ltx_tr">
<td id="A1.T11.1.1.4.3.1" class="ltx_td"></td>
<td id="A1.T11.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T11.1.1.4.3.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.4.3.3.1" class="ltx_text" style="background-color:#E6E6E6;">56.6</span></td>
<td id="A1.T11.1.1.4.3.4" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.4.3.4.1" class="ltx_text" style="background-color:#E6E6E6;">57.2</span></td>
<td id="A1.T11.1.1.4.3.5" class="ltx_td ltx_align_right">52.3</td>
<td id="A1.T11.1.1.4.3.6" class="ltx_td ltx_align_right">58.7</td>
<td id="A1.T11.1.1.4.3.7" class="ltx_td ltx_align_right">65.2</td>
<td id="A1.T11.1.1.4.3.8" class="ltx_td ltx_align_right">60.3</td>
<td id="A1.T11.1.1.4.3.9" class="ltx_td ltx_align_right">60.4</td>
<td id="A1.T11.1.1.4.3.10" class="ltx_td ltx_align_right">45.7</td>
<td id="A1.T11.1.1.4.3.11" class="ltx_td ltx_align_right">57.4</td>
<td id="A1.T11.1.1.4.3.12" class="ltx_td ltx_align_right">59.8</td>
<td id="A1.T11.1.1.4.3.13" class="ltx_td ltx_align_right">58.6</td>
<td id="A1.T11.1.1.4.3.14" class="ltx_td ltx_align_right">60.5</td>
<td id="A1.T11.1.1.4.3.15" class="ltx_td ltx_align_right">57.7</td>
<td id="A1.T11.1.1.4.3.16" class="ltx_td ltx_align_right">56.5</td>
<td id="A1.T11.1.1.4.3.17" class="ltx_td ltx_align_right">55.0</td>
<td id="A1.T11.1.1.4.3.18" class="ltx_td ltx_align_right">53.8</td>
<td id="A1.T11.1.1.4.3.19" class="ltx_td ltx_nopad_r ltx_align_right">56.1</td>
</tr>
<tr id="A1.T11.1.1.5.4" class="ltx_tr">
<td id="A1.T11.1.1.5.4.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T11.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T11.1.1.5.4.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.5.4.3.1" class="ltx_text" style="background-color:#E6E6E6;">48.2</span></td>
<td id="A1.T11.1.1.5.4.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.5.4.4.1" class="ltx_text" style="background-color:#E6E6E6;">48.6</span></td>
<td id="A1.T11.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_t">45.1</td>
<td id="A1.T11.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_t">50.0</td>
<td id="A1.T11.1.1.5.4.7" class="ltx_td ltx_align_right ltx_border_t">54.6</td>
<td id="A1.T11.1.1.5.4.8" class="ltx_td ltx_align_right ltx_border_t">50.9</td>
<td id="A1.T11.1.1.5.4.9" class="ltx_td ltx_align_right ltx_border_t">51.0</td>
<td id="A1.T11.1.1.5.4.10" class="ltx_td ltx_align_right ltx_border_t">39.7</td>
<td id="A1.T11.1.1.5.4.11" class="ltx_td ltx_align_right ltx_border_t">48.8</td>
<td id="A1.T11.1.1.5.4.12" class="ltx_td ltx_align_right ltx_border_t">50.7</td>
<td id="A1.T11.1.1.5.4.13" class="ltx_td ltx_align_right ltx_border_t">49.7</td>
<td id="A1.T11.1.1.5.4.14" class="ltx_td ltx_align_right ltx_border_t">50.8</td>
<td id="A1.T11.1.1.5.4.15" class="ltx_td ltx_align_right ltx_border_t">49.9</td>
<td id="A1.T11.1.1.5.4.16" class="ltx_td ltx_align_right ltx_border_t">47.8</td>
<td id="A1.T11.1.1.5.4.17" class="ltx_td ltx_align_right ltx_border_t">46.8</td>
<td id="A1.T11.1.1.5.4.18" class="ltx_td ltx_align_right ltx_border_t">46.5</td>
<td id="A1.T11.1.1.5.4.19" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">47.1</td>
</tr>
<tr id="A1.T11.1.1.6.5" class="ltx_tr">
<td id="A1.T11.1.1.6.5.1" class="ltx_td ltx_align_center">Aya-23-8b</td>
<td id="A1.T11.1.1.6.5.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T11.1.1.6.5.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.6.5.3.1" class="ltx_text" style="background-color:#E6E6E6;">47.8</span></td>
<td id="A1.T11.1.1.6.5.4" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.6.5.4.1" class="ltx_text" style="background-color:#E6E6E6;">48.2</span></td>
<td id="A1.T11.1.1.6.5.5" class="ltx_td ltx_align_right">44.9</td>
<td id="A1.T11.1.1.6.5.6" class="ltx_td ltx_align_right">49.9</td>
<td id="A1.T11.1.1.6.5.7" class="ltx_td ltx_align_right">54.2</td>
<td id="A1.T11.1.1.6.5.8" class="ltx_td ltx_align_right">50.5</td>
<td id="A1.T11.1.1.6.5.9" class="ltx_td ltx_align_right">50.6</td>
<td id="A1.T11.1.1.6.5.10" class="ltx_td ltx_align_right">39.4</td>
<td id="A1.T11.1.1.6.5.11" class="ltx_td ltx_align_right">48.5</td>
<td id="A1.T11.1.1.6.5.12" class="ltx_td ltx_align_right">50.2</td>
<td id="A1.T11.1.1.6.5.13" class="ltx_td ltx_align_right">49.4</td>
<td id="A1.T11.1.1.6.5.14" class="ltx_td ltx_align_right">50.6</td>
<td id="A1.T11.1.1.6.5.15" class="ltx_td ltx_align_right">49.2</td>
<td id="A1.T11.1.1.6.5.16" class="ltx_td ltx_align_right">47.4</td>
<td id="A1.T11.1.1.6.5.17" class="ltx_td ltx_align_right">46.3</td>
<td id="A1.T11.1.1.6.5.18" class="ltx_td ltx_align_right">45.7</td>
<td id="A1.T11.1.1.6.5.19" class="ltx_td ltx_nopad_r ltx_align_right">46.4</td>
</tr>
<tr id="A1.T11.1.1.7.6" class="ltx_tr">
<td id="A1.T11.1.1.7.6.1" class="ltx_td ltx_border_bb"></td>
<td id="A1.T11.1.1.7.6.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4</td>
<td id="A1.T11.1.1.7.6.3" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.7.6.3.1" class="ltx_text" style="background-color:#E6E6E6;">46.7</span></td>
<td id="A1.T11.1.1.7.6.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r" style="background-color:#E6E6E6;"><span id="A1.T11.1.1.7.6.4.1" class="ltx_text" style="background-color:#E6E6E6;">47.1</span></td>
<td id="A1.T11.1.1.7.6.5" class="ltx_td ltx_align_right ltx_border_bb">43.9</td>
<td id="A1.T11.1.1.7.6.6" class="ltx_td ltx_align_right ltx_border_bb">48.4</td>
<td id="A1.T11.1.1.7.6.7" class="ltx_td ltx_align_right ltx_border_bb">53.4</td>
<td id="A1.T11.1.1.7.6.8" class="ltx_td ltx_align_right ltx_border_bb">49.4</td>
<td id="A1.T11.1.1.7.6.9" class="ltx_td ltx_align_right ltx_border_bb">49.0</td>
<td id="A1.T11.1.1.7.6.10" class="ltx_td ltx_align_right ltx_border_bb">38.4</td>
<td id="A1.T11.1.1.7.6.11" class="ltx_td ltx_align_right ltx_border_bb">47.5</td>
<td id="A1.T11.1.1.7.6.12" class="ltx_td ltx_align_right ltx_border_bb">49.1</td>
<td id="A1.T11.1.1.7.6.13" class="ltx_td ltx_align_right ltx_border_bb">47.9</td>
<td id="A1.T11.1.1.7.6.14" class="ltx_td ltx_align_right ltx_border_bb">49.1</td>
<td id="A1.T11.1.1.7.6.15" class="ltx_td ltx_align_right ltx_border_bb">48.0</td>
<td id="A1.T11.1.1.7.6.16" class="ltx_td ltx_align_right ltx_border_bb">46.2</td>
<td id="A1.T11.1.1.7.6.17" class="ltx_td ltx_align_right ltx_border_bb">45.6</td>
<td id="A1.T11.1.1.7.6.18" class="ltx_td ltx_align_right ltx_border_bb">44.9</td>
<td id="A1.T11.1.1.7.6.19" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb">46.1</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A11: </span><span id="A1.T11.3.1" class="ltx_text ltx_font_bold">Aya 23 language-specific results for mMMLU (Okapi).</span> (Accuracy)</figcaption>
</figure>
<figure id="A1.T12" class="ltx_table">
<div id="A1.T12.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:45.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-212.8pt,22.3pt) scale(0.504665000664979,0.504665000664979) ;">
<table id="A1.T12.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T12.1.1.1" class="ltx_tr">
<th id="A1.T12.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_tt"></th>
<th id="A1.T12.1.1.1.3" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="A1.T12.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.4.1" class="ltx_text ltx_font_bold">ar</span></th>
<th id="A1.T12.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.5.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T12.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.6.1" class="ltx_text ltx_font_bold">en</span></th>
<th id="A1.T12.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.7.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T12.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.8.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T12.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.9.1" class="ltx_text ltx_font_bold">hi</span></th>
<th id="A1.T12.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.10.1" class="ltx_text ltx_font_bold">id</span></th>
<th id="A1.T12.1.1.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.11.1" class="ltx_text ltx_font_bold">it</span></th>
<th id="A1.T12.1.1.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.12.1" class="ltx_text ltx_font_bold">nl</span></th>
<th id="A1.T12.1.1.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.13.1" class="ltx_text ltx_font_bold">pt</span></th>
<th id="A1.T12.1.1.1.14" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.14.1" class="ltx_text ltx_font_bold">ro</span></th>
<th id="A1.T12.1.1.1.15" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.15.1" class="ltx_text ltx_font_bold">ru</span></th>
<th id="A1.T12.1.1.1.16" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.16.1" class="ltx_text ltx_font_bold">uk</span></th>
<th id="A1.T12.1.1.1.17" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.17.1" class="ltx_text ltx_font_bold">vi</span></th>
<th id="A1.T12.1.1.1.18" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.18.1" class="ltx_text ltx_font_bold">zh</span></th>
<th id="A1.T12.1.1.1.19" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_l ltx_border_tt"><span id="A1.T12.1.1.1.19.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="A1.T12.1.1.1.20" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.20.1" class="ltx_text ltx_font_bold">Avg (<span id="A1.T12.1.1.1.20.1.1" class="ltx_text ltx_font_italic">-en</span>)</span></th>
<th id="A1.T12.1.1.1.21" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T12.1.1.1.21.1" class="ltx_text ltx_font_bold">Ltn</span></th>
<th id="A1.T12.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="A1.T12.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T12.1.1.1.1.m1.1a"><mo id="A1.T12.1.1.1.1.m1.1.1" xref="A1.T12.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T12.1.1.1.1.m1.1b"><not id="A1.T12.1.1.1.1.m1.1.1.cmml" xref="A1.T12.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T12.1.1.1.1.m1.1c">\neg</annotation></semantics></math></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T12.1.1.2.1" class="ltx_tr">
<td id="A1.T12.1.1.2.1.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T12.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T12.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFA;"><span id="A1.T12.1.1.2.1.3.1" class="ltx_text" style="background-color:#FEFBFA;">-0.2%</span></td>
<td id="A1.T12.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF1F1;"><span id="A1.T12.1.1.2.1.4.1" class="ltx_text" style="background-color:#FCF1F1;">-0.7%</span></td>
<td id="A1.T12.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F2;"><span id="A1.T12.1.1.2.1.5.1" class="ltx_text" style="background-color:#FCF3F2;">-0.6%</span></td>
<td id="A1.T12.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F1F9F5;"><span id="A1.T12.1.1.2.1.6.1" class="ltx_text" style="background-color:#F1F9F5;">0.2%</span></td>
<td id="A1.T12.1.1.2.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F6;"><span id="A1.T12.1.1.2.1.7.1" class="ltx_text" style="background-color:#FDF7F6;">-0.4%</span></td>
<td id="A1.T12.1.1.2.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF2F1;"><span id="A1.T12.1.1.2.1.8.1" class="ltx_text" style="background-color:#FCF2F1;">-0.7%</span></td>
<td id="A1.T12.1.1.2.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F8;"><span id="A1.T12.1.1.2.1.9.1" class="ltx_text" style="background-color:#FDF9F8;">-0.3%</span></td>
<td id="A1.T12.1.1.2.1.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF4F3;"><span id="A1.T12.1.1.2.1.10.1" class="ltx_text" style="background-color:#FDF4F3;">-0.6%</span></td>
<td id="A1.T12.1.1.2.1.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F5;"><span id="A1.T12.1.1.2.1.11.1" class="ltx_text" style="background-color:#FDF6F5;">-0.5%</span></td>
<td id="A1.T12.1.1.2.1.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF4F3;"><span id="A1.T12.1.1.2.1.12.1" class="ltx_text" style="background-color:#FCF4F3;">-0.6%</span></td>
<td id="A1.T12.1.1.2.1.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEBE9;"><span id="A1.T12.1.1.2.1.13.1" class="ltx_text" style="background-color:#FBEBE9;">-1.1%</span></td>
<td id="A1.T12.1.1.2.1.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F2;"><span id="A1.T12.1.1.2.1.14.1" class="ltx_text" style="background-color:#FCF3F2;">-0.6%</span></td>
<td id="A1.T12.1.1.2.1.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F8;"><span id="A1.T12.1.1.2.1.15.1" class="ltx_text" style="background-color:#FDF9F8;">-0.3%</span></td>
<td id="A1.T12.1.1.2.1.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF0EF;"><span id="A1.T12.1.1.2.1.16.1" class="ltx_text" style="background-color:#FCF0EF;">-0.8%</span></td>
<td id="A1.T12.1.1.2.1.17" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T12.1.1.2.1.17.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T12.1.1.2.1.18" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FDF6F5;"><span id="A1.T12.1.1.2.1.18.1" class="ltx_text" style="background-color:#FDF6F5;">-0.5%</span></td>
<td id="A1.T12.1.1.2.1.19" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FDF6F5;"><span id="A1.T12.1.1.2.1.19.1" class="ltx_text" style="background-color:#FDF6F5;">-0.5%</span></td>
<td id="A1.T12.1.1.2.1.20" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FDF5F4;"><span id="A1.T12.1.1.2.1.20.1" class="ltx_text" style="background-color:#FDF5F4;">-0.5%</span></td>
<td id="A1.T12.1.1.2.1.21" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FDF7F7;"><span id="A1.T12.1.1.2.1.21.1" class="ltx_text" style="background-color:#FDF7F7;">-0.4%</span></td>
</tr>
<tr id="A1.T12.1.1.3.2" class="ltx_tr">
<td id="A1.T12.1.1.3.2.1" class="ltx_td ltx_align_left"><span id="A1.T12.1.1.3.2.1.1" class="ltx_text">Aya-23-35b</span></td>
<td id="A1.T12.1.1.3.2.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T12.1.1.3.2.3" class="ltx_td ltx_align_right" style="background-color:#F4C5C1;"><span id="A1.T12.1.1.3.2.3.1" class="ltx_text" style="background-color:#F4C5C1;">-3.1%</span></td>
<td id="A1.T12.1.1.3.2.4" class="ltx_td ltx_align_right" style="background-color:#F5CBC7;"><span id="A1.T12.1.1.3.2.4.1" class="ltx_text" style="background-color:#F5CBC7;">-2.8%</span></td>
<td id="A1.T12.1.1.3.2.5" class="ltx_td ltx_align_right" style="background-color:#F7D6D3;"><span id="A1.T12.1.1.3.2.5.1" class="ltx_text" style="background-color:#F7D6D3;">-2.2%</span></td>
<td id="A1.T12.1.1.3.2.6" class="ltx_td ltx_align_right" style="background-color:#F7D8D6;"><span id="A1.T12.1.1.3.2.6.1" class="ltx_text" style="background-color:#F7D8D6;">-2.0%</span></td>
<td id="A1.T12.1.1.3.2.7" class="ltx_td ltx_align_right" style="background-color:#F6D0CD;"><span id="A1.T12.1.1.3.2.7.1" class="ltx_text" style="background-color:#F6D0CD;">-2.5%</span></td>
<td id="A1.T12.1.1.3.2.8" class="ltx_td ltx_align_right" style="background-color:#EFADA7;"><span id="A1.T12.1.1.3.2.8.1" class="ltx_text" style="background-color:#EFADA7;">-4.4%</span></td>
<td id="A1.T12.1.1.3.2.9" class="ltx_td ltx_align_right" style="background-color:#F6D0CD;"><span id="A1.T12.1.1.3.2.9.1" class="ltx_text" style="background-color:#F6D0CD;">-2.5%</span></td>
<td id="A1.T12.1.1.3.2.10" class="ltx_td ltx_align_right" style="background-color:#F5CCC8;"><span id="A1.T12.1.1.3.2.10.1" class="ltx_text" style="background-color:#F5CCC8;">-2.7%</span></td>
<td id="A1.T12.1.1.3.2.11" class="ltx_td ltx_align_right" style="background-color:#F4C9C6;"><span id="A1.T12.1.1.3.2.11.1" class="ltx_text" style="background-color:#F4C9C6;">-2.8%</span></td>
<td id="A1.T12.1.1.3.2.12" class="ltx_td ltx_align_right" style="background-color:#F6D0CD;"><span id="A1.T12.1.1.3.2.12.1" class="ltx_text" style="background-color:#F6D0CD;">-2.5%</span></td>
<td id="A1.T12.1.1.3.2.13" class="ltx_td ltx_align_right" style="background-color:#F3C0BC;"><span id="A1.T12.1.1.3.2.13.1" class="ltx_text" style="background-color:#F3C0BC;">-3.3%</span></td>
<td id="A1.T12.1.1.3.2.14" class="ltx_td ltx_align_right" style="background-color:#F6D4D2;"><span id="A1.T12.1.1.3.2.14.1" class="ltx_text" style="background-color:#F6D4D2;">-2.2%</span></td>
<td id="A1.T12.1.1.3.2.15" class="ltx_td ltx_align_right" style="background-color:#F7D5D2;"><span id="A1.T12.1.1.3.2.15.1" class="ltx_text" style="background-color:#F7D5D2;">-2.2%</span></td>
<td id="A1.T12.1.1.3.2.16" class="ltx_td ltx_align_right" style="background-color:#F5CCC8;"><span id="A1.T12.1.1.3.2.16.1" class="ltx_text" style="background-color:#F5CCC8;">-2.7%</span></td>
<td id="A1.T12.1.1.3.2.17" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F5CECB;"><span id="A1.T12.1.1.3.2.17.1" class="ltx_text" style="background-color:#F5CECB;">-2.6%</span></td>
<td id="A1.T12.1.1.3.2.18" class="ltx_td ltx_align_center" style="background-color:#F5CCC9;"><span id="A1.T12.1.1.3.2.18.1" class="ltx_text" style="background-color:#F5CCC9;">-2.7%</span></td>
<td id="A1.T12.1.1.3.2.19" class="ltx_td ltx_align_center" style="background-color:#F5CCC9;"><span id="A1.T12.1.1.3.2.19.1" class="ltx_text" style="background-color:#F5CCC9;">-2.7%</span></td>
<td id="A1.T12.1.1.3.2.20" class="ltx_td ltx_align_center" style="background-color:#F5CECB;"><span id="A1.T12.1.1.3.2.20.1" class="ltx_text" style="background-color:#F5CECB;">-2.6%</span></td>
<td id="A1.T12.1.1.3.2.21" class="ltx_td ltx_align_center" style="background-color:#F4C8C5;"><span id="A1.T12.1.1.3.2.21.1" class="ltx_text" style="background-color:#F4C8C5;">-2.9%</span></td>
</tr>
<tr id="A1.T12.1.1.4.3" class="ltx_tr">
<td id="A1.T12.1.1.4.3.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T12.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T12.1.1.4.3.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF5F5;"><span id="A1.T12.1.1.4.3.3.1" class="ltx_text" style="background-color:#FDF5F5;">-0.5%</span></td>
<td id="A1.T12.1.1.4.3.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF8F8;"><span id="A1.T12.1.1.4.3.4.1" class="ltx_text" style="background-color:#FDF8F8;">-0.3%</span></td>
<td id="A1.T12.1.1.4.3.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF0EF;"><span id="A1.T12.1.1.4.3.5.1" class="ltx_text" style="background-color:#FCF0EF;">-0.8%</span></td>
<td id="A1.T12.1.1.4.3.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF0EF;"><span id="A1.T12.1.1.4.3.6.1" class="ltx_text" style="background-color:#FCF0EF;">-0.8%</span></td>
<td id="A1.T12.1.1.4.3.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCEFEE;"><span id="A1.T12.1.1.4.3.7.1" class="ltx_text" style="background-color:#FCEFEE;">-0.8%</span></td>
<td id="A1.T12.1.1.4.3.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEDEC;"><span id="A1.T12.1.1.4.3.8.1" class="ltx_text" style="background-color:#FBEDEC;">-0.9%</span></td>
<td id="A1.T12.1.1.4.3.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F3;"><span id="A1.T12.1.1.4.3.9.1" class="ltx_text" style="background-color:#FCF3F3;">-0.6%</span></td>
<td id="A1.T12.1.1.4.3.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBECEA;"><span id="A1.T12.1.1.4.3.10.1" class="ltx_text" style="background-color:#FBECEA;">-1.0%</span></td>
<td id="A1.T12.1.1.4.3.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF2F1;"><span id="A1.T12.1.1.4.3.11.1" class="ltx_text" style="background-color:#FCF2F1;">-0.7%</span></td>
<td id="A1.T12.1.1.4.3.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F7;"><span id="A1.T12.1.1.4.3.12.1" class="ltx_text" style="background-color:#FDF7F7;">-0.4%</span></td>
<td id="A1.T12.1.1.4.3.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE5E3;"><span id="A1.T12.1.1.4.3.13.1" class="ltx_text" style="background-color:#FAE5E3;">-1.4%</span></td>
<td id="A1.T12.1.1.4.3.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEEED;"><span id="A1.T12.1.1.4.3.14.1" class="ltx_text" style="background-color:#FBEEED;">-0.9%</span></td>
<td id="A1.T12.1.1.4.3.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE9E8;"><span id="A1.T12.1.1.4.3.15.1" class="ltx_text" style="background-color:#FAE9E8;">-1.1%</span></td>
<td id="A1.T12.1.1.4.3.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F9DFDD;"><span id="A1.T12.1.1.4.3.16.1" class="ltx_text" style="background-color:#F9DFDD;">-1.7%</span></td>
<td id="A1.T12.1.1.4.3.17" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#F9E2E0;"><span id="A1.T12.1.1.4.3.17.1" class="ltx_text" style="background-color:#F9E2E0;">-1.5%</span></td>
<td id="A1.T12.1.1.4.3.18" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FBEEED;"><span id="A1.T12.1.1.4.3.18.1" class="ltx_text" style="background-color:#FBEEED;">-0.9%</span></td>
<td id="A1.T12.1.1.4.3.19" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FBEEED;"><span id="A1.T12.1.1.4.3.19.1" class="ltx_text" style="background-color:#FBEEED;">-0.9%</span></td>
<td id="A1.T12.1.1.4.3.20" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FCEFEE;"><span id="A1.T12.1.1.4.3.20.1" class="ltx_text" style="background-color:#FCEFEE;">-0.8%</span></td>
<td id="A1.T12.1.1.4.3.21" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FBECEB;"><span id="A1.T12.1.1.4.3.21.1" class="ltx_text" style="background-color:#FBECEB;">-1.0%</span></td>
</tr>
<tr id="A1.T12.1.1.5.4" class="ltx_tr">
<td id="A1.T12.1.1.5.4.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A1.T12.1.1.5.4.1.1" class="ltx_text">Aya-23-8b</span></td>
<td id="A1.T12.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4</td>
<td id="A1.T12.1.1.5.4.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F5CDC9;"><span id="A1.T12.1.1.5.4.3.1" class="ltx_text" style="background-color:#F5CDC9;">-2.7%</span></td>
<td id="A1.T12.1.1.5.4.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C1BD;"><span id="A1.T12.1.1.5.4.4.1" class="ltx_text" style="background-color:#F3C1BD;">-3.3%</span></td>
<td id="A1.T12.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F7D8D5;"><span id="A1.T12.1.1.5.4.5.1" class="ltx_text" style="background-color:#F7D8D5;">-2.1%</span></td>
<td id="A1.T12.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F4C7C4;"><span id="A1.T12.1.1.5.4.6.1" class="ltx_text" style="background-color:#F4C7C4;">-2.9%</span></td>
<td id="A1.T12.1.1.5.4.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F0B5B0;"><span id="A1.T12.1.1.5.4.7.1" class="ltx_text" style="background-color:#F0B5B0;">-3.9%</span></td>
<td id="A1.T12.1.1.5.4.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F2BEBA;"><span id="A1.T12.1.1.5.4.8.1" class="ltx_text" style="background-color:#F2BEBA;">-3.4%</span></td>
<td id="A1.T12.1.1.5.4.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F5CDCA;"><span id="A1.T12.1.1.5.4.9.1" class="ltx_text" style="background-color:#F5CDCA;">-2.6%</span></td>
<td id="A1.T12.1.1.5.4.10" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C3BF;"><span id="A1.T12.1.1.5.4.10.1" class="ltx_text" style="background-color:#F3C3BF;">-3.2%</span></td>
<td id="A1.T12.1.1.5.4.11" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F1BAB5;"><span id="A1.T12.1.1.5.4.11.1" class="ltx_text" style="background-color:#F1BAB5;">-3.7%</span></td>
<td id="A1.T12.1.1.5.4.12" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C0BC;"><span id="A1.T12.1.1.5.4.12.1" class="ltx_text" style="background-color:#F3C0BC;">-3.3%</span></td>
<td id="A1.T12.1.1.5.4.13" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F1B7B2;"><span id="A1.T12.1.1.5.4.13.1" class="ltx_text" style="background-color:#F1B7B2;">-3.8%</span></td>
<td id="A1.T12.1.1.5.4.14" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F2BFBB;"><span id="A1.T12.1.1.5.4.14.1" class="ltx_text" style="background-color:#F2BFBB;">-3.4%</span></td>
<td id="A1.T12.1.1.5.4.15" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F5CBC8;"><span id="A1.T12.1.1.5.4.15.1" class="ltx_text" style="background-color:#F5CBC8;">-2.7%</span></td>
<td id="A1.T12.1.1.5.4.16" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F2BDB9;"><span id="A1.T12.1.1.5.4.16.1" class="ltx_text" style="background-color:#F2BDB9;">-3.5%</span></td>
<td id="A1.T12.1.1.5.4.17" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#F7D5D2;"><span id="A1.T12.1.1.5.4.17.1" class="ltx_text" style="background-color:#F7D5D2;">-2.2%</span></td>
<td id="A1.T12.1.1.5.4.18" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#F3C4C0;"><span id="A1.T12.1.1.5.4.18.1" class="ltx_text" style="background-color:#F3C4C0;">-3.1%</span></td>
<td id="A1.T12.1.1.5.4.19" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#F3C4C0;"><span id="A1.T12.1.1.5.4.19.1" class="ltx_text" style="background-color:#F3C4C0;">-3.2%</span></td>
<td id="A1.T12.1.1.5.4.20" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#F3C3BF;"><span id="A1.T12.1.1.5.4.20.1" class="ltx_text" style="background-color:#F3C3BF;">-3.2%</span></td>
<td id="A1.T12.1.1.5.4.21" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#F4C8C5;"><span id="A1.T12.1.1.5.4.21.1" class="ltx_text" style="background-color:#F4C8C5;">-2.9%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A12: </span><span id="A1.T12.4.1" class="ltx_text ltx_font_bold">Relative performance (<math id="A1.T12.4.1.m1.1" class="ltx_math_unparsed" alttext="\%\Delta" display="inline"><semantics id="A1.T12.4.1.m1.1b"><mrow id="A1.T12.4.1.m1.1c"><mo id="A1.T12.4.1.m1.1.1">%</mo><mi mathvariant="normal" id="A1.T12.4.1.m1.1.2">Δ</mi></mrow><annotation encoding="application/x-tex" id="A1.T12.4.1.m1.1d">\%\Delta</annotation></semantics></math>) vs. FP16 for Aya Models on mMMLU (Okapi).</span> Ltn are Latin-script languages: de, en, es, fr, id, it, nl, pt, ro, vi. <math id="A1.T12.5.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T12.5.m1.1b"><mo id="A1.T12.5.m1.1.1" xref="A1.T12.5.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T12.5.m1.1c"><not id="A1.T12.5.m1.1.1.cmml" xref="A1.T12.5.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T12.5.m1.1d">\neg</annotation></semantics></math> are the rest.</figcaption>
</figure>
<figure id="A1.T13" class="ltx_table">
<div id="A1.T13.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:126.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-152.5pt,44.6pt) scale(0.587051848749181,0.587051848749181) ;">
<table id="A1.T13.2.2" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T13.2.2.2" class="ltx_tr">
<th id="A1.T13.2.2.2.3" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="A1.T13.2.2.2.4" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt"></th>
<th id="A1.T13.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="10"><span id="A1.T13.1.1.1.1.1" class="ltx_text ltx_font_bold">English <math id="A1.T13.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A1.T13.1.1.1.1.1.m1.1a"><mo stretchy="false" id="A1.T13.1.1.1.1.1.m1.1.1" xref="A1.T13.1.1.1.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A1.T13.1.1.1.1.1.m1.1b"><ci id="A1.T13.1.1.1.1.1.m1.1.1.cmml" xref="A1.T13.1.1.1.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T13.1.1.1.1.1.m1.1c">\rightarrow</annotation></semantics></math> L2</span></th>
<th id="A1.T13.2.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="10"><span id="A1.T13.2.2.2.2.1" class="ltx_text ltx_font_bold">L2 <math id="A1.T13.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A1.T13.2.2.2.2.1.m1.1a"><mo stretchy="false" id="A1.T13.2.2.2.2.1.m1.1.1" xref="A1.T13.2.2.2.2.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A1.T13.2.2.2.2.1.m1.1b"><ci id="A1.T13.2.2.2.2.1.m1.1.1.cmml" xref="A1.T13.2.2.2.2.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T13.2.2.2.2.1.m1.1c">\rightarrow</annotation></semantics></math> English</span></th>
</tr>
<tr id="A1.T13.2.2.3.1" class="ltx_tr">
<th id="A1.T13.2.2.3.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt ltx_border_t"></th>
<th id="A1.T13.2.2.3.1.2" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt ltx_border_t"></th>
<th id="A1.T13.2.2.3.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.3.1" class="ltx_text ltx_font_bold">ar</span></th>
<th id="A1.T13.2.2.3.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.4.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T13.2.2.3.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.5.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T13.2.2.3.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.6.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T13.2.2.3.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.7.1" class="ltx_text ltx_font_bold">it</span></th>
<th id="A1.T13.2.2.3.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.8.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T13.2.2.3.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.9.1" class="ltx_text ltx_font_bold">ko</span></th>
<th id="A1.T13.2.2.3.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.10.1" class="ltx_text ltx_font_bold">pt</span></th>
<th id="A1.T13.2.2.3.1.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.11.1" class="ltx_text ltx_font_bold">zh</span></th>
<th id="A1.T13.2.2.3.1.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.12.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="A1.T13.2.2.3.1.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.13.1" class="ltx_text ltx_font_bold">ar</span></th>
<th id="A1.T13.2.2.3.1.14" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.14.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T13.2.2.3.1.15" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.15.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T13.2.2.3.1.16" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.16.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T13.2.2.3.1.17" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.17.1" class="ltx_text ltx_font_bold">it</span></th>
<th id="A1.T13.2.2.3.1.18" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.18.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T13.2.2.3.1.19" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.19.1" class="ltx_text ltx_font_bold">ko</span></th>
<th id="A1.T13.2.2.3.1.20" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.20.1" class="ltx_text ltx_font_bold">pt</span></th>
<th id="A1.T13.2.2.3.1.21" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.21.1" class="ltx_text ltx_font_bold">zh</span></th>
<th id="A1.T13.2.2.3.1.22" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt ltx_border_t"><span id="A1.T13.2.2.3.1.22.1" class="ltx_text ltx_font_bold">Avg</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T13.2.2.4.1" class="ltx_tr">
<td id="A1.T13.2.2.4.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" rowspan="6"><span id="A1.T13.2.2.4.1.1.1" class="ltx_text">103B</span></td>
<td id="A1.T13.2.2.4.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T13.2.2.4.1.3" class="ltx_td ltx_align_right ltx_border_t">27.1</td>
<td id="A1.T13.2.2.4.1.4" class="ltx_td ltx_align_right ltx_border_t">40.0</td>
<td id="A1.T13.2.2.4.1.5" class="ltx_td ltx_align_right ltx_border_t">30.1</td>
<td id="A1.T13.2.2.4.1.6" class="ltx_td ltx_align_right ltx_border_t">50.6</td>
<td id="A1.T13.2.2.4.1.7" class="ltx_td ltx_align_right ltx_border_t">33.1</td>
<td id="A1.T13.2.2.4.1.8" class="ltx_td ltx_align_right ltx_border_t">33.1</td>
<td id="A1.T13.2.2.4.1.9" class="ltx_td ltx_align_right ltx_border_t">29.1</td>
<td id="A1.T13.2.2.4.1.10" class="ltx_td ltx_align_right ltx_border_t">51.0</td>
<td id="A1.T13.2.2.4.1.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">45.1</td>
<td id="A1.T13.2.2.4.1.12" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t">37.7</td>
<td id="A1.T13.2.2.4.1.13" class="ltx_td ltx_align_right ltx_border_t">45.0</td>
<td id="A1.T13.2.2.4.1.14" class="ltx_td ltx_align_right ltx_border_t">46.3</td>
<td id="A1.T13.2.2.4.1.15" class="ltx_td ltx_align_right ltx_border_t">33.4</td>
<td id="A1.T13.2.2.4.1.16" class="ltx_td ltx_align_right ltx_border_t">48.6</td>
<td id="A1.T13.2.2.4.1.17" class="ltx_td ltx_align_right ltx_border_t">36.5</td>
<td id="A1.T13.2.2.4.1.18" class="ltx_td ltx_align_right ltx_border_t">29.5</td>
<td id="A1.T13.2.2.4.1.19" class="ltx_td ltx_align_right ltx_border_t">33.0</td>
<td id="A1.T13.2.2.4.1.20" class="ltx_td ltx_align_right ltx_border_t">52.2</td>
<td id="A1.T13.2.2.4.1.21" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">32.1</td>
<td id="A1.T13.2.2.4.1.22" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">39.6</td>
</tr>
<tr id="A1.T13.2.2.5.2" class="ltx_tr">
<td id="A1.T13.2.2.5.2.1" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T13.2.2.5.2.2" class="ltx_td ltx_align_right">27.2</td>
<td id="A1.T13.2.2.5.2.3" class="ltx_td ltx_align_right">40.0</td>
<td id="A1.T13.2.2.5.2.4" class="ltx_td ltx_align_right">30.0</td>
<td id="A1.T13.2.2.5.2.5" class="ltx_td ltx_align_right">50.7</td>
<td id="A1.T13.2.2.5.2.6" class="ltx_td ltx_align_right">33.1</td>
<td id="A1.T13.2.2.5.2.7" class="ltx_td ltx_align_right">33.2</td>
<td id="A1.T13.2.2.5.2.8" class="ltx_td ltx_align_right">29.1</td>
<td id="A1.T13.2.2.5.2.9" class="ltx_td ltx_align_right">50.9</td>
<td id="A1.T13.2.2.5.2.10" class="ltx_td ltx_align_right ltx_border_r">45.1</td>
<td id="A1.T13.2.2.5.2.11" class="ltx_td ltx_align_right ltx_border_rr">37.7</td>
<td id="A1.T13.2.2.5.2.12" class="ltx_td ltx_align_right">45.2</td>
<td id="A1.T13.2.2.5.2.13" class="ltx_td ltx_align_right">46.3</td>
<td id="A1.T13.2.2.5.2.14" class="ltx_td ltx_align_right">33.4</td>
<td id="A1.T13.2.2.5.2.15" class="ltx_td ltx_align_right">48.5</td>
<td id="A1.T13.2.2.5.2.16" class="ltx_td ltx_align_right">36.5</td>
<td id="A1.T13.2.2.5.2.17" class="ltx_td ltx_align_right">29.5</td>
<td id="A1.T13.2.2.5.2.18" class="ltx_td ltx_align_right">33.0</td>
<td id="A1.T13.2.2.5.2.19" class="ltx_td ltx_align_right">52.1</td>
<td id="A1.T13.2.2.5.2.20" class="ltx_td ltx_align_right ltx_border_r">32.0</td>
<td id="A1.T13.2.2.5.2.21" class="ltx_td ltx_nopad_r ltx_align_right">39.6</td>
</tr>
<tr id="A1.T13.2.2.6.3" class="ltx_tr">
<td id="A1.T13.2.2.6.3.1" class="ltx_td ltx_align_left ltx_border_r">W8A8-sq</td>
<td id="A1.T13.2.2.6.3.2" class="ltx_td ltx_align_right">26.8</td>
<td id="A1.T13.2.2.6.3.3" class="ltx_td ltx_align_right">40.3</td>
<td id="A1.T13.2.2.6.3.4" class="ltx_td ltx_align_right">30.0</td>
<td id="A1.T13.2.2.6.3.5" class="ltx_td ltx_align_right">51.0</td>
<td id="A1.T13.2.2.6.3.6" class="ltx_td ltx_align_right">33.0</td>
<td id="A1.T13.2.2.6.3.7" class="ltx_td ltx_align_right">33.1</td>
<td id="A1.T13.2.2.6.3.8" class="ltx_td ltx_align_right">29.3</td>
<td id="A1.T13.2.2.6.3.9" class="ltx_td ltx_align_right">51.2</td>
<td id="A1.T13.2.2.6.3.10" class="ltx_td ltx_align_right ltx_border_r">45.1</td>
<td id="A1.T13.2.2.6.3.11" class="ltx_td ltx_align_right ltx_border_rr">37.8</td>
<td id="A1.T13.2.2.6.3.12" class="ltx_td ltx_align_right">44.5</td>
<td id="A1.T13.2.2.6.3.13" class="ltx_td ltx_align_right">46.2</td>
<td id="A1.T13.2.2.6.3.14" class="ltx_td ltx_align_right">32.9</td>
<td id="A1.T13.2.2.6.3.15" class="ltx_td ltx_align_right">48.1</td>
<td id="A1.T13.2.2.6.3.16" class="ltx_td ltx_align_right">35.9</td>
<td id="A1.T13.2.2.6.3.17" class="ltx_td ltx_align_right">29.3</td>
<td id="A1.T13.2.2.6.3.18" class="ltx_td ltx_align_right">32.5</td>
<td id="A1.T13.2.2.6.3.19" class="ltx_td ltx_align_right">51.6</td>
<td id="A1.T13.2.2.6.3.20" class="ltx_td ltx_align_right ltx_border_r">31.2</td>
<td id="A1.T13.2.2.6.3.21" class="ltx_td ltx_nopad_r ltx_align_right">39.1</td>
</tr>
<tr id="A1.T13.2.2.7.4" class="ltx_tr">
<td id="A1.T13.2.2.7.4.1" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T13.2.2.7.4.2" class="ltx_td ltx_align_right">26.9</td>
<td id="A1.T13.2.2.7.4.3" class="ltx_td ltx_align_right">39.8</td>
<td id="A1.T13.2.2.7.4.4" class="ltx_td ltx_align_right">30.0</td>
<td id="A1.T13.2.2.7.4.5" class="ltx_td ltx_align_right">50.9</td>
<td id="A1.T13.2.2.7.4.6" class="ltx_td ltx_align_right">33.0</td>
<td id="A1.T13.2.2.7.4.7" class="ltx_td ltx_align_right">33.7</td>
<td id="A1.T13.2.2.7.4.8" class="ltx_td ltx_align_right">29.0</td>
<td id="A1.T13.2.2.7.4.9" class="ltx_td ltx_align_right">51.1</td>
<td id="A1.T13.2.2.7.4.10" class="ltx_td ltx_align_right ltx_border_r">45.1</td>
<td id="A1.T13.2.2.7.4.11" class="ltx_td ltx_align_right ltx_border_rr">37.7</td>
<td id="A1.T13.2.2.7.4.12" class="ltx_td ltx_align_right">44.4</td>
<td id="A1.T13.2.2.7.4.13" class="ltx_td ltx_align_right">45.9</td>
<td id="A1.T13.2.2.7.4.14" class="ltx_td ltx_align_right">33.1</td>
<td id="A1.T13.2.2.7.4.15" class="ltx_td ltx_align_right">47.9</td>
<td id="A1.T13.2.2.7.4.16" class="ltx_td ltx_align_right">36.2</td>
<td id="A1.T13.2.2.7.4.17" class="ltx_td ltx_align_right">29.2</td>
<td id="A1.T13.2.2.7.4.18" class="ltx_td ltx_align_right">32.5</td>
<td id="A1.T13.2.2.7.4.19" class="ltx_td ltx_align_right">51.8</td>
<td id="A1.T13.2.2.7.4.20" class="ltx_td ltx_align_right ltx_border_r">31.4</td>
<td id="A1.T13.2.2.7.4.21" class="ltx_td ltx_nopad_r ltx_align_right">39.1</td>
</tr>
<tr id="A1.T13.2.2.8.5" class="ltx_tr">
<td id="A1.T13.2.2.8.5.1" class="ltx_td ltx_align_left ltx_border_r">W4-g</td>
<td id="A1.T13.2.2.8.5.2" class="ltx_td ltx_align_right">27.3</td>
<td id="A1.T13.2.2.8.5.3" class="ltx_td ltx_align_right">40.4</td>
<td id="A1.T13.2.2.8.5.4" class="ltx_td ltx_align_right">30.1</td>
<td id="A1.T13.2.2.8.5.5" class="ltx_td ltx_align_right">51.0</td>
<td id="A1.T13.2.2.8.5.6" class="ltx_td ltx_align_right">33.0</td>
<td id="A1.T13.2.2.8.5.7" class="ltx_td ltx_align_right">33.9</td>
<td id="A1.T13.2.2.8.5.8" class="ltx_td ltx_align_right">29.3</td>
<td id="A1.T13.2.2.8.5.9" class="ltx_td ltx_align_right">50.9</td>
<td id="A1.T13.2.2.8.5.10" class="ltx_td ltx_align_right ltx_border_r">44.7</td>
<td id="A1.T13.2.2.8.5.11" class="ltx_td ltx_align_right ltx_border_rr">37.8</td>
<td id="A1.T13.2.2.8.5.12" class="ltx_td ltx_align_right">44.9</td>
<td id="A1.T13.2.2.8.5.13" class="ltx_td ltx_align_right">46.4</td>
<td id="A1.T13.2.2.8.5.14" class="ltx_td ltx_align_right">33.2</td>
<td id="A1.T13.2.2.8.5.15" class="ltx_td ltx_align_right">48.4</td>
<td id="A1.T13.2.2.8.5.16" class="ltx_td ltx_align_right">36.3</td>
<td id="A1.T13.2.2.8.5.17" class="ltx_td ltx_align_right">29.3</td>
<td id="A1.T13.2.2.8.5.18" class="ltx_td ltx_align_right">32.7</td>
<td id="A1.T13.2.2.8.5.19" class="ltx_td ltx_align_right">52.0</td>
<td id="A1.T13.2.2.8.5.20" class="ltx_td ltx_align_right ltx_border_r">31.6</td>
<td id="A1.T13.2.2.8.5.21" class="ltx_td ltx_nopad_r ltx_align_right">39.4</td>
</tr>
<tr id="A1.T13.2.2.9.6" class="ltx_tr">
<td id="A1.T13.2.2.9.6.1" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T13.2.2.9.6.2" class="ltx_td ltx_align_right">26.9</td>
<td id="A1.T13.2.2.9.6.3" class="ltx_td ltx_align_right">39.1</td>
<td id="A1.T13.2.2.9.6.4" class="ltx_td ltx_align_right">29.9</td>
<td id="A1.T13.2.2.9.6.5" class="ltx_td ltx_align_right">50.0</td>
<td id="A1.T13.2.2.9.6.6" class="ltx_td ltx_align_right">32.8</td>
<td id="A1.T13.2.2.9.6.7" class="ltx_td ltx_align_right">32.8</td>
<td id="A1.T13.2.2.9.6.8" class="ltx_td ltx_align_right">27.9</td>
<td id="A1.T13.2.2.9.6.9" class="ltx_td ltx_align_right">50.3</td>
<td id="A1.T13.2.2.9.6.10" class="ltx_td ltx_align_right ltx_border_r">44.0</td>
<td id="A1.T13.2.2.9.6.11" class="ltx_td ltx_align_right ltx_border_rr">37.1</td>
<td id="A1.T13.2.2.9.6.12" class="ltx_td ltx_align_right">44.2</td>
<td id="A1.T13.2.2.9.6.13" class="ltx_td ltx_align_right">45.8</td>
<td id="A1.T13.2.2.9.6.14" class="ltx_td ltx_align_right">33.1</td>
<td id="A1.T13.2.2.9.6.15" class="ltx_td ltx_align_right">47.9</td>
<td id="A1.T13.2.2.9.6.16" class="ltx_td ltx_align_right">36.0</td>
<td id="A1.T13.2.2.9.6.17" class="ltx_td ltx_align_right">29.0</td>
<td id="A1.T13.2.2.9.6.18" class="ltx_td ltx_align_right">32.3</td>
<td id="A1.T13.2.2.9.6.19" class="ltx_td ltx_align_right">51.8</td>
<td id="A1.T13.2.2.9.6.20" class="ltx_td ltx_align_right ltx_border_r">30.9</td>
<td id="A1.T13.2.2.9.6.21" class="ltx_td ltx_nopad_r ltx_align_right">39.0</td>
</tr>
<tr id="A1.T13.2.2.10.7" class="ltx_tr">
<td id="A1.T13.2.2.10.7.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r ltx_border_t" rowspan="4"><span id="A1.T13.2.2.10.7.1.1" class="ltx_text">35B</span></td>
<td id="A1.T13.2.2.10.7.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T13.2.2.10.7.3" class="ltx_td ltx_align_right ltx_border_t">20.1</td>
<td id="A1.T13.2.2.10.7.4" class="ltx_td ltx_align_right ltx_border_t">33.5</td>
<td id="A1.T13.2.2.10.7.5" class="ltx_td ltx_align_right ltx_border_t">27.8</td>
<td id="A1.T13.2.2.10.7.6" class="ltx_td ltx_align_right ltx_border_t">44.5</td>
<td id="A1.T13.2.2.10.7.7" class="ltx_td ltx_align_right ltx_border_t">29.7</td>
<td id="A1.T13.2.2.10.7.8" class="ltx_td ltx_align_right ltx_border_t">27.0</td>
<td id="A1.T13.2.2.10.7.9" class="ltx_td ltx_align_right ltx_border_t">22.7</td>
<td id="A1.T13.2.2.10.7.10" class="ltx_td ltx_align_right ltx_border_t">45.5</td>
<td id="A1.T13.2.2.10.7.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">40.4</td>
<td id="A1.T13.2.2.10.7.12" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t">32.4</td>
<td id="A1.T13.2.2.10.7.13" class="ltx_td ltx_align_right ltx_border_t">38.4</td>
<td id="A1.T13.2.2.10.7.14" class="ltx_td ltx_align_right ltx_border_t">41.2</td>
<td id="A1.T13.2.2.10.7.15" class="ltx_td ltx_align_right ltx_border_t">31.8</td>
<td id="A1.T13.2.2.10.7.16" class="ltx_td ltx_align_right ltx_border_t">43.1</td>
<td id="A1.T13.2.2.10.7.17" class="ltx_td ltx_align_right ltx_border_t">34.0</td>
<td id="A1.T13.2.2.10.7.18" class="ltx_td ltx_align_right ltx_border_t">26.2</td>
<td id="A1.T13.2.2.10.7.19" class="ltx_td ltx_align_right ltx_border_t">28.4</td>
<td id="A1.T13.2.2.10.7.20" class="ltx_td ltx_align_right ltx_border_t">48.1</td>
<td id="A1.T13.2.2.10.7.21" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">28.4</td>
<td id="A1.T13.2.2.10.7.22" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">35.5</td>
</tr>
<tr id="A1.T13.2.2.11.8" class="ltx_tr">
<td id="A1.T13.2.2.11.8.1" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T13.2.2.11.8.2" class="ltx_td ltx_align_right">20.0</td>
<td id="A1.T13.2.2.11.8.3" class="ltx_td ltx_align_right">33.4</td>
<td id="A1.T13.2.2.11.8.4" class="ltx_td ltx_align_right">27.8</td>
<td id="A1.T13.2.2.11.8.5" class="ltx_td ltx_align_right">44.5</td>
<td id="A1.T13.2.2.11.8.6" class="ltx_td ltx_align_right">29.7</td>
<td id="A1.T13.2.2.11.8.7" class="ltx_td ltx_align_right">26.9</td>
<td id="A1.T13.2.2.11.8.8" class="ltx_td ltx_align_right">22.9</td>
<td id="A1.T13.2.2.11.8.9" class="ltx_td ltx_align_right">45.3</td>
<td id="A1.T13.2.2.11.8.10" class="ltx_td ltx_align_right ltx_border_r">40.3</td>
<td id="A1.T13.2.2.11.8.11" class="ltx_td ltx_align_right ltx_border_rr">32.3</td>
<td id="A1.T13.2.2.11.8.12" class="ltx_td ltx_align_right">38.3</td>
<td id="A1.T13.2.2.11.8.13" class="ltx_td ltx_align_right">41.1</td>
<td id="A1.T13.2.2.11.8.14" class="ltx_td ltx_align_right">31.7</td>
<td id="A1.T13.2.2.11.8.15" class="ltx_td ltx_align_right">43.0</td>
<td id="A1.T13.2.2.11.8.16" class="ltx_td ltx_align_right">34.0</td>
<td id="A1.T13.2.2.11.8.17" class="ltx_td ltx_align_right">26.4</td>
<td id="A1.T13.2.2.11.8.18" class="ltx_td ltx_align_right">28.2</td>
<td id="A1.T13.2.2.11.8.19" class="ltx_td ltx_align_right">48.0</td>
<td id="A1.T13.2.2.11.8.20" class="ltx_td ltx_align_right ltx_border_r">28.2</td>
<td id="A1.T13.2.2.11.8.21" class="ltx_td ltx_nopad_r ltx_align_right">35.4</td>
</tr>
<tr id="A1.T13.2.2.12.9" class="ltx_tr">
<td id="A1.T13.2.2.12.9.1" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T13.2.2.12.9.2" class="ltx_td ltx_align_right">21.2</td>
<td id="A1.T13.2.2.12.9.3" class="ltx_td ltx_align_right">34.1</td>
<td id="A1.T13.2.2.12.9.4" class="ltx_td ltx_align_right">27.8</td>
<td id="A1.T13.2.2.12.9.5" class="ltx_td ltx_align_right">45.1</td>
<td id="A1.T13.2.2.12.9.6" class="ltx_td ltx_align_right">30.0</td>
<td id="A1.T13.2.2.12.9.7" class="ltx_td ltx_align_right">27.6</td>
<td id="A1.T13.2.2.12.9.8" class="ltx_td ltx_align_right">23.1</td>
<td id="A1.T13.2.2.12.9.9" class="ltx_td ltx_align_right">46.1</td>
<td id="A1.T13.2.2.12.9.10" class="ltx_td ltx_align_right ltx_border_r">40.8</td>
<td id="A1.T13.2.2.12.9.11" class="ltx_td ltx_align_right ltx_border_rr">32.9</td>
<td id="A1.T13.2.2.12.9.12" class="ltx_td ltx_align_right">38.5</td>
<td id="A1.T13.2.2.12.9.13" class="ltx_td ltx_align_right">42.2</td>
<td id="A1.T13.2.2.12.9.14" class="ltx_td ltx_align_right">31.7</td>
<td id="A1.T13.2.2.12.9.15" class="ltx_td ltx_align_right">43.5</td>
<td id="A1.T13.2.2.12.9.16" class="ltx_td ltx_align_right">34.2</td>
<td id="A1.T13.2.2.12.9.17" class="ltx_td ltx_align_right">26.5</td>
<td id="A1.T13.2.2.12.9.18" class="ltx_td ltx_align_right">28.6</td>
<td id="A1.T13.2.2.12.9.19" class="ltx_td ltx_align_right">48.6</td>
<td id="A1.T13.2.2.12.9.20" class="ltx_td ltx_align_right ltx_border_r">28.7</td>
<td id="A1.T13.2.2.12.9.21" class="ltx_td ltx_nopad_r ltx_align_right">35.8</td>
</tr>
<tr id="A1.T13.2.2.13.10" class="ltx_tr">
<td id="A1.T13.2.2.13.10.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4-g</td>
<td id="A1.T13.2.2.13.10.2" class="ltx_td ltx_align_right ltx_border_bb">18.8</td>
<td id="A1.T13.2.2.13.10.3" class="ltx_td ltx_align_right ltx_border_bb">32.9</td>
<td id="A1.T13.2.2.13.10.4" class="ltx_td ltx_align_right ltx_border_bb">27.7</td>
<td id="A1.T13.2.2.13.10.5" class="ltx_td ltx_align_right ltx_border_bb">43.9</td>
<td id="A1.T13.2.2.13.10.6" class="ltx_td ltx_align_right ltx_border_bb">29.6</td>
<td id="A1.T13.2.2.13.10.7" class="ltx_td ltx_align_right ltx_border_bb">26.0</td>
<td id="A1.T13.2.2.13.10.8" class="ltx_td ltx_align_right ltx_border_bb">22.1</td>
<td id="A1.T13.2.2.13.10.9" class="ltx_td ltx_align_right ltx_border_bb">45.1</td>
<td id="A1.T13.2.2.13.10.10" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r">39.7</td>
<td id="A1.T13.2.2.13.10.11" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr">31.7</td>
<td id="A1.T13.2.2.13.10.12" class="ltx_td ltx_align_right ltx_border_bb">38.3</td>
<td id="A1.T13.2.2.13.10.13" class="ltx_td ltx_align_right ltx_border_bb">41.4</td>
<td id="A1.T13.2.2.13.10.14" class="ltx_td ltx_align_right ltx_border_bb">31.0</td>
<td id="A1.T13.2.2.13.10.15" class="ltx_td ltx_align_right ltx_border_bb">43.1</td>
<td id="A1.T13.2.2.13.10.16" class="ltx_td ltx_align_right ltx_border_bb">34.0</td>
<td id="A1.T13.2.2.13.10.17" class="ltx_td ltx_align_right ltx_border_bb">25.5</td>
<td id="A1.T13.2.2.13.10.18" class="ltx_td ltx_align_right ltx_border_bb">28.1</td>
<td id="A1.T13.2.2.13.10.19" class="ltx_td ltx_align_right ltx_border_bb">48.0</td>
<td id="A1.T13.2.2.13.10.20" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r">28.0</td>
<td id="A1.T13.2.2.13.10.21" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb">35.3</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A13: </span><span id="A1.T13.4.1" class="ltx_text ltx_font_bold">Full results on FLORES for Command Models.</span> (SacreBLEU)</figcaption>
</figure>
<figure id="A1.T14" class="ltx_table">
<div id="A1.T14.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:74.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-305.4pt,52.6pt) scale(0.415178525968469,0.415178525968469) ;">
<table id="A1.T14.4.4" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T14.2.2.2" class="ltx_tr">
<th id="A1.T14.2.2.2.3" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A1.T14.2.2.2.4" class="ltx_td ltx_th ltx_th_row ltx_border_r ltx_border_tt"></th>
<td id="A1.T14.1.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="12"><span id="A1.T14.1.1.1.1.1" class="ltx_text ltx_font_bold">English <math id="A1.T14.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A1.T14.1.1.1.1.1.m1.1a"><mo stretchy="false" id="A1.T14.1.1.1.1.1.m1.1.1" xref="A1.T14.1.1.1.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A1.T14.1.1.1.1.1.m1.1b"><ci id="A1.T14.1.1.1.1.1.m1.1.1.cmml" xref="A1.T14.1.1.1.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T14.1.1.1.1.1.m1.1c">\rightarrow</annotation></semantics></math> L2</span></td>
<td id="A1.T14.2.2.2.2" class="ltx_td ltx_align_center ltx_border_tt" colspan="12"><span id="A1.T14.2.2.2.2.1" class="ltx_text ltx_font_bold">L2 <math id="A1.T14.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A1.T14.2.2.2.2.1.m1.1a"><mo stretchy="false" id="A1.T14.2.2.2.2.1.m1.1.1" xref="A1.T14.2.2.2.2.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A1.T14.2.2.2.2.1.m1.1b"><ci id="A1.T14.2.2.2.2.1.m1.1.1.cmml" xref="A1.T14.2.2.2.2.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T14.2.2.2.2.1.m1.1c">\rightarrow</annotation></semantics></math> English</span></td>
</tr>
<tr id="A1.T14.4.4.4" class="ltx_tr">
<th id="A1.T14.4.4.4.3" class="ltx_td ltx_th ltx_th_row ltx_border_tt ltx_border_t"></th>
<th id="A1.T14.4.4.4.4" class="ltx_td ltx_th ltx_th_row ltx_border_r ltx_border_tt ltx_border_t"></th>
<td id="A1.T14.4.4.4.5" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.5.1" class="ltx_text ltx_font_bold">ar</span></td>
<td id="A1.T14.4.4.4.6" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.6.1" class="ltx_text ltx_font_bold">de</span></td>
<td id="A1.T14.4.4.4.7" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.7.1" class="ltx_text ltx_font_bold">es</span></td>
<td id="A1.T14.4.4.4.8" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.8.1" class="ltx_text ltx_font_bold">fr</span></td>
<td id="A1.T14.4.4.4.9" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.9.1" class="ltx_text ltx_font_bold">it</span></td>
<td id="A1.T14.4.4.4.10" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.10.1" class="ltx_text ltx_font_bold">ja</span></td>
<td id="A1.T14.4.4.4.11" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.11.1" class="ltx_text ltx_font_bold">ko</span></td>
<td id="A1.T14.4.4.4.12" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.12.1" class="ltx_text ltx_font_bold">pt</span></td>
<td id="A1.T14.4.4.4.13" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.13.1" class="ltx_text ltx_font_bold">zh</span></td>
<td id="A1.T14.4.4.4.14" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.14.1" class="ltx_text ltx_font_bold">Avg</span></td>
<td id="A1.T14.4.4.4.15" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.15.1" class="ltx_text ltx_font_bold">Ltn/IE</span></td>
<td id="A1.T14.3.3.3.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt ltx_border_t"><math id="A1.T14.3.3.3.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T14.3.3.3.1.m1.1a"><mo id="A1.T14.3.3.3.1.m1.1.1" xref="A1.T14.3.3.3.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T14.3.3.3.1.m1.1b"><not id="A1.T14.3.3.3.1.m1.1.1.cmml" xref="A1.T14.3.3.3.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T14.3.3.3.1.m1.1c">\neg</annotation></semantics></math></td>
<td id="A1.T14.4.4.4.16" class="ltx_td ltx_align_center ltx_border_l ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.16.1" class="ltx_text ltx_font_bold">ar</span></td>
<td id="A1.T14.4.4.4.17" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.17.1" class="ltx_text ltx_font_bold">de</span></td>
<td id="A1.T14.4.4.4.18" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.18.1" class="ltx_text ltx_font_bold">es</span></td>
<td id="A1.T14.4.4.4.19" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.19.1" class="ltx_text ltx_font_bold">fr</span></td>
<td id="A1.T14.4.4.4.20" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.20.1" class="ltx_text ltx_font_bold">it</span></td>
<td id="A1.T14.4.4.4.21" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.21.1" class="ltx_text ltx_font_bold">ja</span></td>
<td id="A1.T14.4.4.4.22" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.22.1" class="ltx_text ltx_font_bold">ko</span></td>
<td id="A1.T14.4.4.4.23" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.23.1" class="ltx_text ltx_font_bold">pt</span></td>
<td id="A1.T14.4.4.4.24" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.24.1" class="ltx_text ltx_font_bold">zh</span></td>
<td id="A1.T14.4.4.4.25" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.25.1" class="ltx_text ltx_font_bold">Avg</span></td>
<td id="A1.T14.4.4.4.26" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><span id="A1.T14.4.4.4.26.1" class="ltx_text ltx_font_bold">Ltn/IE</span></td>
<td id="A1.T14.4.4.4.2" class="ltx_td ltx_align_center ltx_border_tt ltx_border_t"><math id="A1.T14.4.4.4.2.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T14.4.4.4.2.m1.1a"><mo id="A1.T14.4.4.4.2.m1.1.1" xref="A1.T14.4.4.4.2.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T14.4.4.4.2.m1.1b"><not id="A1.T14.4.4.4.2.m1.1.1.cmml" xref="A1.T14.4.4.4.2.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T14.4.4.4.2.m1.1c">\neg</annotation></semantics></math></td>
</tr>
<tr id="A1.T14.4.4.5.1" class="ltx_tr">
<th id="A1.T14.4.4.5.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="A1.T14.4.4.5.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">W8</th>
<td id="A1.T14.4.4.5.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFFFE;"><span id="A1.T14.4.4.5.1.3.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="A1.T14.4.4.5.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDFEFE;"><span id="A1.T14.4.4.5.1.4.1" class="ltx_text" style="background-color:#FDFEFE;">0.1%</span></td>
<td id="A1.T14.4.4.5.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF4F3;"><span id="A1.T14.4.4.5.1.5.1" class="ltx_text" style="background-color:#FDF4F3;">-0.4%</span></td>
<td id="A1.T14.4.4.5.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBFEFD;"><span id="A1.T14.4.4.5.1.6.1" class="ltx_text" style="background-color:#FBFEFD;">0.2%</span></td>
<td id="A1.T14.4.4.5.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCFEFD;"><span id="A1.T14.4.4.5.1.7.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T14.4.4.5.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8FCFA;"><span id="A1.T14.4.4.5.1.8.1" class="ltx_text" style="background-color:#F8FCFA;">0.3%</span></td>
<td id="A1.T14.4.4.5.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F9;"><span id="A1.T14.4.4.5.1.9.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="A1.T14.4.4.5.1.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F6;"><span id="A1.T14.4.4.5.1.10.1" class="ltx_text" style="background-color:#FDF6F6;">-0.3%</span></td>
<td id="A1.T14.4.4.5.1.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FCFEFD;"><span id="A1.T14.4.4.5.1.11.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T14.4.4.5.1.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFEFE;"><span id="A1.T14.4.4.5.1.12.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="A1.T14.4.4.5.1.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="A1.T14.4.4.5.1.13.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T14.4.4.5.1.14" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FEFFFE;"><span id="A1.T14.4.4.5.1.14.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="A1.T14.4.4.5.1.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F5FBF8;"><span id="A1.T14.4.4.5.1.15.1" class="ltx_text" style="background-color:#F5FBF8;">0.4%</span></td>
<td id="A1.T14.4.4.5.1.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T14.4.4.5.1.16.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T14.4.4.5.1.17" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="A1.T14.4.4.5.1.17.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="A1.T14.4.4.5.1.18" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="A1.T14.4.4.5.1.18.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="A1.T14.4.4.5.1.19" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="A1.T14.4.4.5.1.19.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T14.4.4.5.1.20" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="A1.T14.4.4.5.1.20.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T14.4.4.5.1.21" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T14.4.4.5.1.21.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T14.4.4.5.1.22" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T14.4.4.5.1.22.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T14.4.4.5.1.23" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FEFBFB;"><span id="A1.T14.4.4.5.1.23.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="A1.T14.4.4.5.1.24" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFEFE;"><span id="A1.T14.4.4.5.1.24.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="A1.T14.4.4.5.1.25" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T14.4.4.5.1.25.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T14.4.4.5.1.26" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFFFF;"><span id="A1.T14.4.4.5.1.26.1" class="ltx_text" style="background-color:#FEFFFF;">0.1%</span></td>
</tr>
<tr id="A1.T14.4.4.6.2" class="ltx_tr">
<th id="A1.T14.4.4.6.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T14.4.4.6.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="A1.T14.4.4.6.2.3" class="ltx_td ltx_align_right" style="background-color:#F9E2E0;"><span id="A1.T14.4.4.6.2.3.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="A1.T14.4.4.6.2.4" class="ltx_td ltx_align_right" style="background-color:#ECF8F2;"><span id="A1.T14.4.4.6.2.4.1" class="ltx_text" style="background-color:#ECF8F2;">0.8%</span></td>
<td id="A1.T14.4.4.6.2.5" class="ltx_td ltx_align_right" style="background-color:#FDF4F3;"><span id="A1.T14.4.4.6.2.5.1" class="ltx_text" style="background-color:#FDF4F3;">-0.4%</span></td>
<td id="A1.T14.4.4.6.2.6" class="ltx_td ltx_align_right" style="background-color:#EEF8F3;"><span id="A1.T14.4.4.6.2.6.1" class="ltx_text" style="background-color:#EEF8F3;">0.7%</span></td>
<td id="A1.T14.4.4.6.2.7" class="ltx_td ltx_align_right" style="background-color:#FEFAF9;"><span id="A1.T14.4.4.6.2.7.1" class="ltx_text" style="background-color:#FEFAF9;">-0.2%</span></td>
<td id="A1.T14.4.4.6.2.8" class="ltx_td ltx_align_right" style="background-color:#FBFEFC;"><span id="A1.T14.4.4.6.2.8.1" class="ltx_text" style="background-color:#FBFEFC;">0.2%</span></td>
<td id="A1.T14.4.4.6.2.9" class="ltx_td ltx_align_right" style="background-color:#EEF8F3;"><span id="A1.T14.4.4.6.2.9.1" class="ltx_text" style="background-color:#EEF8F3;">0.7%</span></td>
<td id="A1.T14.4.4.6.2.10" class="ltx_td ltx_align_right" style="background-color:#F7FCFA;"><span id="A1.T14.4.4.6.2.10.1" class="ltx_text" style="background-color:#F7FCFA;">0.3%</span></td>
<td id="A1.T14.4.4.6.2.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FDFFFE;"><span id="A1.T14.4.4.6.2.11.1" class="ltx_text" style="background-color:#FDFFFE;">0.1%</span></td>
<td id="A1.T14.4.4.6.2.12" class="ltx_td ltx_align_right" style="background-color:#FDFEFD;"><span id="A1.T14.4.4.6.2.12.1" class="ltx_text" style="background-color:#FDFEFD;">0.1%</span></td>
<td id="A1.T14.4.4.6.2.13" class="ltx_td ltx_align_right" style="background-color:#FAFDFB;"><span id="A1.T14.4.4.6.2.13.1" class="ltx_text" style="background-color:#FAFDFB;">0.2%</span></td>
<td id="A1.T14.4.4.6.2.14" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#FEFEFD;"><span id="A1.T14.4.4.6.2.14.1" class="ltx_text" style="background-color:#FEFEFD;">0.0%</span></td>
<td id="A1.T14.4.4.6.2.15" class="ltx_td ltx_align_right" style="background-color:#F9DFDD;"><span id="A1.T14.4.4.6.2.15.1" class="ltx_text" style="background-color:#F9DFDD;">-1.2%</span></td>
<td id="A1.T14.4.4.6.2.16" class="ltx_td ltx_align_right" style="background-color:#FEFBFB;"><span id="A1.T14.4.4.6.2.16.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="A1.T14.4.4.6.2.17" class="ltx_td ltx_align_right" style="background-color:#F7D9D6;"><span id="A1.T14.4.4.6.2.17.1" class="ltx_text" style="background-color:#F7D9D6;">-1.4%</span></td>
<td id="A1.T14.4.4.6.2.18" class="ltx_td ltx_align_right" style="background-color:#FAE5E3;"><span id="A1.T14.4.4.6.2.18.1" class="ltx_text" style="background-color:#FAE5E3;">-1.0%</span></td>
<td id="A1.T14.4.4.6.2.19" class="ltx_td ltx_align_right" style="background-color:#F5CFCC;"><span id="A1.T14.4.4.6.2.19.1" class="ltx_text" style="background-color:#F5CFCC;">-1.8%</span></td>
<td id="A1.T14.4.4.6.2.20" class="ltx_td ltx_align_right" style="background-color:#FBEBEA;"><span id="A1.T14.4.4.6.2.20.1" class="ltx_text" style="background-color:#FBEBEA;">-0.7%</span></td>
<td id="A1.T14.4.4.6.2.21" class="ltx_td ltx_align_right" style="background-color:#F6D4D1;"><span id="A1.T14.4.4.6.2.21.1" class="ltx_text" style="background-color:#F6D4D1;">-1.6%</span></td>
<td id="A1.T14.4.4.6.2.22" class="ltx_td ltx_align_right" style="background-color:#F9E0DE;"><span id="A1.T14.4.4.6.2.22.1" class="ltx_text" style="background-color:#F9E0DE;">-1.1%</span></td>
<td id="A1.T14.4.4.6.2.23" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F0B3AE;"><span id="A1.T14.4.4.6.2.23.1" class="ltx_text" style="background-color:#F0B3AE;">-2.9%</span></td>
<td id="A1.T14.4.4.6.2.24" class="ltx_td ltx_align_right" style="background-color:#F8DCD9;"><span id="A1.T14.4.4.6.2.24.1" class="ltx_text" style="background-color:#F8DCD9;">-1.3%</span></td>
<td id="A1.T14.4.4.6.2.25" class="ltx_td ltx_align_right" style="background-color:#F9E2E0;"><span id="A1.T14.4.4.6.2.25.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="A1.T14.4.4.6.2.26" class="ltx_td ltx_align_right" style="background-color:#F6D4D1;"><span id="A1.T14.4.4.6.2.26.1" class="ltx_text" style="background-color:#F6D4D1;">-1.6%</span></td>
</tr>
<tr id="A1.T14.4.4.7.3" class="ltx_tr">
<th id="A1.T14.4.4.7.3.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T14.4.4.7.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8</th>
<td id="A1.T14.4.4.7.3.3" class="ltx_td ltx_align_right" style="background-color:#FAE5E4;"><span id="A1.T14.4.4.7.3.3.1" class="ltx_text" style="background-color:#FAE5E4;">-1.0%</span></td>
<td id="A1.T14.4.4.7.3.4" class="ltx_td ltx_align_right" style="background-color:#FCF4F3;"><span id="A1.T14.4.4.7.3.4.1" class="ltx_text" style="background-color:#FCF4F3;">-0.4%</span></td>
<td id="A1.T14.4.4.7.3.5" class="ltx_td ltx_align_right" style="background-color:#FCF2F1;"><span id="A1.T14.4.4.7.3.5.1" class="ltx_text" style="background-color:#FCF2F1;">-0.5%</span></td>
<td id="A1.T14.4.4.7.3.6" class="ltx_td ltx_align_right" style="background-color:#F2FAF6;"><span id="A1.T14.4.4.7.3.6.1" class="ltx_text" style="background-color:#F2FAF6;">0.5%</span></td>
<td id="A1.T14.4.4.7.3.7" class="ltx_td ltx_align_right" style="background-color:#FDF5F4;"><span id="A1.T14.4.4.7.3.7.1" class="ltx_text" style="background-color:#FDF5F4;">-0.4%</span></td>
<td id="A1.T14.4.4.7.3.8" class="ltx_td ltx_align_right" style="background-color:#D1EDDF;"><span id="A1.T14.4.4.7.3.8.1" class="ltx_text" style="background-color:#D1EDDF;">1.8%</span></td>
<td id="A1.T14.4.4.7.3.9" class="ltx_td ltx_align_right" style="background-color:#FCF4F3;"><span id="A1.T14.4.4.7.3.9.1" class="ltx_text" style="background-color:#FCF4F3;">-0.4%</span></td>
<td id="A1.T14.4.4.7.3.10" class="ltx_td ltx_align_right" style="background-color:#FDFFFE;"><span id="A1.T14.4.4.7.3.10.1" class="ltx_text" style="background-color:#FDFFFE;">0.1%</span></td>
<td id="A1.T14.4.4.7.3.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FEFFFF;"><span id="A1.T14.4.4.7.3.11.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
<td id="A1.T14.4.4.7.3.12" class="ltx_td ltx_align_right" style="background-color:#FEFEFE;"><span id="A1.T14.4.4.7.3.12.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="A1.T14.4.4.7.3.13" class="ltx_td ltx_align_right" style="background-color:#FEFBFB;"><span id="A1.T14.4.4.7.3.13.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="A1.T14.4.4.7.3.14" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#FCFEFD;"><span id="A1.T14.4.4.7.3.14.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T14.4.4.7.3.15" class="ltx_td ltx_align_right" style="background-color:#F8DCD9;"><span id="A1.T14.4.4.7.3.15.1" class="ltx_text" style="background-color:#F8DCD9;">-1.3%</span></td>
<td id="A1.T14.4.4.7.3.16" class="ltx_td ltx_align_right" style="background-color:#FAE9E8;"><span id="A1.T14.4.4.7.3.16.1" class="ltx_text" style="background-color:#FAE9E8;">-0.8%</span></td>
<td id="A1.T14.4.4.7.3.17" class="ltx_td ltx_align_right" style="background-color:#F9E2E0;"><span id="A1.T14.4.4.7.3.17.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="A1.T14.4.4.7.3.18" class="ltx_td ltx_align_right" style="background-color:#F7DAD7;"><span id="A1.T14.4.4.7.3.18.1" class="ltx_text" style="background-color:#F7DAD7;">-1.4%</span></td>
<td id="A1.T14.4.4.7.3.19" class="ltx_td ltx_align_right" style="background-color:#F9E3E1;"><span id="A1.T14.4.4.7.3.19.1" class="ltx_text" style="background-color:#F9E3E1;">-1.0%</span></td>
<td id="A1.T14.4.4.7.3.20" class="ltx_td ltx_align_right" style="background-color:#F8DFDC;"><span id="A1.T14.4.4.7.3.20.1" class="ltx_text" style="background-color:#F8DFDC;">-1.2%</span></td>
<td id="A1.T14.4.4.7.3.21" class="ltx_td ltx_align_right" style="background-color:#F6D2CF;"><span id="A1.T14.4.4.7.3.21.1" class="ltx_text" style="background-color:#F6D2CF;">-1.7%</span></td>
<td id="A1.T14.4.4.7.3.22" class="ltx_td ltx_align_right" style="background-color:#FBEAE9;"><span id="A1.T14.4.4.7.3.22.1" class="ltx_text" style="background-color:#FBEAE9;">-0.8%</span></td>
<td id="A1.T14.4.4.7.3.23" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F4C9C5;"><span id="A1.T14.4.4.7.3.23.1" class="ltx_text" style="background-color:#F4C9C5;">-2.1%</span></td>
<td id="A1.T14.4.4.7.3.24" class="ltx_td ltx_align_right" style="background-color:#F8DDDB;"><span id="A1.T14.4.4.7.3.24.1" class="ltx_text" style="background-color:#F8DDDB;">-1.3%</span></td>
<td id="A1.T14.4.4.7.3.25" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="A1.T14.4.4.7.3.25.1" class="ltx_text" style="background-color:#F9E4E2;">-1.0%</span></td>
<td id="A1.T14.4.4.7.3.26" class="ltx_td ltx_align_right" style="background-color:#F7D5D2;"><span id="A1.T14.4.4.7.3.26.1" class="ltx_text" style="background-color:#F7D5D2;">-1.6%</span></td>
</tr>
<tr id="A1.T14.4.4.8.4" class="ltx_tr">
<th id="A1.T14.4.4.8.4.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T14.4.4.8.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4-g</th>
<td id="A1.T14.4.4.8.4.3" class="ltx_td ltx_align_right" style="background-color:#EEF9F4;"><span id="A1.T14.4.4.8.4.3.1" class="ltx_text" style="background-color:#EEF9F4;">0.7%</span></td>
<td id="A1.T14.4.4.8.4.4" class="ltx_td ltx_align_right" style="background-color:#E7F6EE;"><span id="A1.T14.4.4.8.4.4.1" class="ltx_text" style="background-color:#E7F6EE;">1.0%</span></td>
<td id="A1.T14.4.4.8.4.5" class="ltx_td ltx_align_right" style="background-color:#FDF8F7;"><span id="A1.T14.4.4.8.4.5.1" class="ltx_text" style="background-color:#FDF8F7;">-0.3%</span></td>
<td id="A1.T14.4.4.8.4.6" class="ltx_td ltx_align_right" style="background-color:#ECF8F2;"><span id="A1.T14.4.4.8.4.6.1" class="ltx_text" style="background-color:#ECF8F2;">0.8%</span></td>
<td id="A1.T14.4.4.8.4.7" class="ltx_td ltx_align_right" style="background-color:#FDF7F6;"><span id="A1.T14.4.4.8.4.7.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
<td id="A1.T14.4.4.8.4.8" class="ltx_td ltx_align_right" style="background-color:#BCE4D1;"><span id="A1.T14.4.4.8.4.8.1" class="ltx_text" style="background-color:#BCE4D1;">2.6%</span></td>
<td id="A1.T14.4.4.8.4.9" class="ltx_td ltx_align_right" style="background-color:#F1FAF6;"><span id="A1.T14.4.4.8.4.9.1" class="ltx_text" style="background-color:#F1FAF6;">0.5%</span></td>
<td id="A1.T14.4.4.8.4.10" class="ltx_td ltx_align_right" style="background-color:#FDF7F7;"><span id="A1.T14.4.4.8.4.10.1" class="ltx_text" style="background-color:#FDF7F7;">-0.3%</span></td>
<td id="A1.T14.4.4.8.4.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FAE8E7;"><span id="A1.T14.4.4.8.4.11.1" class="ltx_text" style="background-color:#FAE8E7;">-0.8%</span></td>
<td id="A1.T14.4.4.8.4.12" class="ltx_td ltx_align_right" style="background-color:#F5FBF8;"><span id="A1.T14.4.4.8.4.12.1" class="ltx_text" style="background-color:#F5FBF8;">0.4%</span></td>
<td id="A1.T14.4.4.8.4.13" class="ltx_td ltx_align_right" style="background-color:#FBFEFC;"><span id="A1.T14.4.4.8.4.13.1" class="ltx_text" style="background-color:#FBFEFC;">0.2%</span></td>
<td id="A1.T14.4.4.8.4.14" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#ECF8F2;"><span id="A1.T14.4.4.8.4.14.1" class="ltx_text" style="background-color:#ECF8F2;">0.7%</span></td>
<td id="A1.T14.4.4.8.4.15" class="ltx_td ltx_align_right" style="background-color:#FDF6F6;"><span id="A1.T14.4.4.8.4.15.1" class="ltx_text" style="background-color:#FDF6F6;">-0.3%</span></td>
<td id="A1.T14.4.4.8.4.16" class="ltx_td ltx_align_right" style="background-color:#FBFEFC;"><span id="A1.T14.4.4.8.4.16.1" class="ltx_text" style="background-color:#FBFEFC;">0.2%</span></td>
<td id="A1.T14.4.4.8.4.17" class="ltx_td ltx_align_right" style="background-color:#FCEFEE;"><span id="A1.T14.4.4.8.4.17.1" class="ltx_text" style="background-color:#FCEFEE;">-0.6%</span></td>
<td id="A1.T14.4.4.8.4.18" class="ltx_td ltx_align_right" style="background-color:#FDF6F5;"><span id="A1.T14.4.4.8.4.18.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="A1.T14.4.4.8.4.19" class="ltx_td ltx_align_right" style="background-color:#FBEFEE;"><span id="A1.T14.4.4.8.4.19.1" class="ltx_text" style="background-color:#FBEFEE;">-0.6%</span></td>
<td id="A1.T14.4.4.8.4.20" class="ltx_td ltx_align_right" style="background-color:#FBEDEC;"><span id="A1.T14.4.4.8.4.20.1" class="ltx_text" style="background-color:#FBEDEC;">-0.7%</span></td>
<td id="A1.T14.4.4.8.4.21" class="ltx_td ltx_align_right" style="background-color:#FAE7E5;"><span id="A1.T14.4.4.8.4.21.1" class="ltx_text" style="background-color:#FAE7E5;">-0.9%</span></td>
<td id="A1.T14.4.4.8.4.22" class="ltx_td ltx_align_right" style="background-color:#FDF6F6;"><span id="A1.T14.4.4.8.4.22.1" class="ltx_text" style="background-color:#FDF6F6;">-0.3%</span></td>
<td id="A1.T14.4.4.8.4.23" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F7D9D6;"><span id="A1.T14.4.4.8.4.23.1" class="ltx_text" style="background-color:#F7D9D6;">-1.4%</span></td>
<td id="A1.T14.4.4.8.4.24" class="ltx_td ltx_align_right" style="background-color:#FCF0EF;"><span id="A1.T14.4.4.8.4.24.1" class="ltx_text" style="background-color:#FCF0EF;">-0.6%</span></td>
<td id="A1.T14.4.4.8.4.25" class="ltx_td ltx_align_right" style="background-color:#FDF6F5;"><span id="A1.T14.4.4.8.4.25.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="A1.T14.4.4.8.4.26" class="ltx_td ltx_align_right" style="background-color:#FAE9E7;"><span id="A1.T14.4.4.8.4.26.1" class="ltx_text" style="background-color:#FAE9E7;">-0.8%</span></td>
</tr>
<tr id="A1.T14.4.4.9.5" class="ltx_tr">
<th id="A1.T14.4.4.9.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T14.4.4.9.5.1.1" class="ltx_text">103B</span></th>
<th id="A1.T14.4.4.9.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4</th>
<td id="A1.T14.4.4.9.5.3" class="ltx_td ltx_align_right" style="background-color:#FAE9E8;"><span id="A1.T14.4.4.9.5.3.1" class="ltx_text" style="background-color:#FAE9E8;">-0.8%</span></td>
<td id="A1.T14.4.4.9.5.4" class="ltx_td ltx_align_right" style="background-color:#F4C6C2;"><span id="A1.T14.4.4.9.5.4.1" class="ltx_text" style="background-color:#F4C6C2;">-2.2%</span></td>
<td id="A1.T14.4.4.9.5.5" class="ltx_td ltx_align_right" style="background-color:#FBEDEC;"><span id="A1.T14.4.4.9.5.5.1" class="ltx_text" style="background-color:#FBEDEC;">-0.7%</span></td>
<td id="A1.T14.4.4.9.5.6" class="ltx_td ltx_align_right" style="background-color:#F8DBD9;"><span id="A1.T14.4.4.9.5.6.1" class="ltx_text" style="background-color:#F8DBD9;">-1.3%</span></td>
<td id="A1.T14.4.4.9.5.7" class="ltx_td ltx_align_right" style="background-color:#FAE7E5;"><span id="A1.T14.4.4.9.5.7.1" class="ltx_text" style="background-color:#FAE7E5;">-0.9%</span></td>
<td id="A1.T14.4.4.9.5.8" class="ltx_td ltx_align_right" style="background-color:#FBEAE8;"><span id="A1.T14.4.4.9.5.8.1" class="ltx_text" style="background-color:#FBEAE8;">-0.8%</span></td>
<td id="A1.T14.4.4.9.5.9" class="ltx_td ltx_align_right" style="background-color:#E98D85;"><span id="A1.T14.4.4.9.5.9.1" class="ltx_text" style="background-color:#E98D85;">-4.3%</span></td>
<td id="A1.T14.4.4.9.5.10" class="ltx_td ltx_align_right" style="background-color:#F7D6D4;"><span id="A1.T14.4.4.9.5.10.1" class="ltx_text" style="background-color:#F7D6D4;">-1.5%</span></td>
<td id="A1.T14.4.4.9.5.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F3C3BF;"><span id="A1.T14.4.4.9.5.11.1" class="ltx_text" style="background-color:#F3C3BF;">-2.3%</span></td>
<td id="A1.T14.4.4.9.5.12" class="ltx_td ltx_align_right" style="background-color:#F6D3D1;"><span id="A1.T14.4.4.9.5.12.1" class="ltx_text" style="background-color:#F6D3D1;">-1.6%</span></td>
<td id="A1.T14.4.4.9.5.13" class="ltx_td ltx_align_right" style="background-color:#F8DCDA;"><span id="A1.T14.4.4.9.5.13.1" class="ltx_text" style="background-color:#F8DCDA;">-1.3%</span></td>
<td id="A1.T14.4.4.9.5.14" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F4C9C5;"><span id="A1.T14.4.4.9.5.14.1" class="ltx_text" style="background-color:#F4C9C5;">-2.0%</span></td>
<td id="A1.T14.4.4.9.5.15" class="ltx_td ltx_align_right" style="background-color:#F5CFCC;"><span id="A1.T14.4.4.9.5.15.1" class="ltx_text" style="background-color:#F5CFCC;">-1.8%</span></td>
<td id="A1.T14.4.4.9.5.16" class="ltx_td ltx_align_right" style="background-color:#F9E1DF;"><span id="A1.T14.4.4.9.5.16.1" class="ltx_text" style="background-color:#F9E1DF;">-1.1%</span></td>
<td id="A1.T14.4.4.9.5.17" class="ltx_td ltx_align_right" style="background-color:#FAE9E7;"><span id="A1.T14.4.4.9.5.17.1" class="ltx_text" style="background-color:#FAE9E7;">-0.8%</span></td>
<td id="A1.T14.4.4.9.5.18" class="ltx_td ltx_align_right" style="background-color:#F8DBD8;"><span id="A1.T14.4.4.9.5.18.1" class="ltx_text" style="background-color:#F8DBD8;">-1.4%</span></td>
<td id="A1.T14.4.4.9.5.19" class="ltx_td ltx_align_right" style="background-color:#F7D5D2;"><span id="A1.T14.4.4.9.5.19.1" class="ltx_text" style="background-color:#F7D5D2;">-1.6%</span></td>
<td id="A1.T14.4.4.9.5.20" class="ltx_td ltx_align_right" style="background-color:#F6D2CF;"><span id="A1.T14.4.4.9.5.20.1" class="ltx_text" style="background-color:#F6D2CF;">-1.7%</span></td>
<td id="A1.T14.4.4.9.5.21" class="ltx_td ltx_align_right" style="background-color:#F4C5C1;"><span id="A1.T14.4.4.9.5.21.1" class="ltx_text" style="background-color:#F4C5C1;">-2.2%</span></td>
<td id="A1.T14.4.4.9.5.22" class="ltx_td ltx_align_right" style="background-color:#FBEBEA;"><span id="A1.T14.4.4.9.5.22.1" class="ltx_text" style="background-color:#FBEBEA;">-0.7%</span></td>
<td id="A1.T14.4.4.9.5.23" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#ECA099;"><span id="A1.T14.4.4.9.5.23.1" class="ltx_text" style="background-color:#ECA099;">-3.6%</span></td>
<td id="A1.T14.4.4.9.5.24" class="ltx_td ltx_align_right" style="background-color:#F6D3D0;"><span id="A1.T14.4.4.9.5.24.1" class="ltx_text" style="background-color:#F6D3D0;">-1.7%</span></td>
<td id="A1.T14.4.4.9.5.25" class="ltx_td ltx_align_right" style="background-color:#F9E1DF;"><span id="A1.T14.4.4.9.5.25.1" class="ltx_text" style="background-color:#F9E1DF;">-1.1%</span></td>
<td id="A1.T14.4.4.9.5.26" class="ltx_td ltx_align_right" style="background-color:#F3C2BD;"><span id="A1.T14.4.4.9.5.26.1" class="ltx_text" style="background-color:#F3C2BD;">-2.3%</span></td>
</tr>
<tr id="A1.T14.4.4.10.6" class="ltx_tr">
<th id="A1.T14.4.4.10.6.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="A1.T14.4.4.10.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">W8</th>
<td id="A1.T14.4.4.10.6.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBECEB;"><span id="A1.T14.4.4.10.6.3.1" class="ltx_text" style="background-color:#FBECEB;">-0.7%</span></td>
<td id="A1.T14.4.4.10.6.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF4F3;"><span id="A1.T14.4.4.10.6.4.1" class="ltx_text" style="background-color:#FCF4F3;">-0.4%</span></td>
<td id="A1.T14.4.4.10.6.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFC;"><span id="A1.T14.4.4.10.6.5.1" class="ltx_text" style="background-color:#FEFDFC;">-0.1%</span></td>
<td id="A1.T14.4.4.10.6.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T14.4.4.10.6.6.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T14.4.4.10.6.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T14.4.4.10.6.7.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T14.4.4.10.6.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F8;"><span id="A1.T14.4.4.10.6.8.1" class="ltx_text" style="background-color:#FDF9F8;">-0.2%</span></td>
<td id="A1.T14.4.4.10.6.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#EDF8F3;"><span id="A1.T14.4.4.10.6.9.1" class="ltx_text" style="background-color:#EDF8F3;">0.7%</span></td>
<td id="A1.T14.4.4.10.6.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF5F5;"><span id="A1.T14.4.4.10.6.10.1" class="ltx_text" style="background-color:#FDF5F5;">-0.4%</span></td>
<td id="A1.T14.4.4.10.6.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FEF9F9;"><span id="A1.T14.4.4.10.6.11.1" class="ltx_text" style="background-color:#FEF9F9;">-0.2%</span></td>
<td id="A1.T14.4.4.10.6.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="A1.T14.4.4.10.6.12.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="A1.T14.4.4.10.6.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAFA;"><span id="A1.T14.4.4.10.6.13.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="A1.T14.4.4.10.6.14" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T14.4.4.10.6.14.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T14.4.4.10.6.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F9;"><span id="A1.T14.4.4.10.6.15.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="A1.T14.4.4.10.6.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEF9F9;"><span id="A1.T14.4.4.10.6.16.1" class="ltx_text" style="background-color:#FEF9F9;">-0.2%</span></td>
<td id="A1.T14.4.4.10.6.17" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF1F0;"><span id="A1.T14.4.4.10.6.17.1" class="ltx_text" style="background-color:#FCF1F0;">-0.5%</span></td>
<td id="A1.T14.4.4.10.6.18" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF6F5;"><span id="A1.T14.4.4.10.6.18.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="A1.T14.4.4.10.6.19" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T14.4.4.10.6.19.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T14.4.4.10.6.20" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#ECF8F2;"><span id="A1.T14.4.4.10.6.20.1" class="ltx_text" style="background-color:#ECF8F2;">0.8%</span></td>
<td id="A1.T14.4.4.10.6.21" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF2F1;"><span id="A1.T14.4.4.10.6.21.1" class="ltx_text" style="background-color:#FCF2F1;">-0.5%</span></td>
<td id="A1.T14.4.4.10.6.22" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T14.4.4.10.6.22.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T14.4.4.10.6.23" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FBEEED;"><span id="A1.T14.4.4.10.6.23.1" class="ltx_text" style="background-color:#FBEEED;">-0.6%</span></td>
<td id="A1.T14.4.4.10.6.24" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAF9;"><span id="A1.T14.4.4.10.6.24.1" class="ltx_text" style="background-color:#FEFAF9;">-0.2%</span></td>
<td id="A1.T14.4.4.10.6.25" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F8;"><span id="A1.T14.4.4.10.6.25.1" class="ltx_text" style="background-color:#FDF9F8;">-0.2%</span></td>
<td id="A1.T14.4.4.10.6.26" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFA;"><span id="A1.T14.4.4.10.6.26.1" class="ltx_text" style="background-color:#FEFBFA;">-0.1%</span></td>
</tr>
<tr id="A1.T14.4.4.11.7" class="ltx_tr">
<th id="A1.T14.4.4.11.7.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T14.4.4.11.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8</th>
<td id="A1.T14.4.4.11.7.3" class="ltx_td ltx_align_right" style="background-color:#72C69D;"><span id="A1.T14.4.4.11.7.3.1" class="ltx_text" style="background-color:#72C69D;">5.5%</span></td>
<td id="A1.T14.4.4.11.7.4" class="ltx_td ltx_align_right" style="background-color:#D0ECDE;"><span id="A1.T14.4.4.11.7.4.1" class="ltx_text" style="background-color:#D0ECDE;">1.9%</span></td>
<td id="A1.T14.4.4.11.7.5" class="ltx_td ltx_align_right" style="background-color:#FEFFFE;"><span id="A1.T14.4.4.11.7.5.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="A1.T14.4.4.11.7.6" class="ltx_td ltx_align_right" style="background-color:#DAF0E6;"><span id="A1.T14.4.4.11.7.6.1" class="ltx_text" style="background-color:#DAF0E6;">1.4%</span></td>
<td id="A1.T14.4.4.11.7.7" class="ltx_td ltx_align_right" style="background-color:#E7F6EF;"><span id="A1.T14.4.4.11.7.7.1" class="ltx_text" style="background-color:#E7F6EF;">0.9%</span></td>
<td id="A1.T14.4.4.11.7.8" class="ltx_td ltx_align_right" style="background-color:#C8E9D9;"><span id="A1.T14.4.4.11.7.8.1" class="ltx_text" style="background-color:#C8E9D9;">2.1%</span></td>
<td id="A1.T14.4.4.11.7.9" class="ltx_td ltx_align_right" style="background-color:#D0ECDE;"><span id="A1.T14.4.4.11.7.9.1" class="ltx_text" style="background-color:#D0ECDE;">1.9%</span></td>
<td id="A1.T14.4.4.11.7.10" class="ltx_td ltx_align_right" style="background-color:#DCF1E7;"><span id="A1.T14.4.4.11.7.10.1" class="ltx_text" style="background-color:#DCF1E7;">1.4%</span></td>
<td id="A1.T14.4.4.11.7.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E7F6EF;"><span id="A1.T14.4.4.11.7.11.1" class="ltx_text" style="background-color:#E7F6EF;">0.9%</span></td>
<td id="A1.T14.4.4.11.7.12" class="ltx_td ltx_align_right" style="background-color:#D1EDDF;"><span id="A1.T14.4.4.11.7.12.1" class="ltx_text" style="background-color:#D1EDDF;">1.8%</span></td>
<td id="A1.T14.4.4.11.7.13" class="ltx_td ltx_align_right" style="background-color:#E2F4EB;"><span id="A1.T14.4.4.11.7.13.1" class="ltx_text" style="background-color:#E2F4EB;">1.1%</span></td>
<td id="A1.T14.4.4.11.7.14" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#BCE4D1;"><span id="A1.T14.4.4.11.7.14.1" class="ltx_text" style="background-color:#BCE4D1;">2.6%</span></td>
<td id="A1.T14.4.4.11.7.15" class="ltx_td ltx_align_right" style="background-color:#F3FBF7;"><span id="A1.T14.4.4.11.7.15.1" class="ltx_text" style="background-color:#F3FBF7;">0.5%</span></td>
<td id="A1.T14.4.4.11.7.16" class="ltx_td ltx_align_right" style="background-color:#BEE5D2;"><span id="A1.T14.4.4.11.7.16.1" class="ltx_text" style="background-color:#BEE5D2;">2.5%</span></td>
<td id="A1.T14.4.4.11.7.17" class="ltx_td ltx_align_right" style="background-color:#FCF0EF;"><span id="A1.T14.4.4.11.7.17.1" class="ltx_text" style="background-color:#FCF0EF;">-0.6%</span></td>
<td id="A1.T14.4.4.11.7.18" class="ltx_td ltx_align_right" style="background-color:#E9F6F0;"><span id="A1.T14.4.4.11.7.18.1" class="ltx_text" style="background-color:#E9F6F0;">0.9%</span></td>
<td id="A1.T14.4.4.11.7.19" class="ltx_td ltx_align_right" style="background-color:#F2FAF6;"><span id="A1.T14.4.4.11.7.19.1" class="ltx_text" style="background-color:#F2FAF6;">0.5%</span></td>
<td id="A1.T14.4.4.11.7.20" class="ltx_td ltx_align_right" style="background-color:#E4F4EC;"><span id="A1.T14.4.4.11.7.20.1" class="ltx_text" style="background-color:#E4F4EC;">1.1%</span></td>
<td id="A1.T14.4.4.11.7.21" class="ltx_td ltx_align_right" style="background-color:#EBF7F2;"><span id="A1.T14.4.4.11.7.21.1" class="ltx_text" style="background-color:#EBF7F2;">0.8%</span></td>
<td id="A1.T14.4.4.11.7.22" class="ltx_td ltx_align_right" style="background-color:#E2F4EB;"><span id="A1.T14.4.4.11.7.22.1" class="ltx_text" style="background-color:#E2F4EB;">1.1%</span></td>
<td id="A1.T14.4.4.11.7.23" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E6F5EE;"><span id="A1.T14.4.4.11.7.23.1" class="ltx_text" style="background-color:#E6F5EE;">1.0%</span></td>
<td id="A1.T14.4.4.11.7.24" class="ltx_td ltx_align_right" style="background-color:#E9F6F0;"><span id="A1.T14.4.4.11.7.24.1" class="ltx_text" style="background-color:#E9F6F0;">0.9%</span></td>
<td id="A1.T14.4.4.11.7.25" class="ltx_td ltx_align_right" style="background-color:#E8F6EF;"><span id="A1.T14.4.4.11.7.25.1" class="ltx_text" style="background-color:#E8F6EF;">0.9%</span></td>
<td id="A1.T14.4.4.11.7.26" class="ltx_td ltx_align_right" style="background-color:#EAF7F1;"><span id="A1.T14.4.4.11.7.26.1" class="ltx_text" style="background-color:#EAF7F1;">0.8%</span></td>
</tr>
<tr id="A1.T14.4.4.12.8" class="ltx_tr">
<th id="A1.T14.4.4.12.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A1.T14.4.4.12.8.1.1" class="ltx_text">35B</span></th>
<th id="A1.T14.4.4.12.8.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">W4-g</th>
<td id="A1.T14.4.4.12.8.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E67C73;"><span id="A1.T14.4.4.12.8.3.1" class="ltx_text" style="background-color:#E67C73;">-6.7%</span></td>
<td id="A1.T14.4.4.12.8.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F5CCC9;"><span id="A1.T14.4.4.12.8.4.1" class="ltx_text" style="background-color:#F5CCC9;">-1.9%</span></td>
<td id="A1.T14.4.4.12.8.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FCF3F2;"><span id="A1.T14.4.4.12.8.5.1" class="ltx_text" style="background-color:#FCF3F2;">-0.4%</span></td>
<td id="A1.T14.4.4.12.8.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F8DBD9;"><span id="A1.T14.4.4.12.8.6.1" class="ltx_text" style="background-color:#F8DBD9;">-1.3%</span></td>
<td id="A1.T14.4.4.12.8.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FCF4F3;"><span id="A1.T14.4.4.12.8.7.1" class="ltx_text" style="background-color:#FCF4F3;">-0.4%</span></td>
<td id="A1.T14.4.4.12.8.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EB9A93;"><span id="A1.T14.4.4.12.8.8.1" class="ltx_text" style="background-color:#EB9A93;">-3.9%</span></td>
<td id="A1.T14.4.4.12.8.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F0B5B0;"><span id="A1.T14.4.4.12.8.9.1" class="ltx_text" style="background-color:#F0B5B0;">-2.8%</span></td>
<td id="A1.T14.4.4.12.8.10" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FBEBEA;"><span id="A1.T14.4.4.12.8.10.1" class="ltx_text" style="background-color:#FBEBEA;">-0.7%</span></td>
<td id="A1.T14.4.4.12.8.11" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F6D1CE;"><span id="A1.T14.4.4.12.8.11.1" class="ltx_text" style="background-color:#F6D1CE;">-1.7%</span></td>
<td id="A1.T14.4.4.12.8.12" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C5C1;"><span id="A1.T14.4.4.12.8.12.1" class="ltx_text" style="background-color:#F3C5C1;">-2.2%</span></td>
<td id="A1.T14.4.4.12.8.13" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE5E3;"><span id="A1.T14.4.4.12.8.13.1" class="ltx_text" style="background-color:#FAE5E3;">-1.0%</span></td>
<td id="A1.T14.4.4.12.8.14" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#EC9C95;"><span id="A1.T14.4.4.12.8.14.1" class="ltx_text" style="background-color:#EC9C95;">-3.8%</span></td>
<td id="A1.T14.4.4.12.8.15" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FEFDFD;"><span id="A1.T14.4.4.12.8.15.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T14.4.4.12.8.16" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F0F9F5;"><span id="A1.T14.4.4.12.8.16.1" class="ltx_text" style="background-color:#F0F9F5;">0.6%</span></td>
<td id="A1.T14.4.4.12.8.17" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F2BDB8;"><span id="A1.T14.4.4.12.8.17.1" class="ltx_text" style="background-color:#F2BDB8;">-2.5%</span></td>
<td id="A1.T14.4.4.12.8.18" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FFFFFF;"><span id="A1.T14.4.4.12.8.18.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T14.4.4.12.8.19" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FEFBFB;"><span id="A1.T14.4.4.12.8.19.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="A1.T14.4.4.12.8.20" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F0B5AF;"><span id="A1.T14.4.4.12.8.20.1" class="ltx_text" style="background-color:#F0B5AF;">-2.8%</span></td>
<td id="A1.T14.4.4.12.8.21" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9E3E1;"><span id="A1.T14.4.4.12.8.21.1" class="ltx_text" style="background-color:#F9E3E1;">-1.1%</span></td>
<td id="A1.T14.4.4.12.8.22" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FDF8F8;"><span id="A1.T14.4.4.12.8.22.1" class="ltx_text" style="background-color:#FDF8F8;">-0.2%</span></td>
<td id="A1.T14.4.4.12.8.23" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F7DAD7;"><span id="A1.T14.4.4.12.8.23.1" class="ltx_text" style="background-color:#F7DAD7;">-1.4%</span></td>
<td id="A1.T14.4.4.12.8.24" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE8E7;"><span id="A1.T14.4.4.12.8.24.1" class="ltx_text" style="background-color:#FAE8E7;">-0.8%</span></td>
<td id="A1.T14.4.4.12.8.25" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FCF2F2;"><span id="A1.T14.4.4.12.8.25.1" class="ltx_text" style="background-color:#FCF2F2;">-0.5%</span></td>
<td id="A1.T14.4.4.12.8.26" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F8DCD9;"><span id="A1.T14.4.4.12.8.26.1" class="ltx_text" style="background-color:#F8DCD9;">-1.3%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A14: </span><span id="A1.T14.7.1" class="ltx_text ltx_font_bold">Relative performance (%<math id="A1.T14.7.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="A1.T14.7.1.m1.1b"><mi mathvariant="normal" id="A1.T14.7.1.m1.1.1" xref="A1.T14.7.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="A1.T14.7.1.m1.1c"><ci id="A1.T14.7.1.m1.1.1.cmml" xref="A1.T14.7.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T14.7.1.m1.1d">\Delta</annotation></semantics></math>) vs. FP16 for Command Models on FLORES</span>. Ltn/IE are Latin-script/Indo-European languages: de, es, fr, it, pt. <math id="A1.T14.8.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T14.8.m1.1b"><mo id="A1.T14.8.m1.1.1" xref="A1.T14.8.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T14.8.m1.1c"><not id="A1.T14.8.m1.1.1.cmml" xref="A1.T14.8.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T14.8.m1.1d">\neg</annotation></semantics></math> are the rest: ar, ja, ko, zh.</figcaption>
</figure>
<figure id="A1.T15" class="ltx_table">
<div id="A1.T15.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:136.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-210.8pt,66.6pt) scale(0.506992390989362,0.506992390989362) ;">
<table id="A1.T15.2.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T15.1.1.1" class="ltx_tr">
<td id="A1.T15.1.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="A1.T15.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" colspan="24"><span id="A1.T15.1.1.1.1.1" class="ltx_text ltx_font_bold">English<math id="A1.T15.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A1.T15.1.1.1.1.1.m1.1a"><mo stretchy="false" id="A1.T15.1.1.1.1.1.m1.1.1" xref="A1.T15.1.1.1.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A1.T15.1.1.1.1.1.m1.1b"><ci id="A1.T15.1.1.1.1.1.m1.1.1.cmml" xref="A1.T15.1.1.1.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T15.1.1.1.1.1.m1.1c">\rightarrow</annotation></semantics></math>L2</span></td>
</tr>
<tr id="A1.T15.2.2.3.1" class="ltx_tr">
<td id="A1.T15.2.2.3.1.1" class="ltx_td"></td>
<td id="A1.T15.2.2.3.1.2" class="ltx_td ltx_border_r"></td>
<td id="A1.T15.2.2.3.1.3" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.3.1" class="ltx_text ltx_font_bold">ar</span></td>
<td id="A1.T15.2.2.3.1.4" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.4.1" class="ltx_text ltx_font_bold">cs</span></td>
<td id="A1.T15.2.2.3.1.5" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.5.1" class="ltx_text ltx_font_bold">de</span></td>
<td id="A1.T15.2.2.3.1.6" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.6.1" class="ltx_text ltx_font_bold">el</span></td>
<td id="A1.T15.2.2.3.1.7" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.7.1" class="ltx_text ltx_font_bold">es</span></td>
<td id="A1.T15.2.2.3.1.8" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.8.1" class="ltx_text ltx_font_bold">fa</span></td>
<td id="A1.T15.2.2.3.1.9" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.9.1" class="ltx_text ltx_font_bold">fr</span></td>
<td id="A1.T15.2.2.3.1.10" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.10.1" class="ltx_text ltx_font_bold">he</span></td>
<td id="A1.T15.2.2.3.1.11" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.11.1" class="ltx_text ltx_font_bold">hi</span></td>
<td id="A1.T15.2.2.3.1.12" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.12.1" class="ltx_text ltx_font_bold">id</span></td>
<td id="A1.T15.2.2.3.1.13" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.13.1" class="ltx_text ltx_font_bold">it</span></td>
<td id="A1.T15.2.2.3.1.14" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.14.1" class="ltx_text ltx_font_bold">ja</span></td>
<td id="A1.T15.2.2.3.1.15" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.15.1" class="ltx_text ltx_font_bold">ko</span></td>
<td id="A1.T15.2.2.3.1.16" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.16.1" class="ltx_text ltx_font_bold">nl</span></td>
<td id="A1.T15.2.2.3.1.17" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.17.1" class="ltx_text ltx_font_bold">pl</span></td>
<td id="A1.T15.2.2.3.1.18" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.18.1" class="ltx_text ltx_font_bold">pt</span></td>
<td id="A1.T15.2.2.3.1.19" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.19.1" class="ltx_text ltx_font_bold">ro</span></td>
<td id="A1.T15.2.2.3.1.20" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.20.1" class="ltx_text ltx_font_bold">ru</span></td>
<td id="A1.T15.2.2.3.1.21" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.21.1" class="ltx_text ltx_font_bold">tr</span></td>
<td id="A1.T15.2.2.3.1.22" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.22.1" class="ltx_text ltx_font_bold">uk</span></td>
<td id="A1.T15.2.2.3.1.23" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.23.1" class="ltx_text ltx_font_bold">vi</span></td>
<td id="A1.T15.2.2.3.1.24" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T15.2.2.3.1.24.1" class="ltx_text ltx_font_bold">zh</span></td>
<td id="A1.T15.2.2.3.1.25" class="ltx_td ltx_align_center"><span id="A1.T15.2.2.3.1.25.1" class="ltx_text ltx_font_bold">Avg</span></td>
</tr>
<tr id="A1.T15.2.2.4.2" class="ltx_tr">
<td id="A1.T15.2.2.4.2.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T15.2.2.4.2.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T15.2.2.4.2.3" class="ltx_td ltx_align_right ltx_border_t">40.0</td>
<td id="A1.T15.2.2.4.2.4" class="ltx_td ltx_align_right ltx_border_t">39.1</td>
<td id="A1.T15.2.2.4.2.5" class="ltx_td ltx_align_right ltx_border_t">42.5</td>
<td id="A1.T15.2.2.4.2.6" class="ltx_td ltx_align_right ltx_border_t">36.3</td>
<td id="A1.T15.2.2.4.2.7" class="ltx_td ltx_align_right ltx_border_t">32.1</td>
<td id="A1.T15.2.2.4.2.8" class="ltx_td ltx_align_right ltx_border_t">33.4</td>
<td id="A1.T15.2.2.4.2.9" class="ltx_td ltx_align_right ltx_border_t">54.1</td>
<td id="A1.T15.2.2.4.2.10" class="ltx_td ltx_align_right ltx_border_t">39.5</td>
<td id="A1.T15.2.2.4.2.11" class="ltx_td ltx_align_right ltx_border_t">31.9</td>
<td id="A1.T15.2.2.4.2.12" class="ltx_td ltx_align_right ltx_border_t">44.7</td>
<td id="A1.T15.2.2.4.2.13" class="ltx_td ltx_align_right ltx_border_t">36.6</td>
<td id="A1.T15.2.2.4.2.14" class="ltx_td ltx_align_right ltx_border_t">28.7</td>
<td id="A1.T15.2.2.4.2.15" class="ltx_td ltx_align_right ltx_border_t">25.5</td>
<td id="A1.T15.2.2.4.2.16" class="ltx_td ltx_align_right ltx_border_t">33.4</td>
<td id="A1.T15.2.2.4.2.17" class="ltx_td ltx_align_right ltx_border_t">30.7</td>
<td id="A1.T15.2.2.4.2.18" class="ltx_td ltx_align_right ltx_border_t">53.1</td>
<td id="A1.T15.2.2.4.2.19" class="ltx_td ltx_align_right ltx_border_t">43.3</td>
<td id="A1.T15.2.2.4.2.20" class="ltx_td ltx_align_right ltx_border_t">38.9</td>
<td id="A1.T15.2.2.4.2.21" class="ltx_td ltx_align_right ltx_border_t">33.8</td>
<td id="A1.T15.2.2.4.2.22" class="ltx_td ltx_align_right ltx_border_t">38.2</td>
<td id="A1.T15.2.2.4.2.23" class="ltx_td ltx_align_right ltx_border_t">41.0</td>
<td id="A1.T15.2.2.4.2.24" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">34.0</td>
<td id="A1.T15.2.2.4.2.25" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.4.2.25.1" class="ltx_text" style="background-color:#E6E6E6;">37.8</span></td>
</tr>
<tr id="A1.T15.2.2.5.3" class="ltx_tr">
<td id="A1.T15.2.2.5.3.1" class="ltx_td ltx_align_left">Aya-23-35b</td>
<td id="A1.T15.2.2.5.3.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T15.2.2.5.3.3" class="ltx_td ltx_align_right">40.0</td>
<td id="A1.T15.2.2.5.3.4" class="ltx_td ltx_align_right">39.0</td>
<td id="A1.T15.2.2.5.3.5" class="ltx_td ltx_align_right">42.9</td>
<td id="A1.T15.2.2.5.3.6" class="ltx_td ltx_align_right">36.2</td>
<td id="A1.T15.2.2.5.3.7" class="ltx_td ltx_align_right">32.2</td>
<td id="A1.T15.2.2.5.3.8" class="ltx_td ltx_align_right">33.7</td>
<td id="A1.T15.2.2.5.3.9" class="ltx_td ltx_align_right">53.9</td>
<td id="A1.T15.2.2.5.3.10" class="ltx_td ltx_align_right">40.0</td>
<td id="A1.T15.2.2.5.3.11" class="ltx_td ltx_align_right">32.3</td>
<td id="A1.T15.2.2.5.3.12" class="ltx_td ltx_align_right">44.8</td>
<td id="A1.T15.2.2.5.3.13" class="ltx_td ltx_align_right">36.5</td>
<td id="A1.T15.2.2.5.3.14" class="ltx_td ltx_align_right">28.9</td>
<td id="A1.T15.2.2.5.3.15" class="ltx_td ltx_align_right">25.5</td>
<td id="A1.T15.2.2.5.3.16" class="ltx_td ltx_align_right">33.6</td>
<td id="A1.T15.2.2.5.3.17" class="ltx_td ltx_align_right">30.9</td>
<td id="A1.T15.2.2.5.3.18" class="ltx_td ltx_align_right">53.2</td>
<td id="A1.T15.2.2.5.3.19" class="ltx_td ltx_align_right">43.4</td>
<td id="A1.T15.2.2.5.3.20" class="ltx_td ltx_align_right">38.7</td>
<td id="A1.T15.2.2.5.3.21" class="ltx_td ltx_align_right">33.8</td>
<td id="A1.T15.2.2.5.3.22" class="ltx_td ltx_align_right">38.3</td>
<td id="A1.T15.2.2.5.3.23" class="ltx_td ltx_align_right">41.4</td>
<td id="A1.T15.2.2.5.3.24" class="ltx_td ltx_align_right ltx_border_r">33.8</td>
<td id="A1.T15.2.2.5.3.25" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.5.3.25.1" class="ltx_text" style="background-color:#E6E6E6;">37.9</span></td>
</tr>
<tr id="A1.T15.2.2.6.4" class="ltx_tr">
<td id="A1.T15.2.2.6.4.1" class="ltx_td"></td>
<td id="A1.T15.2.2.6.4.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T15.2.2.6.4.3" class="ltx_td ltx_align_right">39.3</td>
<td id="A1.T15.2.2.6.4.4" class="ltx_td ltx_align_right">38.0</td>
<td id="A1.T15.2.2.6.4.5" class="ltx_td ltx_align_right">42.5</td>
<td id="A1.T15.2.2.6.4.6" class="ltx_td ltx_align_right">36.0</td>
<td id="A1.T15.2.2.6.4.7" class="ltx_td ltx_align_right">32.0</td>
<td id="A1.T15.2.2.6.4.8" class="ltx_td ltx_align_right">32.6</td>
<td id="A1.T15.2.2.6.4.9" class="ltx_td ltx_align_right">53.3</td>
<td id="A1.T15.2.2.6.4.10" class="ltx_td ltx_align_right">39.1</td>
<td id="A1.T15.2.2.6.4.11" class="ltx_td ltx_align_right">31.2</td>
<td id="A1.T15.2.2.6.4.12" class="ltx_td ltx_align_right">44.6</td>
<td id="A1.T15.2.2.6.4.13" class="ltx_td ltx_align_right">36.1</td>
<td id="A1.T15.2.2.6.4.14" class="ltx_td ltx_align_right">28.2</td>
<td id="A1.T15.2.2.6.4.15" class="ltx_td ltx_align_right">25.1</td>
<td id="A1.T15.2.2.6.4.16" class="ltx_td ltx_align_right">32.9</td>
<td id="A1.T15.2.2.6.4.17" class="ltx_td ltx_align_right">30.0</td>
<td id="A1.T15.2.2.6.4.18" class="ltx_td ltx_align_right">52.8</td>
<td id="A1.T15.2.2.6.4.19" class="ltx_td ltx_align_right">42.6</td>
<td id="A1.T15.2.2.6.4.20" class="ltx_td ltx_align_right">38.3</td>
<td id="A1.T15.2.2.6.4.21" class="ltx_td ltx_align_right">33.2</td>
<td id="A1.T15.2.2.6.4.22" class="ltx_td ltx_align_right">37.8</td>
<td id="A1.T15.2.2.6.4.23" class="ltx_td ltx_align_right">40.8</td>
<td id="A1.T15.2.2.6.4.24" class="ltx_td ltx_align_right ltx_border_r">33.1</td>
<td id="A1.T15.2.2.6.4.25" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.6.4.25.1" class="ltx_text" style="background-color:#E6E6E6;">37.3</span></td>
</tr>
<tr id="A1.T15.2.2.7.5" class="ltx_tr">
<td id="A1.T15.2.2.7.5.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T15.2.2.7.5.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T15.2.2.7.5.3" class="ltx_td ltx_align_right ltx_border_t">36.3</td>
<td id="A1.T15.2.2.7.5.4" class="ltx_td ltx_align_right ltx_border_t">35.7</td>
<td id="A1.T15.2.2.7.5.5" class="ltx_td ltx_align_right ltx_border_t">39.3</td>
<td id="A1.T15.2.2.7.5.6" class="ltx_td ltx_align_right ltx_border_t">34.0</td>
<td id="A1.T15.2.2.7.5.7" class="ltx_td ltx_align_right ltx_border_t">31.5</td>
<td id="A1.T15.2.2.7.5.8" class="ltx_td ltx_align_right ltx_border_t">30.0</td>
<td id="A1.T15.2.2.7.5.9" class="ltx_td ltx_align_right ltx_border_t">51.0</td>
<td id="A1.T15.2.2.7.5.10" class="ltx_td ltx_align_right ltx_border_t">35.0</td>
<td id="A1.T15.2.2.7.5.11" class="ltx_td ltx_align_right ltx_border_t">27.2</td>
<td id="A1.T15.2.2.7.5.12" class="ltx_td ltx_align_right ltx_border_t">43.4</td>
<td id="A1.T15.2.2.7.5.13" class="ltx_td ltx_align_right ltx_border_t">34.7</td>
<td id="A1.T15.2.2.7.5.14" class="ltx_td ltx_align_right ltx_border_t">24.9</td>
<td id="A1.T15.2.2.7.5.15" class="ltx_td ltx_align_right ltx_border_t">22.0</td>
<td id="A1.T15.2.2.7.5.16" class="ltx_td ltx_align_right ltx_border_t">32.2</td>
<td id="A1.T15.2.2.7.5.17" class="ltx_td ltx_align_right ltx_border_t">28.4</td>
<td id="A1.T15.2.2.7.5.18" class="ltx_td ltx_align_right ltx_border_t">50.2</td>
<td id="A1.T15.2.2.7.5.19" class="ltx_td ltx_align_right ltx_border_t">41.6</td>
<td id="A1.T15.2.2.7.5.20" class="ltx_td ltx_align_right ltx_border_t">35.0</td>
<td id="A1.T15.2.2.7.5.21" class="ltx_td ltx_align_right ltx_border_t">29.1</td>
<td id="A1.T15.2.2.7.5.22" class="ltx_td ltx_align_right ltx_border_t">34.2</td>
<td id="A1.T15.2.2.7.5.23" class="ltx_td ltx_align_right ltx_border_t">39.0</td>
<td id="A1.T15.2.2.7.5.24" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">30.1</td>
<td id="A1.T15.2.2.7.5.25" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.7.5.25.1" class="ltx_text" style="background-color:#E6E6E6;">34.8</span></td>
</tr>
<tr id="A1.T15.2.2.8.6" class="ltx_tr">
<td id="A1.T15.2.2.8.6.1" class="ltx_td ltx_align_left">Aya-23-8b</td>
<td id="A1.T15.2.2.8.6.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T15.2.2.8.6.3" class="ltx_td ltx_align_right">36.5</td>
<td id="A1.T15.2.2.8.6.4" class="ltx_td ltx_align_right">36.1</td>
<td id="A1.T15.2.2.8.6.5" class="ltx_td ltx_align_right">39.5</td>
<td id="A1.T15.2.2.8.6.6" class="ltx_td ltx_align_right">33.9</td>
<td id="A1.T15.2.2.8.6.7" class="ltx_td ltx_align_right">31.4</td>
<td id="A1.T15.2.2.8.6.8" class="ltx_td ltx_align_right">30.4</td>
<td id="A1.T15.2.2.8.6.9" class="ltx_td ltx_align_right">51.4</td>
<td id="A1.T15.2.2.8.6.10" class="ltx_td ltx_align_right">35.0</td>
<td id="A1.T15.2.2.8.6.11" class="ltx_td ltx_align_right">27.0</td>
<td id="A1.T15.2.2.8.6.12" class="ltx_td ltx_align_right">43.2</td>
<td id="A1.T15.2.2.8.6.13" class="ltx_td ltx_align_right">34.8</td>
<td id="A1.T15.2.2.8.6.14" class="ltx_td ltx_align_right">24.8</td>
<td id="A1.T15.2.2.8.6.15" class="ltx_td ltx_align_right">22.2</td>
<td id="A1.T15.2.2.8.6.16" class="ltx_td ltx_align_right">32.1</td>
<td id="A1.T15.2.2.8.6.17" class="ltx_td ltx_align_right">28.5</td>
<td id="A1.T15.2.2.8.6.18" class="ltx_td ltx_align_right">50.0</td>
<td id="A1.T15.2.2.8.6.19" class="ltx_td ltx_align_right">42.0</td>
<td id="A1.T15.2.2.8.6.20" class="ltx_td ltx_align_right">34.9</td>
<td id="A1.T15.2.2.8.6.21" class="ltx_td ltx_align_right">28.9</td>
<td id="A1.T15.2.2.8.6.22" class="ltx_td ltx_align_right">34.3</td>
<td id="A1.T15.2.2.8.6.23" class="ltx_td ltx_align_right">39.0</td>
<td id="A1.T15.2.2.8.6.24" class="ltx_td ltx_align_right ltx_border_r">30.6</td>
<td id="A1.T15.2.2.8.6.25" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.8.6.25.1" class="ltx_text" style="background-color:#E6E6E6;">34.8</span></td>
</tr>
<tr id="A1.T15.2.2.9.7" class="ltx_tr">
<td id="A1.T15.2.2.9.7.1" class="ltx_td"></td>
<td id="A1.T15.2.2.9.7.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T15.2.2.9.7.3" class="ltx_td ltx_align_right">35.4</td>
<td id="A1.T15.2.2.9.7.4" class="ltx_td ltx_align_right">35.0</td>
<td id="A1.T15.2.2.9.7.5" class="ltx_td ltx_align_right">39.2</td>
<td id="A1.T15.2.2.9.7.6" class="ltx_td ltx_align_right">33.4</td>
<td id="A1.T15.2.2.9.7.7" class="ltx_td ltx_align_right">31.2</td>
<td id="A1.T15.2.2.9.7.8" class="ltx_td ltx_align_right">29.6</td>
<td id="A1.T15.2.2.9.7.9" class="ltx_td ltx_align_right">50.2</td>
<td id="A1.T15.2.2.9.7.10" class="ltx_td ltx_align_right">33.3</td>
<td id="A1.T15.2.2.9.7.11" class="ltx_td ltx_align_right">26.4</td>
<td id="A1.T15.2.2.9.7.12" class="ltx_td ltx_align_right">42.8</td>
<td id="A1.T15.2.2.9.7.13" class="ltx_td ltx_align_right">34.3</td>
<td id="A1.T15.2.2.9.7.14" class="ltx_td ltx_align_right">24.3</td>
<td id="A1.T15.2.2.9.7.15" class="ltx_td ltx_align_right">21.5</td>
<td id="A1.T15.2.2.9.7.16" class="ltx_td ltx_align_right">31.8</td>
<td id="A1.T15.2.2.9.7.17" class="ltx_td ltx_align_right">28.0</td>
<td id="A1.T15.2.2.9.7.18" class="ltx_td ltx_align_right">49.8</td>
<td id="A1.T15.2.2.9.7.19" class="ltx_td ltx_align_right">40.9</td>
<td id="A1.T15.2.2.9.7.20" class="ltx_td ltx_align_right">34.2</td>
<td id="A1.T15.2.2.9.7.21" class="ltx_td ltx_align_right">28.1</td>
<td id="A1.T15.2.2.9.7.22" class="ltx_td ltx_align_right">33.7</td>
<td id="A1.T15.2.2.9.7.23" class="ltx_td ltx_align_right">38.5</td>
<td id="A1.T15.2.2.9.7.24" class="ltx_td ltx_align_right ltx_border_r">29.4</td>
<td id="A1.T15.2.2.9.7.25" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.9.7.25.1" class="ltx_text" style="background-color:#E6E6E6;">34.1</span></td>
</tr>
<tr id="A1.T15.2.2.2" class="ltx_tr">
<td id="A1.T15.2.2.2.2" class="ltx_td ltx_border_tt"></td>
<td id="A1.T15.2.2.2.1" class="ltx_td ltx_align_center ltx_border_tt" colspan="24"><span id="A1.T15.2.2.2.1.1" class="ltx_text ltx_font_bold">L2<math id="A1.T15.2.2.2.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A1.T15.2.2.2.1.1.m1.1a"><mo stretchy="false" id="A1.T15.2.2.2.1.1.m1.1.1" xref="A1.T15.2.2.2.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A1.T15.2.2.2.1.1.m1.1b"><ci id="A1.T15.2.2.2.1.1.m1.1.1.cmml" xref="A1.T15.2.2.2.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T15.2.2.2.1.1.m1.1c">\rightarrow</annotation></semantics></math>English</span></td>
</tr>
<tr id="A1.T15.2.2.10.8" class="ltx_tr">
<td id="A1.T15.2.2.10.8.1" class="ltx_td"></td>
<td id="A1.T15.2.2.10.8.2" class="ltx_td ltx_align_left ltx_border_r">FP16</td>
<td id="A1.T15.2.2.10.8.3" class="ltx_td ltx_align_right">46.4</td>
<td id="A1.T15.2.2.10.8.4" class="ltx_td ltx_align_right">45.3</td>
<td id="A1.T15.2.2.10.8.5" class="ltx_td ltx_align_right">48.9</td>
<td id="A1.T15.2.2.10.8.6" class="ltx_td ltx_align_right">42.4</td>
<td id="A1.T15.2.2.10.8.7" class="ltx_td ltx_align_right">37.7</td>
<td id="A1.T15.2.2.10.8.8" class="ltx_td ltx_align_right">41.3</td>
<td id="A1.T15.2.2.10.8.9" class="ltx_td ltx_align_right">50.6</td>
<td id="A1.T15.2.2.10.8.10" class="ltx_td ltx_align_right">48.3</td>
<td id="A1.T15.2.2.10.8.11" class="ltx_td ltx_align_right">42.7</td>
<td id="A1.T15.2.2.10.8.12" class="ltx_td ltx_align_right">48.5</td>
<td id="A1.T15.2.2.10.8.13" class="ltx_td ltx_align_right">40.5</td>
<td id="A1.T15.2.2.10.8.14" class="ltx_td ltx_align_right">33.7</td>
<td id="A1.T15.2.2.10.8.15" class="ltx_td ltx_align_right">35.3</td>
<td id="A1.T15.2.2.10.8.16" class="ltx_td ltx_align_right">37.7</td>
<td id="A1.T15.2.2.10.8.17" class="ltx_td ltx_align_right">36.4</td>
<td id="A1.T15.2.2.10.8.18" class="ltx_td ltx_align_right">54.8</td>
<td id="A1.T15.2.2.10.8.19" class="ltx_td ltx_align_right">49.5</td>
<td id="A1.T15.2.2.10.8.20" class="ltx_td ltx_align_right">41.6</td>
<td id="A1.T15.2.2.10.8.21" class="ltx_td ltx_align_right">42.2</td>
<td id="A1.T15.2.2.10.8.22" class="ltx_td ltx_align_right">44.8</td>
<td id="A1.T15.2.2.10.8.23" class="ltx_td ltx_align_right">41.4</td>
<td id="A1.T15.2.2.10.8.24" class="ltx_td ltx_align_right ltx_border_r">34.8</td>
<td id="A1.T15.2.2.10.8.25" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.10.8.25.1" class="ltx_text" style="background-color:#E6E6E6;">42.9</span></td>
</tr>
<tr id="A1.T15.2.2.11.9" class="ltx_tr">
<td id="A1.T15.2.2.11.9.1" class="ltx_td ltx_align_left">Aya-23-35b</td>
<td id="A1.T15.2.2.11.9.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T15.2.2.11.9.3" class="ltx_td ltx_align_right">46.4</td>
<td id="A1.T15.2.2.11.9.4" class="ltx_td ltx_align_right">45.4</td>
<td id="A1.T15.2.2.11.9.5" class="ltx_td ltx_align_right">49.0</td>
<td id="A1.T15.2.2.11.9.6" class="ltx_td ltx_align_right">42.2</td>
<td id="A1.T15.2.2.11.9.7" class="ltx_td ltx_align_right">37.3</td>
<td id="A1.T15.2.2.11.9.8" class="ltx_td ltx_align_right">41.4</td>
<td id="A1.T15.2.2.11.9.9" class="ltx_td ltx_align_right">50.7</td>
<td id="A1.T15.2.2.11.9.10" class="ltx_td ltx_align_right">48.6</td>
<td id="A1.T15.2.2.11.9.11" class="ltx_td ltx_align_right">42.9</td>
<td id="A1.T15.2.2.11.9.12" class="ltx_td ltx_align_right">48.7</td>
<td id="A1.T15.2.2.11.9.13" class="ltx_td ltx_align_right">40.5</td>
<td id="A1.T15.2.2.11.9.14" class="ltx_td ltx_align_right">34.0</td>
<td id="A1.T15.2.2.11.9.15" class="ltx_td ltx_align_right">35.1</td>
<td id="A1.T15.2.2.11.9.16" class="ltx_td ltx_align_right">37.5</td>
<td id="A1.T15.2.2.11.9.17" class="ltx_td ltx_align_right">36.4</td>
<td id="A1.T15.2.2.11.9.18" class="ltx_td ltx_align_right">54.8</td>
<td id="A1.T15.2.2.11.9.19" class="ltx_td ltx_align_right">49.5</td>
<td id="A1.T15.2.2.11.9.20" class="ltx_td ltx_align_right">41.7</td>
<td id="A1.T15.2.2.11.9.21" class="ltx_td ltx_align_right">42.2</td>
<td id="A1.T15.2.2.11.9.22" class="ltx_td ltx_align_right">45.0</td>
<td id="A1.T15.2.2.11.9.23" class="ltx_td ltx_align_right">41.5</td>
<td id="A1.T15.2.2.11.9.24" class="ltx_td ltx_align_right ltx_border_r">34.9</td>
<td id="A1.T15.2.2.11.9.25" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.11.9.25.1" class="ltx_text" style="background-color:#E6E6E6;">43.0</span></td>
</tr>
<tr id="A1.T15.2.2.12.10" class="ltx_tr">
<td id="A1.T15.2.2.12.10.1" class="ltx_td"></td>
<td id="A1.T15.2.2.12.10.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T15.2.2.12.10.3" class="ltx_td ltx_align_right">45.7</td>
<td id="A1.T15.2.2.12.10.4" class="ltx_td ltx_align_right">44.9</td>
<td id="A1.T15.2.2.12.10.5" class="ltx_td ltx_align_right">48.5</td>
<td id="A1.T15.2.2.12.10.6" class="ltx_td ltx_align_right">41.8</td>
<td id="A1.T15.2.2.12.10.7" class="ltx_td ltx_align_right">37.5</td>
<td id="A1.T15.2.2.12.10.8" class="ltx_td ltx_align_right">40.5</td>
<td id="A1.T15.2.2.12.10.9" class="ltx_td ltx_align_right">50.4</td>
<td id="A1.T15.2.2.12.10.10" class="ltx_td ltx_align_right">47.3</td>
<td id="A1.T15.2.2.12.10.11" class="ltx_td ltx_align_right">41.8</td>
<td id="A1.T15.2.2.12.10.12" class="ltx_td ltx_align_right">48.0</td>
<td id="A1.T15.2.2.12.10.13" class="ltx_td ltx_align_right">40.8</td>
<td id="A1.T15.2.2.12.10.14" class="ltx_td ltx_align_right">33.1</td>
<td id="A1.T15.2.2.12.10.15" class="ltx_td ltx_align_right">34.4</td>
<td id="A1.T15.2.2.12.10.16" class="ltx_td ltx_align_right">37.2</td>
<td id="A1.T15.2.2.12.10.17" class="ltx_td ltx_align_right">35.8</td>
<td id="A1.T15.2.2.12.10.18" class="ltx_td ltx_align_right">54.3</td>
<td id="A1.T15.2.2.12.10.19" class="ltx_td ltx_align_right">49.2</td>
<td id="A1.T15.2.2.12.10.20" class="ltx_td ltx_align_right">41.6</td>
<td id="A1.T15.2.2.12.10.21" class="ltx_td ltx_align_right">41.4</td>
<td id="A1.T15.2.2.12.10.22" class="ltx_td ltx_align_right">44.2</td>
<td id="A1.T15.2.2.12.10.23" class="ltx_td ltx_align_right">40.9</td>
<td id="A1.T15.2.2.12.10.24" class="ltx_td ltx_align_right ltx_border_r">34.2</td>
<td id="A1.T15.2.2.12.10.25" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.12.10.25.1" class="ltx_text" style="background-color:#E6E6E6;">42.4</span></td>
</tr>
<tr id="A1.T15.2.2.13.11" class="ltx_tr">
<td id="A1.T15.2.2.13.11.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T15.2.2.13.11.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">FP16</td>
<td id="A1.T15.2.2.13.11.3" class="ltx_td ltx_align_right ltx_border_t">42.4</td>
<td id="A1.T15.2.2.13.11.4" class="ltx_td ltx_align_right ltx_border_t">42.0</td>
<td id="A1.T15.2.2.13.11.5" class="ltx_td ltx_align_right ltx_border_t">46.5</td>
<td id="A1.T15.2.2.13.11.6" class="ltx_td ltx_align_right ltx_border_t">38.7</td>
<td id="A1.T15.2.2.13.11.7" class="ltx_td ltx_align_right ltx_border_t">35.4</td>
<td id="A1.T15.2.2.13.11.8" class="ltx_td ltx_align_right ltx_border_t">36.5</td>
<td id="A1.T15.2.2.13.11.9" class="ltx_td ltx_align_right ltx_border_t">48.1</td>
<td id="A1.T15.2.2.13.11.10" class="ltx_td ltx_align_right ltx_border_t">43.7</td>
<td id="A1.T15.2.2.13.11.11" class="ltx_td ltx_align_right ltx_border_t">37.4</td>
<td id="A1.T15.2.2.13.11.12" class="ltx_td ltx_align_right ltx_border_t">45.5</td>
<td id="A1.T15.2.2.13.11.13" class="ltx_td ltx_align_right ltx_border_t">37.9</td>
<td id="A1.T15.2.2.13.11.14" class="ltx_td ltx_align_right ltx_border_t">29.9</td>
<td id="A1.T15.2.2.13.11.15" class="ltx_td ltx_align_right ltx_border_t">30.9</td>
<td id="A1.T15.2.2.13.11.16" class="ltx_td ltx_align_right ltx_border_t">35.8</td>
<td id="A1.T15.2.2.13.11.17" class="ltx_td ltx_align_right ltx_border_t">33.6</td>
<td id="A1.T15.2.2.13.11.18" class="ltx_td ltx_align_right ltx_border_t">51.7</td>
<td id="A1.T15.2.2.13.11.19" class="ltx_td ltx_align_right ltx_border_t">46.7</td>
<td id="A1.T15.2.2.13.11.20" class="ltx_td ltx_align_right ltx_border_t">38.6</td>
<td id="A1.T15.2.2.13.11.21" class="ltx_td ltx_align_right ltx_border_t">36.9</td>
<td id="A1.T15.2.2.13.11.22" class="ltx_td ltx_align_right ltx_border_t">41.2</td>
<td id="A1.T15.2.2.13.11.23" class="ltx_td ltx_align_right ltx_border_t">38.2</td>
<td id="A1.T15.2.2.13.11.24" class="ltx_td ltx_align_right ltx_border_r ltx_border_t">31.6</td>
<td id="A1.T15.2.2.13.11.25" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.13.11.25.1" class="ltx_text" style="background-color:#E6E6E6;">39.5</span></td>
</tr>
<tr id="A1.T15.2.2.14.12" class="ltx_tr">
<td id="A1.T15.2.2.14.12.1" class="ltx_td ltx_align_left">Aya-23-8b</td>
<td id="A1.T15.2.2.14.12.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T15.2.2.14.12.3" class="ltx_td ltx_align_right">42.1</td>
<td id="A1.T15.2.2.14.12.4" class="ltx_td ltx_align_right">42.5</td>
<td id="A1.T15.2.2.14.12.5" class="ltx_td ltx_align_right">46.7</td>
<td id="A1.T15.2.2.14.12.6" class="ltx_td ltx_align_right">39.2</td>
<td id="A1.T15.2.2.14.12.7" class="ltx_td ltx_align_right">35.5</td>
<td id="A1.T15.2.2.14.12.8" class="ltx_td ltx_align_right">36.8</td>
<td id="A1.T15.2.2.14.12.9" class="ltx_td ltx_align_right">48.1</td>
<td id="A1.T15.2.2.14.12.10" class="ltx_td ltx_align_right">44.2</td>
<td id="A1.T15.2.2.14.12.11" class="ltx_td ltx_align_right">37.7</td>
<td id="A1.T15.2.2.14.12.12" class="ltx_td ltx_align_right">45.5</td>
<td id="A1.T15.2.2.14.12.13" class="ltx_td ltx_align_right">38.2</td>
<td id="A1.T15.2.2.14.12.14" class="ltx_td ltx_align_right">30.0</td>
<td id="A1.T15.2.2.14.12.15" class="ltx_td ltx_align_right">31.3</td>
<td id="A1.T15.2.2.14.12.16" class="ltx_td ltx_align_right">35.6</td>
<td id="A1.T15.2.2.14.12.17" class="ltx_td ltx_align_right">33.7</td>
<td id="A1.T15.2.2.14.12.18" class="ltx_td ltx_align_right">52.0</td>
<td id="A1.T15.2.2.14.12.19" class="ltx_td ltx_align_right">46.6</td>
<td id="A1.T15.2.2.14.12.20" class="ltx_td ltx_align_right">38.5</td>
<td id="A1.T15.2.2.14.12.21" class="ltx_td ltx_align_right">37.0</td>
<td id="A1.T15.2.2.14.12.22" class="ltx_td ltx_align_right">41.6</td>
<td id="A1.T15.2.2.14.12.23" class="ltx_td ltx_align_right">38.4</td>
<td id="A1.T15.2.2.14.12.24" class="ltx_td ltx_align_right ltx_border_r">31.9</td>
<td id="A1.T15.2.2.14.12.25" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.14.12.25.1" class="ltx_text" style="background-color:#E6E6E6;">39.7</span></td>
</tr>
<tr id="A1.T15.2.2.15.13" class="ltx_tr">
<td id="A1.T15.2.2.15.13.1" class="ltx_td ltx_border_bb"></td>
<td id="A1.T15.2.2.15.13.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4</td>
<td id="A1.T15.2.2.15.13.3" class="ltx_td ltx_align_right ltx_border_bb">41.4</td>
<td id="A1.T15.2.2.15.13.4" class="ltx_td ltx_align_right ltx_border_bb">42.2</td>
<td id="A1.T15.2.2.15.13.5" class="ltx_td ltx_align_right ltx_border_bb">46.2</td>
<td id="A1.T15.2.2.15.13.6" class="ltx_td ltx_align_right ltx_border_bb">38.1</td>
<td id="A1.T15.2.2.15.13.7" class="ltx_td ltx_align_right ltx_border_bb">35.8</td>
<td id="A1.T15.2.2.15.13.8" class="ltx_td ltx_align_right ltx_border_bb">36.7</td>
<td id="A1.T15.2.2.15.13.9" class="ltx_td ltx_align_right ltx_border_bb">47.4</td>
<td id="A1.T15.2.2.15.13.10" class="ltx_td ltx_align_right ltx_border_bb">42.8</td>
<td id="A1.T15.2.2.15.13.11" class="ltx_td ltx_align_right ltx_border_bb">36.2</td>
<td id="A1.T15.2.2.15.13.12" class="ltx_td ltx_align_right ltx_border_bb">44.7</td>
<td id="A1.T15.2.2.15.13.13" class="ltx_td ltx_align_right ltx_border_bb">38.5</td>
<td id="A1.T15.2.2.15.13.14" class="ltx_td ltx_align_right ltx_border_bb">29.7</td>
<td id="A1.T15.2.2.15.13.15" class="ltx_td ltx_align_right ltx_border_bb">30.1</td>
<td id="A1.T15.2.2.15.13.16" class="ltx_td ltx_align_right ltx_border_bb">35.7</td>
<td id="A1.T15.2.2.15.13.17" class="ltx_td ltx_align_right ltx_border_bb">33.1</td>
<td id="A1.T15.2.2.15.13.18" class="ltx_td ltx_align_right ltx_border_bb">51.9</td>
<td id="A1.T15.2.2.15.13.19" class="ltx_td ltx_align_right ltx_border_bb">46.1</td>
<td id="A1.T15.2.2.15.13.20" class="ltx_td ltx_align_right ltx_border_bb">38.3</td>
<td id="A1.T15.2.2.15.13.21" class="ltx_td ltx_align_right ltx_border_bb">35.7</td>
<td id="A1.T15.2.2.15.13.22" class="ltx_td ltx_align_right ltx_border_bb">40.7</td>
<td id="A1.T15.2.2.15.13.23" class="ltx_td ltx_align_right ltx_border_bb">37.6</td>
<td id="A1.T15.2.2.15.13.24" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r">31.6</td>
<td id="A1.T15.2.2.15.13.25" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;"><span id="A1.T15.2.2.15.13.25.1" class="ltx_text" style="background-color:#E6E6E6;">39.1</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A15: </span><span id="A1.T15.4.1" class="ltx_text ltx_font_bold">Aya 23 language-specific results for FLORES.</span> spBLEU with FLORES200 tokenizer.</figcaption>
</figure>
<figure id="A1.T16" class="ltx_table">
<div id="A1.T16.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:79.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-323.2pt,59.3pt) scale(0.401458130413189,0.401458130413189) ;">
<table id="A1.T16.3.3" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T16.1.1.1" class="ltx_tr">
<td id="A1.T16.1.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="A1.T16.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" colspan="24"><span id="A1.T16.1.1.1.1.1" class="ltx_text ltx_font_bold">English<math id="A1.T16.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A1.T16.1.1.1.1.1.m1.1a"><mo stretchy="false" id="A1.T16.1.1.1.1.1.m1.1.1" xref="A1.T16.1.1.1.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A1.T16.1.1.1.1.1.m1.1b"><ci id="A1.T16.1.1.1.1.1.m1.1.1.cmml" xref="A1.T16.1.1.1.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T16.1.1.1.1.1.m1.1c">\rightarrow</annotation></semantics></math>L2</span></td>
<td id="A1.T16.1.1.1.3" class="ltx_td ltx_border_tt"></td>
<td id="A1.T16.1.1.1.4" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="A1.T16.2.2.2" class="ltx_tr">
<td id="A1.T16.2.2.2.2" class="ltx_td"></td>
<td id="A1.T16.2.2.2.3" class="ltx_td ltx_border_r"></td>
<td id="A1.T16.2.2.2.4" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.4.1" class="ltx_text ltx_font_bold">ar</span></td>
<td id="A1.T16.2.2.2.5" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.5.1" class="ltx_text ltx_font_bold">cs</span></td>
<td id="A1.T16.2.2.2.6" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.6.1" class="ltx_text ltx_font_bold">de</span></td>
<td id="A1.T16.2.2.2.7" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.7.1" class="ltx_text ltx_font_bold">el</span></td>
<td id="A1.T16.2.2.2.8" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.8.1" class="ltx_text ltx_font_bold">es</span></td>
<td id="A1.T16.2.2.2.9" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.9.1" class="ltx_text ltx_font_bold">fa</span></td>
<td id="A1.T16.2.2.2.10" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.10.1" class="ltx_text ltx_font_bold">fr</span></td>
<td id="A1.T16.2.2.2.11" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.11.1" class="ltx_text ltx_font_bold">he</span></td>
<td id="A1.T16.2.2.2.12" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.12.1" class="ltx_text ltx_font_bold">hi</span></td>
<td id="A1.T16.2.2.2.13" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.13.1" class="ltx_text ltx_font_bold">id</span></td>
<td id="A1.T16.2.2.2.14" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.14.1" class="ltx_text ltx_font_bold">it</span></td>
<td id="A1.T16.2.2.2.15" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.15.1" class="ltx_text ltx_font_bold">ja</span></td>
<td id="A1.T16.2.2.2.16" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.16.1" class="ltx_text ltx_font_bold">ko</span></td>
<td id="A1.T16.2.2.2.17" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.17.1" class="ltx_text ltx_font_bold">nl</span></td>
<td id="A1.T16.2.2.2.18" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.18.1" class="ltx_text ltx_font_bold">pl</span></td>
<td id="A1.T16.2.2.2.19" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.19.1" class="ltx_text ltx_font_bold">pt</span></td>
<td id="A1.T16.2.2.2.20" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.20.1" class="ltx_text ltx_font_bold">ro</span></td>
<td id="A1.T16.2.2.2.21" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.21.1" class="ltx_text ltx_font_bold">ru</span></td>
<td id="A1.T16.2.2.2.22" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.22.1" class="ltx_text ltx_font_bold">tr</span></td>
<td id="A1.T16.2.2.2.23" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.23.1" class="ltx_text ltx_font_bold">uk</span></td>
<td id="A1.T16.2.2.2.24" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.24.1" class="ltx_text ltx_font_bold">vi</span></td>
<td id="A1.T16.2.2.2.25" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T16.2.2.2.25.1" class="ltx_text ltx_font_bold">zh</span></td>
<td id="A1.T16.2.2.2.26" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.26.1" class="ltx_text ltx_font_bold">Avg</span></td>
<td id="A1.T16.2.2.2.27" class="ltx_td ltx_align_center"><span id="A1.T16.2.2.2.27.1" class="ltx_text ltx_font_bold">Ltn</span></td>
<td id="A1.T16.2.2.2.1" class="ltx_td ltx_align_center"><math id="A1.T16.2.2.2.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T16.2.2.2.1.m1.1a"><mo id="A1.T16.2.2.2.1.m1.1.1" xref="A1.T16.2.2.2.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T16.2.2.2.1.m1.1b"><not id="A1.T16.2.2.2.1.m1.1.1.cmml" xref="A1.T16.2.2.2.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T16.2.2.2.1.m1.1c">\neg</annotation></semantics></math></td>
</tr>
<tr id="A1.T16.3.3.4.1" class="ltx_tr">
<td id="A1.T16.3.3.4.1.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T16.3.3.4.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T16.3.3.4.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="A1.T16.3.3.4.1.3.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T16.3.3.4.1.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="A1.T16.3.3.4.1.4.1" class="ltx_text" style="background-color:#FEFBFB;">-0.2%</span></td>
<td id="A1.T16.3.3.4.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#A6DBC2;"><span id="A1.T16.3.3.4.1.5.1" class="ltx_text" style="background-color:#A6DBC2;">1.1%</span></td>
<td id="A1.T16.3.3.4.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="A1.T16.3.3.4.1.6.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T16.3.3.4.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F0F9F5;"><span id="A1.T16.3.3.4.1.7.1" class="ltx_text" style="background-color:#F0F9F5;">0.2%</span></td>
<td id="A1.T16.3.3.4.1.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#C3E7D6;"><span id="A1.T16.3.3.4.1.8.1" class="ltx_text" style="background-color:#C3E7D6;">0.7%</span></td>
<td id="A1.T16.3.3.4.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F7;"><span id="A1.T16.3.3.4.1.9.1" class="ltx_text" style="background-color:#FDF7F7;">-0.4%</span></td>
<td id="A1.T16.3.3.4.1.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#99D6B8;"><span id="A1.T16.3.3.4.1.10.1" class="ltx_text" style="background-color:#99D6B8;">1.2%</span></td>
<td id="A1.T16.3.3.4.1.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#91D3B2;"><span id="A1.T16.3.3.4.1.11.1" class="ltx_text" style="background-color:#91D3B2;">1.3%</span></td>
<td id="A1.T16.3.3.4.1.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E3F4EC;"><span id="A1.T16.3.3.4.1.12.1" class="ltx_text" style="background-color:#E3F4EC;">0.3%</span></td>
<td id="A1.T16.3.3.4.1.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF8F7;"><span id="A1.T16.3.3.4.1.13.1" class="ltx_text" style="background-color:#FDF8F7;">-0.4%</span></td>
<td id="A1.T16.3.3.4.1.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D1EDDF;"><span id="A1.T16.3.3.4.1.14.1" class="ltx_text" style="background-color:#D1EDDF;">0.6%</span></td>
<td id="A1.T16.3.3.4.1.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCFEFD;"><span id="A1.T16.3.3.4.1.15.1" class="ltx_text" style="background-color:#FCFEFD;">0.0%</span></td>
<td id="A1.T16.3.3.4.1.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#CDEBDC;"><span id="A1.T16.3.3.4.1.16.1" class="ltx_text" style="background-color:#CDEBDC;">0.6%</span></td>
<td id="A1.T16.3.3.4.1.17" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#C6E8D7;"><span id="A1.T16.3.3.4.1.17.1" class="ltx_text" style="background-color:#C6E8D7;">0.7%</span></td>
<td id="A1.T16.3.3.4.1.18" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E9F7F0;"><span id="A1.T16.3.3.4.1.18.1" class="ltx_text" style="background-color:#E9F7F0;">0.3%</span></td>
<td id="A1.T16.3.3.4.1.19" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#DEF2E9;"><span id="A1.T16.3.3.4.1.19.1" class="ltx_text" style="background-color:#DEF2E9;">0.4%</span></td>
<td id="A1.T16.3.3.4.1.20" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F3;"><span id="A1.T16.3.3.4.1.20.1" class="ltx_text" style="background-color:#FCF3F3;">-0.6%</span></td>
<td id="A1.T16.3.3.4.1.21" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F6FBF9;"><span id="A1.T16.3.3.4.1.21.1" class="ltx_text" style="background-color:#F6FBF9;">0.1%</span></td>
<td id="A1.T16.3.3.4.1.22" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#ECF7F2;"><span id="A1.T16.3.3.4.1.22.1" class="ltx_text" style="background-color:#ECF7F2;">0.2%</span></td>
<td id="A1.T16.3.3.4.1.23" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#B2E0C9;"><span id="A1.T16.3.3.4.1.23.1" class="ltx_text" style="background-color:#B2E0C9;">0.9%</span></td>
<td id="A1.T16.3.3.4.1.24" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#FCF0EF;"><span id="A1.T16.3.3.4.1.24.1" class="ltx_text" style="background-color:#FCF0EF;">-0.8%</span></td>
<td id="A1.T16.3.3.4.1.25" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E8F6EF;"><span id="A1.T16.3.3.4.1.25.1" class="ltx_text" style="background-color:#E8F6EF;">0.3%</span></td>
<td id="A1.T16.3.3.4.1.26" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E6F5EE;"><span id="A1.T16.3.3.4.1.26.1" class="ltx_text" style="background-color:#E6F5EE;">0.3%</span></td>
<td id="A1.T16.3.3.4.1.27" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#EAF7F0;"><span id="A1.T16.3.3.4.1.27.1" class="ltx_text" style="background-color:#EAF7F0;">0.3%</span></td>
</tr>
<tr id="A1.T16.3.3.5.2" class="ltx_tr">
<td id="A1.T16.3.3.5.2.1" class="ltx_td ltx_align_left"><span id="A1.T16.3.3.5.2.1.1" class="ltx_text">Aya-23-35b</span></td>
<td id="A1.T16.3.3.5.2.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T16.3.3.5.2.3" class="ltx_td ltx_align_right" style="background-color:#F8DEDC;"><span id="A1.T16.3.3.5.2.3.1" class="ltx_text" style="background-color:#F8DEDC;">-1.7%</span></td>
<td id="A1.T16.3.3.5.2.4" class="ltx_td ltx_align_right" style="background-color:#F5CCC8;"><span id="A1.T16.3.3.5.2.4.1" class="ltx_text" style="background-color:#F5CCC8;">-2.7%</span></td>
<td id="A1.T16.3.3.5.2.5" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T16.3.3.5.2.5.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T16.3.3.5.2.6" class="ltx_td ltx_align_right" style="background-color:#FCF2F1;"><span id="A1.T16.3.3.5.2.6.1" class="ltx_text" style="background-color:#FCF2F1;">-0.7%</span></td>
<td id="A1.T16.3.3.5.2.7" class="ltx_td ltx_align_right" style="background-color:#FDF7F6;"><span id="A1.T16.3.3.5.2.7.1" class="ltx_text" style="background-color:#FDF7F6;">-0.4%</span></td>
<td id="A1.T16.3.3.5.2.8" class="ltx_td ltx_align_right" style="background-color:#F6D1CE;"><span id="A1.T16.3.3.5.2.8.1" class="ltx_text" style="background-color:#F6D1CE;">-2.4%</span></td>
<td id="A1.T16.3.3.5.2.9" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="A1.T16.3.3.5.2.9.1" class="ltx_text" style="background-color:#F9E4E2;">-1.4%</span></td>
<td id="A1.T16.3.3.5.2.10" class="ltx_td ltx_align_right" style="background-color:#FBECEB;"><span id="A1.T16.3.3.5.2.10.1" class="ltx_text" style="background-color:#FBECEB;">-1.0%</span></td>
<td id="A1.T16.3.3.5.2.11" class="ltx_td ltx_align_right" style="background-color:#F7D8D6;"><span id="A1.T16.3.3.5.2.11.1" class="ltx_text" style="background-color:#F7D8D6;">-2.0%</span></td>
<td id="A1.T16.3.3.5.2.12" class="ltx_td ltx_align_right" style="background-color:#FEFAFA;"><span id="A1.T16.3.3.5.2.12.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
<td id="A1.T16.3.3.5.2.13" class="ltx_td ltx_align_right" style="background-color:#F9E3E1;"><span id="A1.T16.3.3.5.2.13.1" class="ltx_text" style="background-color:#F9E3E1;">-1.5%</span></td>
<td id="A1.T16.3.3.5.2.14" class="ltx_td ltx_align_right" style="background-color:#F7D9D7;"><span id="A1.T16.3.3.5.2.14.1" class="ltx_text" style="background-color:#F7D9D7;">-2.0%</span></td>
<td id="A1.T16.3.3.5.2.15" class="ltx_td ltx_align_right" style="background-color:#F9E3E1;"><span id="A1.T16.3.3.5.2.15.1" class="ltx_text" style="background-color:#F9E3E1;">-1.5%</span></td>
<td id="A1.T16.3.3.5.2.16" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="A1.T16.3.3.5.2.16.1" class="ltx_text" style="background-color:#F9E4E2;">-1.4%</span></td>
<td id="A1.T16.3.3.5.2.17" class="ltx_td ltx_align_right" style="background-color:#F6D4D2;"><span id="A1.T16.3.3.5.2.17.1" class="ltx_text" style="background-color:#F6D4D2;">-2.2%</span></td>
<td id="A1.T16.3.3.5.2.18" class="ltx_td ltx_align_right" style="background-color:#FDF5F4;"><span id="A1.T16.3.3.5.2.18.1" class="ltx_text" style="background-color:#FDF5F4;">-0.5%</span></td>
<td id="A1.T16.3.3.5.2.19" class="ltx_td ltx_align_right" style="background-color:#F9E0DE;"><span id="A1.T16.3.3.5.2.19.1" class="ltx_text" style="background-color:#F9E0DE;">-1.6%</span></td>
<td id="A1.T16.3.3.5.2.20" class="ltx_td ltx_align_right" style="background-color:#F9E1DF;"><span id="A1.T16.3.3.5.2.20.1" class="ltx_text" style="background-color:#F9E1DF;">-1.6%</span></td>
<td id="A1.T16.3.3.5.2.21" class="ltx_td ltx_align_right" style="background-color:#F8DDDA;"><span id="A1.T16.3.3.5.2.21.1" class="ltx_text" style="background-color:#F8DDDA;">-1.8%</span></td>
<td id="A1.T16.3.3.5.2.22" class="ltx_td ltx_align_right" style="background-color:#FAE9E8;"><span id="A1.T16.3.3.5.2.22.1" class="ltx_text" style="background-color:#FAE9E8;">-1.1%</span></td>
<td id="A1.T16.3.3.5.2.23" class="ltx_td ltx_align_right" style="background-color:#FDF4F4;"><span id="A1.T16.3.3.5.2.23.1" class="ltx_text" style="background-color:#FDF4F4;">-0.5%</span></td>
<td id="A1.T16.3.3.5.2.24" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F5CCC9;"><span id="A1.T16.3.3.5.2.24.1" class="ltx_text" style="background-color:#F5CCC9;">-2.7%</span></td>
<td id="A1.T16.3.3.5.2.25" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="A1.T16.3.3.5.2.25.1" class="ltx_text" style="background-color:#F9E4E2;">-1.4%</span></td>
<td id="A1.T16.3.3.5.2.26" class="ltx_td ltx_align_right" style="background-color:#FAE8E7;"><span id="A1.T16.3.3.5.2.26.1" class="ltx_text" style="background-color:#FAE8E7;">-1.2%</span></td>
<td id="A1.T16.3.3.5.2.27" class="ltx_td ltx_align_right" style="background-color:#F9DFDD;"><span id="A1.T16.3.3.5.2.27.1" class="ltx_text" style="background-color:#F9DFDD;">-1.7%</span></td>
</tr>
<tr id="A1.T16.3.3.6.3" class="ltx_tr">
<td id="A1.T16.3.3.6.3.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T16.3.3.6.3.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T16.3.3.6.3.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D8F0E4;"><span id="A1.T16.3.3.6.3.3.1" class="ltx_text" style="background-color:#D8F0E4;">0.5%</span></td>
<td id="A1.T16.3.3.6.3.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#9FD8BC;"><span id="A1.T16.3.3.6.3.4.1" class="ltx_text" style="background-color:#9FD8BC;">1.1%</span></td>
<td id="A1.T16.3.3.6.3.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#CAEADA;"><span id="A1.T16.3.3.6.3.5.1" class="ltx_text" style="background-color:#CAEADA;">0.6%</span></td>
<td id="A1.T16.3.3.6.3.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAF9;"><span id="A1.T16.3.3.6.3.6.1" class="ltx_text" style="background-color:#FEFAF9;">-0.3%</span></td>
<td id="A1.T16.3.3.6.3.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F7;"><span id="A1.T16.3.3.6.3.7.1" class="ltx_text" style="background-color:#FDF7F7;">-0.4%</span></td>
<td id="A1.T16.3.3.6.3.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#90D2B2;"><span id="A1.T16.3.3.6.3.8.1" class="ltx_text" style="background-color:#90D2B2;">1.3%</span></td>
<td id="A1.T16.3.3.6.3.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#BEE5D2;"><span id="A1.T16.3.3.6.3.9.1" class="ltx_text" style="background-color:#BEE5D2;">0.8%</span></td>
<td id="A1.T16.3.3.6.3.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFDFD;"><span id="A1.T16.3.3.6.3.10.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T16.3.3.6.3.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEEED;"><span id="A1.T16.3.3.6.3.11.1" class="ltx_text" style="background-color:#FBEEED;">-0.9%</span></td>
<td id="A1.T16.3.3.6.3.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F9;"><span id="A1.T16.3.3.6.3.12.1" class="ltx_text" style="background-color:#FDF9F9;">-0.3%</span></td>
<td id="A1.T16.3.3.6.3.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F1FAF5;"><span id="A1.T16.3.3.6.3.13.1" class="ltx_text" style="background-color:#F1FAF5;">0.2%</span></td>
<td id="A1.T16.3.3.6.3.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFA;"><span id="A1.T16.3.3.6.3.14.1" class="ltx_text" style="background-color:#FEFBFA;">-0.2%</span></td>
<td id="A1.T16.3.3.6.3.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#ABDDC5;"><span id="A1.T16.3.3.6.3.15.1" class="ltx_text" style="background-color:#ABDDC5;">1.0%</span></td>
<td id="A1.T16.3.3.6.3.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T16.3.3.6.3.16.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T16.3.3.6.3.17" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F1FAF5;"><span id="A1.T16.3.3.6.3.17.1" class="ltx_text" style="background-color:#F1FAF5;">0.2%</span></td>
<td id="A1.T16.3.3.6.3.18" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF5F5;"><span id="A1.T16.3.3.6.3.18.1" class="ltx_text" style="background-color:#FDF5F5;">-0.5%</span></td>
<td id="A1.T16.3.3.6.3.19" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#B7E2CD;"><span id="A1.T16.3.3.6.3.19.1" class="ltx_text" style="background-color:#B7E2CD;">0.9%</span></td>
<td id="A1.T16.3.3.6.3.20" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T16.3.3.6.3.20.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T16.3.3.6.3.21" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF2F1;"><span id="A1.T16.3.3.6.3.21.1" class="ltx_text" style="background-color:#FCF2F1;">-0.7%</span></td>
<td id="A1.T16.3.3.6.3.22" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDFFFE;"><span id="A1.T16.3.3.6.3.22.1" class="ltx_text" style="background-color:#FDFFFE;">0.0%</span></td>
<td id="A1.T16.3.3.6.3.23" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDFFFE;"><span id="A1.T16.3.3.6.3.23.1" class="ltx_text" style="background-color:#FDFFFE;">0.0%</span></td>
<td id="A1.T16.3.3.6.3.24" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#74C79E;"><span id="A1.T16.3.3.6.3.24.1" class="ltx_text" style="background-color:#74C79E;">1.7%</span></td>
<td id="A1.T16.3.3.6.3.25" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#EDF8F3;"><span id="A1.T16.3.3.6.3.25.1" class="ltx_text" style="background-color:#EDF8F3;">0.2%</span></td>
<td id="A1.T16.3.3.6.3.26" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F3FAF7;"><span id="A1.T16.3.3.6.3.26.1" class="ltx_text" style="background-color:#F3FAF7;">0.2%</span></td>
<td id="A1.T16.3.3.6.3.27" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E7F5EE;"><span id="A1.T16.3.3.6.3.27.1" class="ltx_text" style="background-color:#E7F5EE;">0.3%</span></td>
</tr>
<tr id="A1.T16.3.3.7.4" class="ltx_tr">
<td id="A1.T16.3.3.7.4.1" class="ltx_td ltx_align_left"><span id="A1.T16.3.3.7.4.1.1" class="ltx_text">Aya-23-8b</span></td>
<td id="A1.T16.3.3.7.4.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T16.3.3.7.4.3" class="ltx_td ltx_align_right" style="background-color:#F5CFCC;"><span id="A1.T16.3.3.7.4.3.1" class="ltx_text" style="background-color:#F5CFCC;">-2.5%</span></td>
<td id="A1.T16.3.3.7.4.4" class="ltx_td ltx_align_right" style="background-color:#F7D9D7;"><span id="A1.T16.3.3.7.4.4.1" class="ltx_text" style="background-color:#F7D9D7;">-2.0%</span></td>
<td id="A1.T16.3.3.7.4.5" class="ltx_td ltx_align_right" style="background-color:#FDF8F8;"><span id="A1.T16.3.3.7.4.5.1" class="ltx_text" style="background-color:#FDF8F8;">-0.3%</span></td>
<td id="A1.T16.3.3.7.4.6" class="ltx_td ltx_align_right" style="background-color:#F8DFDC;"><span id="A1.T16.3.3.7.4.6.1" class="ltx_text" style="background-color:#F8DFDC;">-1.7%</span></td>
<td id="A1.T16.3.3.7.4.7" class="ltx_td ltx_align_right" style="background-color:#FBEDEB;"><span id="A1.T16.3.3.7.4.7.1" class="ltx_text" style="background-color:#FBEDEB;">-1.0%</span></td>
<td id="A1.T16.3.3.7.4.8" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="A1.T16.3.3.7.4.8.1" class="ltx_text" style="background-color:#F9E4E2;">-1.4%</span></td>
<td id="A1.T16.3.3.7.4.9" class="ltx_td ltx_align_right" style="background-color:#F9E1DF;"><span id="A1.T16.3.3.7.4.9.1" class="ltx_text" style="background-color:#F9E1DF;">-1.6%</span></td>
<td id="A1.T16.3.3.7.4.10" class="ltx_td ltx_align_right" style="background-color:#ECA099;"><span id="A1.T16.3.3.7.4.10.1" class="ltx_text" style="background-color:#ECA099;">-5.1%</span></td>
<td id="A1.T16.3.3.7.4.11" class="ltx_td ltx_align_right" style="background-color:#F3C5C1;"><span id="A1.T16.3.3.7.4.11.1" class="ltx_text" style="background-color:#F3C5C1;">-3.1%</span></td>
<td id="A1.T16.3.3.7.4.12" class="ltx_td ltx_align_right" style="background-color:#FAE7E6;"><span id="A1.T16.3.3.7.4.12.1" class="ltx_text" style="background-color:#FAE7E6;">-1.2%</span></td>
<td id="A1.T16.3.3.7.4.13" class="ltx_td ltx_align_right" style="background-color:#FAE6E5;"><span id="A1.T16.3.3.7.4.13.1" class="ltx_text" style="background-color:#FAE6E5;">-1.3%</span></td>
<td id="A1.T16.3.3.7.4.14" class="ltx_td ltx_align_right" style="background-color:#F7D7D4;"><span id="A1.T16.3.3.7.4.14.1" class="ltx_text" style="background-color:#F7D7D4;">-2.1%</span></td>
<td id="A1.T16.3.3.7.4.15" class="ltx_td ltx_align_right" style="background-color:#F8DAD7;"><span id="A1.T16.3.3.7.4.15.1" class="ltx_text" style="background-color:#F8DAD7;">-2.0%</span></td>
<td id="A1.T16.3.3.7.4.16" class="ltx_td ltx_align_right" style="background-color:#FBEBE9;"><span id="A1.T16.3.3.7.4.16.1" class="ltx_text" style="background-color:#FBEBE9;">-1.1%</span></td>
<td id="A1.T16.3.3.7.4.17" class="ltx_td ltx_align_right" style="background-color:#FAE5E4;"><span id="A1.T16.3.3.7.4.17.1" class="ltx_text" style="background-color:#FAE5E4;">-1.3%</span></td>
<td id="A1.T16.3.3.7.4.18" class="ltx_td ltx_align_right" style="background-color:#FBEEED;"><span id="A1.T16.3.3.7.4.18.1" class="ltx_text" style="background-color:#FBEEED;">-0.9%</span></td>
<td id="A1.T16.3.3.7.4.19" class="ltx_td ltx_align_right" style="background-color:#F8DEDB;"><span id="A1.T16.3.3.7.4.19.1" class="ltx_text" style="background-color:#F8DEDB;">-1.8%</span></td>
<td id="A1.T16.3.3.7.4.20" class="ltx_td ltx_align_right" style="background-color:#F7D7D4;"><span id="A1.T16.3.3.7.4.20.1" class="ltx_text" style="background-color:#F7D7D4;">-2.1%</span></td>
<td id="A1.T16.3.3.7.4.21" class="ltx_td ltx_align_right" style="background-color:#F2BBB6;"><span id="A1.T16.3.3.7.4.21.1" class="ltx_text" style="background-color:#F2BBB6;">-3.6%</span></td>
<td id="A1.T16.3.3.7.4.22" class="ltx_td ltx_align_right" style="background-color:#F8DFDD;"><span id="A1.T16.3.3.7.4.22.1" class="ltx_text" style="background-color:#F8DFDD;">-1.7%</span></td>
<td id="A1.T16.3.3.7.4.23" class="ltx_td ltx_align_right" style="background-color:#FBEAE8;"><span id="A1.T16.3.3.7.4.23.1" class="ltx_text" style="background-color:#FBEAE8;">-1.1%</span></td>
<td id="A1.T16.3.3.7.4.24" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F7D6D3;"><span id="A1.T16.3.3.7.4.24.1" class="ltx_text" style="background-color:#F7D6D3;">-2.2%</span></td>
<td id="A1.T16.3.3.7.4.25" class="ltx_td ltx_align_right" style="background-color:#F8DCD9;"><span id="A1.T16.3.3.7.4.25.1" class="ltx_text" style="background-color:#F8DCD9;">-1.9%</span></td>
<td id="A1.T16.3.3.7.4.26" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="A1.T16.3.3.7.4.26.1" class="ltx_text" style="background-color:#F9E4E2;">-1.4%</span></td>
<td id="A1.T16.3.3.7.4.27" class="ltx_td ltx_align_right" style="background-color:#F6D2CF;"><span id="A1.T16.3.3.7.4.27.1" class="ltx_text" style="background-color:#F6D2CF;">-2.4%</span></td>
</tr>
<tr id="A1.T16.3.3.3" class="ltx_tr">
<td id="A1.T16.3.3.3.2" class="ltx_td ltx_border_tt"></td>
<td id="A1.T16.3.3.3.1" class="ltx_td ltx_align_center ltx_border_tt" colspan="24"><span id="A1.T16.3.3.3.1.1" class="ltx_text ltx_font_bold">L2<math id="A1.T16.3.3.3.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="A1.T16.3.3.3.1.1.m1.1a"><mo stretchy="false" id="A1.T16.3.3.3.1.1.m1.1.1" xref="A1.T16.3.3.3.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="A1.T16.3.3.3.1.1.m1.1b"><ci id="A1.T16.3.3.3.1.1.m1.1.1.cmml" xref="A1.T16.3.3.3.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T16.3.3.3.1.1.m1.1c">\rightarrow</annotation></semantics></math>English</span></td>
<td id="A1.T16.3.3.3.3" class="ltx_td ltx_border_tt"></td>
<td id="A1.T16.3.3.3.4" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="A1.T16.3.3.8.5" class="ltx_tr">
<td id="A1.T16.3.3.8.5.1" class="ltx_td"></td>
<td id="A1.T16.3.3.8.5.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T16.3.3.8.5.3" class="ltx_td ltx_align_right" style="background-color:#FEFEFE;"><span id="A1.T16.3.3.8.5.3.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="A1.T16.3.3.8.5.4" class="ltx_td ltx_align_right" style="background-color:#E6F5ED;"><span id="A1.T16.3.3.8.5.4.1" class="ltx_text" style="background-color:#E6F5ED;">0.3%</span></td>
<td id="A1.T16.3.3.8.5.5" class="ltx_td ltx_align_right" style="background-color:#EDF8F2;"><span id="A1.T16.3.3.8.5.5.1" class="ltx_text" style="background-color:#EDF8F2;">0.2%</span></td>
<td id="A1.T16.3.3.8.5.6" class="ltx_td ltx_align_right" style="background-color:#FDF5F5;"><span id="A1.T16.3.3.8.5.6.1" class="ltx_text" style="background-color:#FDF5F5;">-0.5%</span></td>
<td id="A1.T16.3.3.8.5.7" class="ltx_td ltx_align_right" style="background-color:#FBEAE8;"><span id="A1.T16.3.3.8.5.7.1" class="ltx_text" style="background-color:#FBEAE8;">-1.1%</span></td>
<td id="A1.T16.3.3.8.5.8" class="ltx_td ltx_align_right" style="background-color:#E3F4EC;"><span id="A1.T16.3.3.8.5.8.1" class="ltx_text" style="background-color:#E3F4EC;">0.3%</span></td>
<td id="A1.T16.3.3.8.5.9" class="ltx_td ltx_align_right" style="background-color:#E7F5EE;"><span id="A1.T16.3.3.8.5.9.1" class="ltx_text" style="background-color:#E7F5EE;">0.3%</span></td>
<td id="A1.T16.3.3.8.5.10" class="ltx_td ltx_align_right" style="background-color:#C8E9D9;"><span id="A1.T16.3.3.8.5.10.1" class="ltx_text" style="background-color:#C8E9D9;">0.7%</span></td>
<td id="A1.T16.3.3.8.5.11" class="ltx_td ltx_align_right" style="background-color:#E2F4EB;"><span id="A1.T16.3.3.8.5.11.1" class="ltx_text" style="background-color:#E2F4EB;">0.4%</span></td>
<td id="A1.T16.3.3.8.5.12" class="ltx_td ltx_align_right" style="background-color:#E7F6EF;"><span id="A1.T16.3.3.8.5.12.1" class="ltx_text" style="background-color:#E7F6EF;">0.3%</span></td>
<td id="A1.T16.3.3.8.5.13" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T16.3.3.8.5.13.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T16.3.3.8.5.14" class="ltx_td ltx_align_right" style="background-color:#BAE3CF;"><span id="A1.T16.3.3.8.5.14.1" class="ltx_text" style="background-color:#BAE3CF;">0.8%</span></td>
<td id="A1.T16.3.3.8.5.15" class="ltx_td ltx_align_right" style="background-color:#FCF2F1;"><span id="A1.T16.3.3.8.5.15.1" class="ltx_text" style="background-color:#FCF2F1;">-0.7%</span></td>
<td id="A1.T16.3.3.8.5.16" class="ltx_td ltx_align_right" style="background-color:#FDF5F4;"><span id="A1.T16.3.3.8.5.16.1" class="ltx_text" style="background-color:#FDF5F4;">-0.5%</span></td>
<td id="A1.T16.3.3.8.5.17" class="ltx_td ltx_align_right" style="background-color:#F4FBF7;"><span id="A1.T16.3.3.8.5.17.1" class="ltx_text" style="background-color:#F4FBF7;">0.1%</span></td>
<td id="A1.T16.3.3.8.5.18" class="ltx_td ltx_align_right" style="background-color:#FEFDFD;"><span id="A1.T16.3.3.8.5.18.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T16.3.3.8.5.19" class="ltx_td ltx_align_right" style="background-color:#FEFDFD;"><span id="A1.T16.3.3.8.5.19.1" class="ltx_text" style="background-color:#FEFDFD;">-0.1%</span></td>
<td id="A1.T16.3.3.8.5.20" class="ltx_td ltx_align_right" style="background-color:#E5F5ED;"><span id="A1.T16.3.3.8.5.20.1" class="ltx_text" style="background-color:#E5F5ED;">0.3%</span></td>
<td id="A1.T16.3.3.8.5.21" class="ltx_td ltx_align_right" style="background-color:#FEFCFC;"><span id="A1.T16.3.3.8.5.21.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T16.3.3.8.5.22" class="ltx_td ltx_align_right" style="background-color:#E0F3E9;"><span id="A1.T16.3.3.8.5.22.1" class="ltx_text" style="background-color:#E0F3E9;">0.4%</span></td>
<td id="A1.T16.3.3.8.5.23" class="ltx_td ltx_align_right" style="background-color:#EDF8F3;"><span id="A1.T16.3.3.8.5.23.1" class="ltx_text" style="background-color:#EDF8F3;">0.2%</span></td>
<td id="A1.T16.3.3.8.5.24" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F1FAF5;"><span id="A1.T16.3.3.8.5.24.1" class="ltx_text" style="background-color:#F1FAF5;">0.2%</span></td>
<td id="A1.T16.3.3.8.5.25" class="ltx_td ltx_align_right" style="background-color:#FAFDFC;"><span id="A1.T16.3.3.8.5.25.1" class="ltx_text" style="background-color:#FAFDFC;">0.1%</span></td>
<td id="A1.T16.3.3.8.5.26" class="ltx_td ltx_align_right" style="background-color:#FEFEFE;"><span id="A1.T16.3.3.8.5.26.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="A1.T16.3.3.8.5.27" class="ltx_td ltx_align_right" style="background-color:#F0F9F5;"><span id="A1.T16.3.3.8.5.27.1" class="ltx_text" style="background-color:#F0F9F5;">0.2%</span></td>
</tr>
<tr id="A1.T16.3.3.9.6" class="ltx_tr">
<td id="A1.T16.3.3.9.6.1" class="ltx_td ltx_align_left"><span id="A1.T16.3.3.9.6.1.1" class="ltx_text">Aya-23-35b</span></td>
<td id="A1.T16.3.3.9.6.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T16.3.3.9.6.3" class="ltx_td ltx_align_right" style="background-color:#F9E1DF;"><span id="A1.T16.3.3.9.6.3.1" class="ltx_text" style="background-color:#F9E1DF;">-1.6%</span></td>
<td id="A1.T16.3.3.9.6.4" class="ltx_td ltx_align_right" style="background-color:#FBEEEC;"><span id="A1.T16.3.3.9.6.4.1" class="ltx_text" style="background-color:#FBEEEC;">-0.9%</span></td>
<td id="A1.T16.3.3.9.6.5" class="ltx_td ltx_align_right" style="background-color:#FBECEB;"><span id="A1.T16.3.3.9.6.5.1" class="ltx_text" style="background-color:#FBECEB;">-1.0%</span></td>
<td id="A1.T16.3.3.9.6.6" class="ltx_td ltx_align_right" style="background-color:#FAE4E3;"><span id="A1.T16.3.3.9.6.6.1" class="ltx_text" style="background-color:#FAE4E3;">-1.4%</span></td>
<td id="A1.T16.3.3.9.6.7" class="ltx_td ltx_align_right" style="background-color:#FDF8F7;"><span id="A1.T16.3.3.9.6.7.1" class="ltx_text" style="background-color:#FDF8F7;">-0.4%</span></td>
<td id="A1.T16.3.3.9.6.8" class="ltx_td ltx_align_right" style="background-color:#F7DAD7;"><span id="A1.T16.3.3.9.6.8.1" class="ltx_text" style="background-color:#F7DAD7;">-2.0%</span></td>
<td id="A1.T16.3.3.9.6.9" class="ltx_td ltx_align_right" style="background-color:#FDF8F7;"><span id="A1.T16.3.3.9.6.9.1" class="ltx_text" style="background-color:#FDF8F7;">-0.4%</span></td>
<td id="A1.T16.3.3.9.6.10" class="ltx_td ltx_align_right" style="background-color:#F7D9D6;"><span id="A1.T16.3.3.9.6.10.1" class="ltx_text" style="background-color:#F7D9D6;">-2.0%</span></td>
<td id="A1.T16.3.3.9.6.11" class="ltx_td ltx_align_right" style="background-color:#F7D6D3;"><span id="A1.T16.3.3.9.6.11.1" class="ltx_text" style="background-color:#F7D6D3;">-2.2%</span></td>
<td id="A1.T16.3.3.9.6.12" class="ltx_td ltx_align_right" style="background-color:#FAE9E8;"><span id="A1.T16.3.3.9.6.12.1" class="ltx_text" style="background-color:#FAE9E8;">-1.1%</span></td>
<td id="A1.T16.3.3.9.6.13" class="ltx_td ltx_align_right" style="background-color:#BFE5D3;"><span id="A1.T16.3.3.9.6.13.1" class="ltx_text" style="background-color:#BFE5D3;">0.8%</span></td>
<td id="A1.T16.3.3.9.6.14" class="ltx_td ltx_align_right" style="background-color:#F9E0DE;"><span id="A1.T16.3.3.9.6.14.1" class="ltx_text" style="background-color:#F9E0DE;">-1.6%</span></td>
<td id="A1.T16.3.3.9.6.15" class="ltx_td ltx_align_right" style="background-color:#F5CDC9;"><span id="A1.T16.3.3.9.6.15.1" class="ltx_text" style="background-color:#F5CDC9;">-2.7%</span></td>
<td id="A1.T16.3.3.9.6.16" class="ltx_td ltx_align_right" style="background-color:#F9E3E1;"><span id="A1.T16.3.3.9.6.16.1" class="ltx_text" style="background-color:#F9E3E1;">-1.5%</span></td>
<td id="A1.T16.3.3.9.6.17" class="ltx_td ltx_align_right" style="background-color:#F9E3E1;"><span id="A1.T16.3.3.9.6.17.1" class="ltx_text" style="background-color:#F9E3E1;">-1.5%</span></td>
<td id="A1.T16.3.3.9.6.18" class="ltx_td ltx_align_right" style="background-color:#FBEDEC;"><span id="A1.T16.3.3.9.6.18.1" class="ltx_text" style="background-color:#FBEDEC;">-0.9%</span></td>
<td id="A1.T16.3.3.9.6.19" class="ltx_td ltx_align_right" style="background-color:#FDF5F5;"><span id="A1.T16.3.3.9.6.19.1" class="ltx_text" style="background-color:#FDF5F5;">-0.5%</span></td>
<td id="A1.T16.3.3.9.6.20" class="ltx_td ltx_align_right" style="background-color:#F7FCFA;"><span id="A1.T16.3.3.9.6.20.1" class="ltx_text" style="background-color:#F7FCFA;">0.1%</span></td>
<td id="A1.T16.3.3.9.6.21" class="ltx_td ltx_align_right" style="background-color:#F7DAD7;"><span id="A1.T16.3.3.9.6.21.1" class="ltx_text" style="background-color:#F7DAD7;">-2.0%</span></td>
<td id="A1.T16.3.3.9.6.22" class="ltx_td ltx_align_right" style="background-color:#F9E3E1;"><span id="A1.T16.3.3.9.6.22.1" class="ltx_text" style="background-color:#F9E3E1;">-1.5%</span></td>
<td id="A1.T16.3.3.9.6.23" class="ltx_td ltx_align_right" style="background-color:#FAE5E3;"><span id="A1.T16.3.3.9.6.23.1" class="ltx_text" style="background-color:#FAE5E3;">-1.4%</span></td>
<td id="A1.T16.3.3.9.6.24" class="ltx_td ltx_align_right ltx_border_rr" style="background-color:#F8DAD7;"><span id="A1.T16.3.3.9.6.24.1" class="ltx_text" style="background-color:#F8DAD7;">-2.0%</span></td>
<td id="A1.T16.3.3.9.6.25" class="ltx_td ltx_align_right" style="background-color:#FAE7E6;"><span id="A1.T16.3.3.9.6.25.1" class="ltx_text" style="background-color:#FAE7E6;">-1.2%</span></td>
<td id="A1.T16.3.3.9.6.26" class="ltx_td ltx_align_right" style="background-color:#FBEEED;"><span id="A1.T16.3.3.9.6.26.1" class="ltx_text" style="background-color:#FBEEED;">-0.9%</span></td>
<td id="A1.T16.3.3.9.6.27" class="ltx_td ltx_align_right" style="background-color:#F9DFDD;"><span id="A1.T16.3.3.9.6.27.1" class="ltx_text" style="background-color:#F9DFDD;">-1.7%</span></td>
</tr>
<tr id="A1.T16.3.3.10.7" class="ltx_tr">
<td id="A1.T16.3.3.10.7.1" class="ltx_td ltx_border_t"></td>
<td id="A1.T16.3.3.10.7.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T16.3.3.10.7.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF3F2;"><span id="A1.T16.3.3.10.7.3.1" class="ltx_text" style="background-color:#FCF3F2;">-0.6%</span></td>
<td id="A1.T16.3.3.10.7.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#8FD2B1;"><span id="A1.T16.3.3.10.7.4.1" class="ltx_text" style="background-color:#8FD2B1;">1.3%</span></td>
<td id="A1.T16.3.3.10.7.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#DFF2E9;"><span id="A1.T16.3.3.10.7.5.1" class="ltx_text" style="background-color:#DFF2E9;">0.4%</span></td>
<td id="A1.T16.3.3.10.7.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#93D4B4;"><span id="A1.T16.3.3.10.7.6.1" class="ltx_text" style="background-color:#93D4B4;">1.3%</span></td>
<td id="A1.T16.3.3.10.7.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E3F4EC;"><span id="A1.T16.3.3.10.7.7.1" class="ltx_text" style="background-color:#E3F4EC;">0.3%</span></td>
<td id="A1.T16.3.3.10.7.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#BDE4D1;"><span id="A1.T16.3.3.10.7.8.1" class="ltx_text" style="background-color:#BDE4D1;">0.8%</span></td>
<td id="A1.T16.3.3.10.7.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAFDFC;"><span id="A1.T16.3.3.10.7.9.1" class="ltx_text" style="background-color:#FAFDFC;">0.1%</span></td>
<td id="A1.T16.3.3.10.7.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#A1D9BE;"><span id="A1.T16.3.3.10.7.10.1" class="ltx_text" style="background-color:#A1D9BE;">1.1%</span></td>
<td id="A1.T16.3.3.10.7.11" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#CCEBDC;"><span id="A1.T16.3.3.10.7.11.1" class="ltx_text" style="background-color:#CCEBDC;">0.6%</span></td>
<td id="A1.T16.3.3.10.7.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFFFE;"><span id="A1.T16.3.3.10.7.12.1" class="ltx_text" style="background-color:#FEFFFE;">0.0%</span></td>
<td id="A1.T16.3.3.10.7.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#C6E8D7;"><span id="A1.T16.3.3.10.7.13.1" class="ltx_text" style="background-color:#C6E8D7;">0.7%</span></td>
<td id="A1.T16.3.3.10.7.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E3F4EC;"><span id="A1.T16.3.3.10.7.14.1" class="ltx_text" style="background-color:#E3F4EC;">0.3%</span></td>
<td id="A1.T16.3.3.10.7.15" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#88CFAC;"><span id="A1.T16.3.3.10.7.15.1" class="ltx_text" style="background-color:#88CFAC;">1.4%</span></td>
<td id="A1.T16.3.3.10.7.16" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF8F7;"><span id="A1.T16.3.3.10.7.16.1" class="ltx_text" style="background-color:#FDF8F7;">-0.4%</span></td>
<td id="A1.T16.3.3.10.7.17" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E1F3EB;"><span id="A1.T16.3.3.10.7.17.1" class="ltx_text" style="background-color:#E1F3EB;">0.4%</span></td>
<td id="A1.T16.3.3.10.7.18" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#CDEBDC;"><span id="A1.T16.3.3.10.7.18.1" class="ltx_text" style="background-color:#CDEBDC;">0.6%</span></td>
<td id="A1.T16.3.3.10.7.19" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T16.3.3.10.7.19.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T16.3.3.10.7.20" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F8;"><span id="A1.T16.3.3.10.7.20.1" class="ltx_text" style="background-color:#FDF9F8;">-0.3%</span></td>
<td id="A1.T16.3.3.10.7.21" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E4F4EC;"><span id="A1.T16.3.3.10.7.21.1" class="ltx_text" style="background-color:#E4F4EC;">0.3%</span></td>
<td id="A1.T16.3.3.10.7.22" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#BCE4D1;"><span id="A1.T16.3.3.10.7.22.1" class="ltx_text" style="background-color:#BCE4D1;">0.8%</span></td>
<td id="A1.T16.3.3.10.7.23" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#CFECDE;"><span id="A1.T16.3.3.10.7.23.1" class="ltx_text" style="background-color:#CFECDE;">0.6%</span></td>
<td id="A1.T16.3.3.10.7.24" class="ltx_td ltx_align_right ltx_border_rr ltx_border_t" style="background-color:#C0E6D3;"><span id="A1.T16.3.3.10.7.24.1" class="ltx_text" style="background-color:#C0E6D3;">0.8%</span></td>
<td id="A1.T16.3.3.10.7.25" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D8EFE4;"><span id="A1.T16.3.3.10.7.25.1" class="ltx_text" style="background-color:#D8EFE4;">0.5%</span></td>
<td id="A1.T16.3.3.10.7.26" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#E2F4EB;"><span id="A1.T16.3.3.10.7.26.1" class="ltx_text" style="background-color:#E2F4EB;">0.4%</span></td>
<td id="A1.T16.3.3.10.7.27" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#CBEADB;"><span id="A1.T16.3.3.10.7.27.1" class="ltx_text" style="background-color:#CBEADB;">0.6%</span></td>
</tr>
<tr id="A1.T16.3.3.11.8" class="ltx_tr">
<td id="A1.T16.3.3.11.8.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A1.T16.3.3.11.8.1.1" class="ltx_text">Aya-23-8b</span></td>
<td id="A1.T16.3.3.11.8.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4</td>
<td id="A1.T16.3.3.11.8.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F7D5D2;"><span id="A1.T16.3.3.11.8.3.1" class="ltx_text" style="background-color:#F7D5D2;">-2.2%</span></td>
<td id="A1.T16.3.3.11.8.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#CDEBDD;"><span id="A1.T16.3.3.11.8.4.1" class="ltx_text" style="background-color:#CDEBDD;">0.6%</span></td>
<td id="A1.T16.3.3.11.8.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FCF3F2;"><span id="A1.T16.3.3.11.8.5.1" class="ltx_text" style="background-color:#FCF3F2;">-0.6%</span></td>
<td id="A1.T16.3.3.11.8.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9E0DE;"><span id="A1.T16.3.3.11.8.6.1" class="ltx_text" style="background-color:#F9E0DE;">-1.6%</span></td>
<td id="A1.T16.3.3.11.8.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#99D6B8;"><span id="A1.T16.3.3.11.8.7.1" class="ltx_text" style="background-color:#99D6B8;">1.2%</span></td>
<td id="A1.T16.3.3.11.8.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#DFF2E9;"><span id="A1.T16.3.3.11.8.8.1" class="ltx_text" style="background-color:#DFF2E9;">0.4%</span></td>
<td id="A1.T16.3.3.11.8.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE6E4;"><span id="A1.T16.3.3.11.8.9.1" class="ltx_text" style="background-color:#FAE6E4;">-1.3%</span></td>
<td id="A1.T16.3.3.11.8.10" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F7D8D5;"><span id="A1.T16.3.3.11.8.10.1" class="ltx_text" style="background-color:#F7D8D5;">-2.1%</span></td>
<td id="A1.T16.3.3.11.8.11" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C3BF;"><span id="A1.T16.3.3.11.8.11.1" class="ltx_text" style="background-color:#F3C3BF;">-3.2%</span></td>
<td id="A1.T16.3.3.11.8.12" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F8DDDB;"><span id="A1.T16.3.3.11.8.12.1" class="ltx_text" style="background-color:#F8DDDB;">-1.8%</span></td>
<td id="A1.T16.3.3.11.8.13" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#7FCCA6;"><span id="A1.T16.3.3.11.8.13.1" class="ltx_text" style="background-color:#7FCCA6;">1.5%</span></td>
<td id="A1.T16.3.3.11.8.14" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FCF4F3;"><span id="A1.T16.3.3.11.8.14.1" class="ltx_text" style="background-color:#FCF4F3;">-0.6%</span></td>
<td id="A1.T16.3.3.11.8.15" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F5CDC9;"><span id="A1.T16.3.3.11.8.15.1" class="ltx_text" style="background-color:#F5CDC9;">-2.7%</span></td>
<td id="A1.T16.3.3.11.8.16" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FEFAF9;"><span id="A1.T16.3.3.11.8.16.1" class="ltx_text" style="background-color:#FEFAF9;">-0.3%</span></td>
<td id="A1.T16.3.3.11.8.17" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE6E4;"><span id="A1.T16.3.3.11.8.17.1" class="ltx_text" style="background-color:#FAE6E4;">-1.3%</span></td>
<td id="A1.T16.3.3.11.8.18" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#D5EEE2;"><span id="A1.T16.3.3.11.8.18.1" class="ltx_text" style="background-color:#D5EEE2;">0.5%</span></td>
<td id="A1.T16.3.3.11.8.19" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE8E7;"><span id="A1.T16.3.3.11.8.19.1" class="ltx_text" style="background-color:#FAE8E7;">-1.2%</span></td>
<td id="A1.T16.3.3.11.8.20" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FBEFED;"><span id="A1.T16.3.3.11.8.20.1" class="ltx_text" style="background-color:#FBEFED;">-0.9%</span></td>
<td id="A1.T16.3.3.11.8.21" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C3BF;"><span id="A1.T16.3.3.11.8.21.1" class="ltx_text" style="background-color:#F3C3BF;">-3.2%</span></td>
<td id="A1.T16.3.3.11.8.22" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE5E3;"><span id="A1.T16.3.3.11.8.22.1" class="ltx_text" style="background-color:#FAE5E3;">-1.4%</span></td>
<td id="A1.T16.3.3.11.8.23" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9E1DF;"><span id="A1.T16.3.3.11.8.23.1" class="ltx_text" style="background-color:#F9E1DF;">-1.6%</span></td>
<td id="A1.T16.3.3.11.8.24" class="ltx_td ltx_align_right ltx_border_bb ltx_border_rr" style="background-color:#FEFCFC;"><span id="A1.T16.3.3.11.8.24.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
<td id="A1.T16.3.3.11.8.25" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FBECEB;"><span id="A1.T16.3.3.11.8.25.1" class="ltx_text" style="background-color:#FBECEB;">-1.0%</span></td>
<td id="A1.T16.3.3.11.8.26" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FCF3F2;"><span id="A1.T16.3.3.11.8.26.1" class="ltx_text" style="background-color:#FCF3F2;">-0.6%</span></td>
<td id="A1.T16.3.3.11.8.27" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9E4E2;"><span id="A1.T16.3.3.11.8.27.1" class="ltx_text" style="background-color:#F9E4E2;">-1.4%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A16: </span><span id="A1.T16.6.1" class="ltx_text ltx_font_bold">Relative performance (<math id="A1.T16.6.1.m1.1" class="ltx_math_unparsed" alttext="\%\Delta" display="inline"><semantics id="A1.T16.6.1.m1.1b"><mrow id="A1.T16.6.1.m1.1c"><mo id="A1.T16.6.1.m1.1.1">%</mo><mi mathvariant="normal" id="A1.T16.6.1.m1.1.2">Δ</mi></mrow><annotation encoding="application/x-tex" id="A1.T16.6.1.m1.1d">\%\Delta</annotation></semantics></math>) vs. FP16 for Aya Models on FLORES.</span> Ltn are Latin-script languages: cs, de, es, fr, id, it, nl, pl, pt, ro, tr, vi. <math id="A1.T16.7.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T16.7.m1.1b"><mo id="A1.T16.7.m1.1.1" xref="A1.T16.7.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T16.7.m1.1c"><not id="A1.T16.7.m1.1.1.cmml" xref="A1.T16.7.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T16.7.m1.1d">\neg</annotation></semantics></math> are the rest.</figcaption>
</figure>
<figure id="A1.T17" class="ltx_table">
<table id="A1.T17.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T17.1.1.1" class="ltx_tr">
<th id="A1.T17.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt"></th>
<th id="A1.T17.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt"></th>
<th id="A1.T17.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Avg</th>
<th id="A1.T17.1.1.1.4" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">XSC</th>
<th id="A1.T17.1.1.1.5" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">XCOPA</th>
<th id="A1.T17.1.1.1.6" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">XWNG</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T17.1.2.1" class="ltx_tr">
<th id="A1.T17.1.2.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_r ltx_border_t"></th>
<th id="A1.T17.1.2.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="A1.T17.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">70.8</td>
<td id="A1.T17.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">65.1</td>
<td id="A1.T17.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">62.8</td>
<td id="A1.T17.1.2.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">84.4</td>
</tr>
<tr id="A1.T17.1.3.2" class="ltx_tr">
<th id="A1.T17.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Aya-23-35b</th>
<th id="A1.T17.1.3.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8</th>
<td id="A1.T17.1.3.2.3" class="ltx_td ltx_align_center">70.6</td>
<td id="A1.T17.1.3.2.4" class="ltx_td ltx_align_center">65.0</td>
<td id="A1.T17.1.3.2.5" class="ltx_td ltx_align_center">62.9</td>
<td id="A1.T17.1.3.2.6" class="ltx_td ltx_nopad_r ltx_align_center">83.9</td>
</tr>
<tr id="A1.T17.1.4.3" class="ltx_tr">
<th id="A1.T17.1.4.3.1" class="ltx_td ltx_th ltx_th_row ltx_border_r"></th>
<th id="A1.T17.1.4.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4</th>
<td id="A1.T17.1.4.3.3" class="ltx_td ltx_align_center">70.5</td>
<td id="A1.T17.1.4.3.4" class="ltx_td ltx_align_center" style="background-color:#FFFFFF;"><span id="A1.T17.1.4.3.4.1" class="ltx_text" style="background-color:#FFFFFF;">64.8</span></td>
<td id="A1.T17.1.4.3.5" class="ltx_td ltx_align_center" style="background-color:#FFFFFF;"><span id="A1.T17.1.4.3.5.1" class="ltx_text" style="background-color:#FFFFFF;">62.3</span></td>
<td id="A1.T17.1.4.3.6" class="ltx_td ltx_nopad_r ltx_align_center" style="background-color:#FFFFFF;"><span id="A1.T17.1.4.3.6.1" class="ltx_text" style="background-color:#FFFFFF;">84.5</span></td>
</tr>
<tr id="A1.T17.1.5.4" class="ltx_tr">
<th id="A1.T17.1.5.4.1" class="ltx_td ltx_th ltx_th_row ltx_border_r ltx_border_t"></th>
<th id="A1.T17.1.5.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="A1.T17.1.5.4.3" class="ltx_td ltx_align_center ltx_border_t">67.6</td>
<td id="A1.T17.1.5.4.4" class="ltx_td ltx_align_center ltx_border_t">62.3</td>
<td id="A1.T17.1.5.4.5" class="ltx_td ltx_align_center ltx_border_t">59.8</td>
<td id="A1.T17.1.5.4.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">80.7</td>
</tr>
<tr id="A1.T17.1.6.5" class="ltx_tr">
<th id="A1.T17.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Aya-23-8b</th>
<th id="A1.T17.1.6.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8</th>
<td id="A1.T17.1.6.5.3" class="ltx_td ltx_align_center">67.6</td>
<td id="A1.T17.1.6.5.4" class="ltx_td ltx_align_center">62.4</td>
<td id="A1.T17.1.6.5.5" class="ltx_td ltx_align_center">60.0</td>
<td id="A1.T17.1.6.5.6" class="ltx_td ltx_nopad_r ltx_align_center">80.6</td>
</tr>
<tr id="A1.T17.1.7.6" class="ltx_tr">
<th id="A1.T17.1.7.6.1" class="ltx_td ltx_th ltx_th_row ltx_border_bb ltx_border_r"></th>
<th id="A1.T17.1.7.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">W4</th>
<td id="A1.T17.1.7.6.3" class="ltx_td ltx_align_center ltx_border_bb">67.5</td>
<td id="A1.T17.1.7.6.4" class="ltx_td ltx_align_center ltx_border_bb">62.3</td>
<td id="A1.T17.1.7.6.5" class="ltx_td ltx_align_center ltx_border_bb">59.6</td>
<td id="A1.T17.1.7.6.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">80.6</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A17: </span><span id="A1.T17.3.1" class="ltx_text ltx_font_bold">Performance of quantized Aya 23 models on unseen discriminative tasks</span>. XStoryCloze (XSC), XCOPA, and XWinograd (XWNG).</figcaption>
</figure>
<figure id="A1.T18" class="ltx_table">
<div id="A1.T18.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:121.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-167.5pt,47.1pt) scale(0.564139197050772,0.564139197050772) ;">
<table id="A1.T18.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T18.1.1.1.1" class="ltx_tr">
<th id="A1.T18.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt"></th>
<th id="A1.T18.1.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt"></th>
<th id="A1.T18.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="10"><span id="A1.T18.1.1.1.1.3.1" class="ltx_text ltx_font_bold">Monolingual</span></th>
<th id="A1.T18.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="10"><span id="A1.T18.1.1.1.1.4.1" class="ltx_text ltx_font_bold">Cross-Lingual</span></th>
</tr>
<tr id="A1.T18.1.1.2.2" class="ltx_tr">
<th id="A1.T18.1.1.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_th_row"></th>
<th id="A1.T18.1.1.2.2.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r"></th>
<th id="A1.T18.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.3.1" class="ltx_text ltx_font_bold">ar</span></th>
<th id="A1.T18.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.4.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T18.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.5.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T18.1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.6.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T18.1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.7.1" class="ltx_text ltx_font_bold">it</span></th>
<th id="A1.T18.1.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.8.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T18.1.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.9.1" class="ltx_text ltx_font_bold">ko</span></th>
<th id="A1.T18.1.1.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.10.1" class="ltx_text ltx_font_bold">pt</span></th>
<th id="A1.T18.1.1.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.11.1" class="ltx_text ltx_font_bold">zh</span></th>
<th id="A1.T18.1.1.2.2.12" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.12.1" class="ltx_text ltx_font_bold">Avg</span></th>
<th id="A1.T18.1.1.2.2.13" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.13.1" class="ltx_text ltx_font_bold">ar</span></th>
<th id="A1.T18.1.1.2.2.14" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.14.1" class="ltx_text ltx_font_bold">de</span></th>
<th id="A1.T18.1.1.2.2.15" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.15.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T18.1.1.2.2.16" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.16.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T18.1.1.2.2.17" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.17.1" class="ltx_text ltx_font_bold">it</span></th>
<th id="A1.T18.1.1.2.2.18" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.18.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T18.1.1.2.2.19" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.19.1" class="ltx_text ltx_font_bold">ko</span></th>
<th id="A1.T18.1.1.2.2.20" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.20.1" class="ltx_text ltx_font_bold">pt</span></th>
<th id="A1.T18.1.1.2.2.21" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.21.1" class="ltx_text ltx_font_bold">zh</span></th>
<th id="A1.T18.1.1.2.2.22" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T18.1.1.2.2.22.1" class="ltx_text ltx_font_bold">Avg</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T18.1.1.3.1" class="ltx_tr">
<th id="A1.T18.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="6"><span id="A1.T18.1.1.3.1.1.1" class="ltx_text">103B</span></th>
<th id="A1.T18.1.1.3.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="A1.T18.1.1.3.1.3" class="ltx_td ltx_align_right ltx_border_t">99.3</td>
<td id="A1.T18.1.1.3.1.4" class="ltx_td ltx_align_right ltx_border_t">100.0</td>
<td id="A1.T18.1.1.3.1.5" class="ltx_td ltx_align_right ltx_border_t">99.3</td>
<td id="A1.T18.1.1.3.1.6" class="ltx_td ltx_align_right ltx_border_t">99.6</td>
<td id="A1.T18.1.1.3.1.7" class="ltx_td ltx_align_right ltx_border_t">100.0</td>
<td id="A1.T18.1.1.3.1.8" class="ltx_td ltx_align_right ltx_border_t">98.6</td>
<td id="A1.T18.1.1.3.1.9" class="ltx_td ltx_align_right ltx_border_t">100.0</td>
<td id="A1.T18.1.1.3.1.10" class="ltx_td ltx_align_right ltx_border_t">98.3</td>
<td id="A1.T18.1.1.3.1.11" class="ltx_td ltx_align_right ltx_border_t">97.9</td>
<td id="A1.T18.1.1.3.1.12" class="ltx_td ltx_align_center ltx_border_rr ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.3.1.12.1" class="ltx_text" style="background-color:#E6E6E6;">99.2</span></td>
<td id="A1.T18.1.1.3.1.13" class="ltx_td ltx_align_right ltx_border_t">93.0</td>
<td id="A1.T18.1.1.3.1.14" class="ltx_td ltx_align_right ltx_border_t">90.6</td>
<td id="A1.T18.1.1.3.1.15" class="ltx_td ltx_align_right ltx_border_t">91.2</td>
<td id="A1.T18.1.1.3.1.16" class="ltx_td ltx_align_right ltx_border_t">91.6</td>
<td id="A1.T18.1.1.3.1.17" class="ltx_td ltx_align_right ltx_border_t">93.0</td>
<td id="A1.T18.1.1.3.1.18" class="ltx_td ltx_align_right ltx_border_t">93.1</td>
<td id="A1.T18.1.1.3.1.19" class="ltx_td ltx_align_right ltx_border_t">91.1</td>
<td id="A1.T18.1.1.3.1.20" class="ltx_td ltx_align_right ltx_border_t">88.3</td>
<td id="A1.T18.1.1.3.1.21" class="ltx_td ltx_align_right ltx_border_t">91.3</td>
<td id="A1.T18.1.1.3.1.22" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.3.1.22.1" class="ltx_text" style="background-color:#E6E6E6;">91.5</span></td>
</tr>
<tr id="A1.T18.1.1.4.2" class="ltx_tr">
<th id="A1.T18.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8</th>
<td id="A1.T18.1.1.4.2.2" class="ltx_td ltx_align_right">99.0</td>
<td id="A1.T18.1.1.4.2.3" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.4.2.4" class="ltx_td ltx_align_right">99.5</td>
<td id="A1.T18.1.1.4.2.5" class="ltx_td ltx_align_right">99.4</td>
<td id="A1.T18.1.1.4.2.6" class="ltx_td ltx_align_right">99.8</td>
<td id="A1.T18.1.1.4.2.7" class="ltx_td ltx_align_right">99.2</td>
<td id="A1.T18.1.1.4.2.8" class="ltx_td ltx_align_right">99.8</td>
<td id="A1.T18.1.1.4.2.9" class="ltx_td ltx_align_right">97.8</td>
<td id="A1.T18.1.1.4.2.10" class="ltx_td ltx_align_right">98.5</td>
<td id="A1.T18.1.1.4.2.11" class="ltx_td ltx_align_center ltx_border_rr" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.4.2.11.1" class="ltx_text" style="background-color:#E6E6E6;">99.2</span></td>
<td id="A1.T18.1.1.4.2.12" class="ltx_td ltx_align_right">92.6</td>
<td id="A1.T18.1.1.4.2.13" class="ltx_td ltx_align_right">91.1</td>
<td id="A1.T18.1.1.4.2.14" class="ltx_td ltx_align_right">91.7</td>
<td id="A1.T18.1.1.4.2.15" class="ltx_td ltx_align_right">91.4</td>
<td id="A1.T18.1.1.4.2.16" class="ltx_td ltx_align_right">92.9</td>
<td id="A1.T18.1.1.4.2.17" class="ltx_td ltx_align_right">92.8</td>
<td id="A1.T18.1.1.4.2.18" class="ltx_td ltx_align_right">91.3</td>
<td id="A1.T18.1.1.4.2.19" class="ltx_td ltx_align_right">87.4</td>
<td id="A1.T18.1.1.4.2.20" class="ltx_td ltx_align_right">89.7</td>
<td id="A1.T18.1.1.4.2.21" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.4.2.21.1" class="ltx_text" style="background-color:#E6E6E6;">91.2</span></td>
</tr>
<tr id="A1.T18.1.1.5.3" class="ltx_tr">
<th id="A1.T18.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="A1.T18.1.1.5.3.2" class="ltx_td ltx_align_right">99.4</td>
<td id="A1.T18.1.1.5.3.3" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.5.3.4" class="ltx_td ltx_align_right">99.3</td>
<td id="A1.T18.1.1.5.3.5" class="ltx_td ltx_align_right">99.6</td>
<td id="A1.T18.1.1.5.3.6" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.5.3.7" class="ltx_td ltx_align_right">98.6</td>
<td id="A1.T18.1.1.5.3.8" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.5.3.9" class="ltx_td ltx_align_right">97.7</td>
<td id="A1.T18.1.1.5.3.10" class="ltx_td ltx_align_right">98.4</td>
<td id="A1.T18.1.1.5.3.11" class="ltx_td ltx_align_center ltx_border_rr" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.5.3.11.1" class="ltx_text" style="background-color:#E6E6E6;">99.2</span></td>
<td id="A1.T18.1.1.5.3.12" class="ltx_td ltx_align_right">93.3</td>
<td id="A1.T18.1.1.5.3.13" class="ltx_td ltx_align_right">91.5</td>
<td id="A1.T18.1.1.5.3.14" class="ltx_td ltx_align_right">91.4</td>
<td id="A1.T18.1.1.5.3.15" class="ltx_td ltx_align_right">92.4</td>
<td id="A1.T18.1.1.5.3.16" class="ltx_td ltx_align_right">92.1</td>
<td id="A1.T18.1.1.5.3.17" class="ltx_td ltx_align_right">93.3</td>
<td id="A1.T18.1.1.5.3.18" class="ltx_td ltx_align_right">92.1</td>
<td id="A1.T18.1.1.5.3.19" class="ltx_td ltx_align_right">87.6</td>
<td id="A1.T18.1.1.5.3.20" class="ltx_td ltx_align_right">90.0</td>
<td id="A1.T18.1.1.5.3.21" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.5.3.21.1" class="ltx_text" style="background-color:#E6E6E6;">91.5</span></td>
</tr>
<tr id="A1.T18.1.1.6.4" class="ltx_tr">
<th id="A1.T18.1.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8</th>
<td id="A1.T18.1.1.6.4.2" class="ltx_td ltx_align_right">99.3</td>
<td id="A1.T18.1.1.6.4.3" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.6.4.4" class="ltx_td ltx_align_right">99.5</td>
<td id="A1.T18.1.1.6.4.5" class="ltx_td ltx_align_right">99.8</td>
<td id="A1.T18.1.1.6.4.6" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.6.4.7" class="ltx_td ltx_align_right">99.0</td>
<td id="A1.T18.1.1.6.4.8" class="ltx_td ltx_align_right">99.8</td>
<td id="A1.T18.1.1.6.4.9" class="ltx_td ltx_align_right">98.1</td>
<td id="A1.T18.1.1.6.4.10" class="ltx_td ltx_align_right">99.1</td>
<td id="A1.T18.1.1.6.4.11" class="ltx_td ltx_align_center ltx_border_rr" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.6.4.11.1" class="ltx_text" style="background-color:#E6E6E6;">99.4</span></td>
<td id="A1.T18.1.1.6.4.12" class="ltx_td ltx_align_right">91.3</td>
<td id="A1.T18.1.1.6.4.13" class="ltx_td ltx_align_right">89.3</td>
<td id="A1.T18.1.1.6.4.14" class="ltx_td ltx_align_right">91.0</td>
<td id="A1.T18.1.1.6.4.15" class="ltx_td ltx_align_right">91.1</td>
<td id="A1.T18.1.1.6.4.16" class="ltx_td ltx_align_right">91.8</td>
<td id="A1.T18.1.1.6.4.17" class="ltx_td ltx_align_right">93.0</td>
<td id="A1.T18.1.1.6.4.18" class="ltx_td ltx_align_right">89.3</td>
<td id="A1.T18.1.1.6.4.19" class="ltx_td ltx_align_right">87.3</td>
<td id="A1.T18.1.1.6.4.20" class="ltx_td ltx_align_right">89.2</td>
<td id="A1.T18.1.1.6.4.21" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.6.4.21.1" class="ltx_text" style="background-color:#E6E6E6;">90.4</span></td>
</tr>
<tr id="A1.T18.1.1.7.5" class="ltx_tr">
<th id="A1.T18.1.1.7.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4-g</th>
<td id="A1.T18.1.1.7.5.2" class="ltx_td ltx_align_right">99.1</td>
<td id="A1.T18.1.1.7.5.3" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.7.5.4" class="ltx_td ltx_align_right">99.6</td>
<td id="A1.T18.1.1.7.5.5" class="ltx_td ltx_align_right">99.9</td>
<td id="A1.T18.1.1.7.5.6" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.7.5.7" class="ltx_td ltx_align_right">97.4</td>
<td id="A1.T18.1.1.7.5.8" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.7.5.9" class="ltx_td ltx_align_right">98.1</td>
<td id="A1.T18.1.1.7.5.10" class="ltx_td ltx_align_right">98.9</td>
<td id="A1.T18.1.1.7.5.11" class="ltx_td ltx_align_center ltx_border_rr" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.7.5.11.1" class="ltx_text" style="background-color:#E6E6E6;">99.2</span></td>
<td id="A1.T18.1.1.7.5.12" class="ltx_td ltx_align_right">90.6</td>
<td id="A1.T18.1.1.7.5.13" class="ltx_td ltx_align_right">89.9</td>
<td id="A1.T18.1.1.7.5.14" class="ltx_td ltx_align_right">90.7</td>
<td id="A1.T18.1.1.7.5.15" class="ltx_td ltx_align_right">91.7</td>
<td id="A1.T18.1.1.7.5.16" class="ltx_td ltx_align_right">93.1</td>
<td id="A1.T18.1.1.7.5.17" class="ltx_td ltx_align_right">92.8</td>
<td id="A1.T18.1.1.7.5.18" class="ltx_td ltx_align_right">90.6</td>
<td id="A1.T18.1.1.7.5.19" class="ltx_td ltx_align_right">85.4</td>
<td id="A1.T18.1.1.7.5.20" class="ltx_td ltx_align_right">89.6</td>
<td id="A1.T18.1.1.7.5.21" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.7.5.21.1" class="ltx_text" style="background-color:#E6E6E6;">90.5</span></td>
</tr>
<tr id="A1.T18.1.1.8.6" class="ltx_tr">
<th id="A1.T18.1.1.8.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4</th>
<td id="A1.T18.1.1.8.6.2" class="ltx_td ltx_align_right">99.4</td>
<td id="A1.T18.1.1.8.6.3" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.8.6.4" class="ltx_td ltx_align_right">99.4</td>
<td id="A1.T18.1.1.8.6.5" class="ltx_td ltx_align_right">99.7</td>
<td id="A1.T18.1.1.8.6.6" class="ltx_td ltx_align_right">99.8</td>
<td id="A1.T18.1.1.8.6.7" class="ltx_td ltx_align_right">99.6</td>
<td id="A1.T18.1.1.8.6.8" class="ltx_td ltx_align_right">99.0</td>
<td id="A1.T18.1.1.8.6.9" class="ltx_td ltx_align_right">98.9</td>
<td id="A1.T18.1.1.8.6.10" class="ltx_td ltx_align_right">98.4</td>
<td id="A1.T18.1.1.8.6.11" class="ltx_td ltx_align_center ltx_border_rr" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.8.6.11.1" class="ltx_text" style="background-color:#E6E6E6;">99.3</span></td>
<td id="A1.T18.1.1.8.6.12" class="ltx_td ltx_align_right">95.8</td>
<td id="A1.T18.1.1.8.6.13" class="ltx_td ltx_align_right">94.3</td>
<td id="A1.T18.1.1.8.6.14" class="ltx_td ltx_align_right">95.9</td>
<td id="A1.T18.1.1.8.6.15" class="ltx_td ltx_align_right">93.8</td>
<td id="A1.T18.1.1.8.6.16" class="ltx_td ltx_align_right">93.6</td>
<td id="A1.T18.1.1.8.6.17" class="ltx_td ltx_align_right">92.6</td>
<td id="A1.T18.1.1.8.6.18" class="ltx_td ltx_align_right">88.9</td>
<td id="A1.T18.1.1.8.6.19" class="ltx_td ltx_align_right">90.5</td>
<td id="A1.T18.1.1.8.6.20" class="ltx_td ltx_align_right">89.7</td>
<td id="A1.T18.1.1.8.6.21" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.8.6.21.1" class="ltx_text" style="background-color:#E6E6E6;">92.8</span></td>
</tr>
<tr id="A1.T18.1.1.9.7" class="ltx_tr">
<th id="A1.T18.1.1.9.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t" rowspan="4"><span id="A1.T18.1.1.9.7.1.1" class="ltx_text">35B</span></th>
<th id="A1.T18.1.1.9.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">FP16</th>
<td id="A1.T18.1.1.9.7.3" class="ltx_td ltx_align_right ltx_border_t">99.2</td>
<td id="A1.T18.1.1.9.7.4" class="ltx_td ltx_align_right ltx_border_t">97.0</td>
<td id="A1.T18.1.1.9.7.5" class="ltx_td ltx_align_right ltx_border_t">98.1</td>
<td id="A1.T18.1.1.9.7.6" class="ltx_td ltx_align_right ltx_border_t">99.2</td>
<td id="A1.T18.1.1.9.7.7" class="ltx_td ltx_align_right ltx_border_t">99.6</td>
<td id="A1.T18.1.1.9.7.8" class="ltx_td ltx_align_right ltx_border_t">99.6</td>
<td id="A1.T18.1.1.9.7.9" class="ltx_td ltx_align_right ltx_border_t">99.0</td>
<td id="A1.T18.1.1.9.7.10" class="ltx_td ltx_align_right ltx_border_t">99.0</td>
<td id="A1.T18.1.1.9.7.11" class="ltx_td ltx_align_right ltx_border_t">97.7</td>
<td id="A1.T18.1.1.9.7.12" class="ltx_td ltx_align_center ltx_border_rr ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.9.7.12.1" class="ltx_text" style="background-color:#E6E6E6;">98.7</span></td>
<td id="A1.T18.1.1.9.7.13" class="ltx_td ltx_align_right ltx_border_t">58.8</td>
<td id="A1.T18.1.1.9.7.14" class="ltx_td ltx_align_right ltx_border_t">59.6</td>
<td id="A1.T18.1.1.9.7.15" class="ltx_td ltx_align_right ltx_border_t">69.0</td>
<td id="A1.T18.1.1.9.7.16" class="ltx_td ltx_align_right ltx_border_t">73.0</td>
<td id="A1.T18.1.1.9.7.17" class="ltx_td ltx_align_right ltx_border_t">63.6</td>
<td id="A1.T18.1.1.9.7.18" class="ltx_td ltx_align_right ltx_border_t">66.3</td>
<td id="A1.T18.1.1.9.7.19" class="ltx_td ltx_align_right ltx_border_t">69.2</td>
<td id="A1.T18.1.1.9.7.20" class="ltx_td ltx_align_right ltx_border_t">64.2</td>
<td id="A1.T18.1.1.9.7.21" class="ltx_td ltx_align_right ltx_border_t">74.6</td>
<td id="A1.T18.1.1.9.7.22" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.9.7.22.1" class="ltx_text" style="background-color:#E6E6E6;">66.5</span></td>
</tr>
<tr id="A1.T18.1.1.10.8" class="ltx_tr">
<th id="A1.T18.1.1.10.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8</th>
<td id="A1.T18.1.1.10.8.2" class="ltx_td ltx_align_right">99.7</td>
<td id="A1.T18.1.1.10.8.3" class="ltx_td ltx_align_right">97.0</td>
<td id="A1.T18.1.1.10.8.4" class="ltx_td ltx_align_right">98.1</td>
<td id="A1.T18.1.1.10.8.5" class="ltx_td ltx_align_right">98.9</td>
<td id="A1.T18.1.1.10.8.6" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.10.8.7" class="ltx_td ltx_align_right">99.8</td>
<td id="A1.T18.1.1.10.8.8" class="ltx_td ltx_align_right">99.0</td>
<td id="A1.T18.1.1.10.8.9" class="ltx_td ltx_align_right">99.3</td>
<td id="A1.T18.1.1.10.8.10" class="ltx_td ltx_align_right">97.4</td>
<td id="A1.T18.1.1.10.8.11" class="ltx_td ltx_align_center ltx_border_rr" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.10.8.11.1" class="ltx_text" style="background-color:#E6E6E6;">98.8</span></td>
<td id="A1.T18.1.1.10.8.12" class="ltx_td ltx_align_right">59.8</td>
<td id="A1.T18.1.1.10.8.13" class="ltx_td ltx_align_right">58.6</td>
<td id="A1.T18.1.1.10.8.14" class="ltx_td ltx_align_right">69.2</td>
<td id="A1.T18.1.1.10.8.15" class="ltx_td ltx_align_right">72.8</td>
<td id="A1.T18.1.1.10.8.16" class="ltx_td ltx_align_right">62.3</td>
<td id="A1.T18.1.1.10.8.17" class="ltx_td ltx_align_right">66.8</td>
<td id="A1.T18.1.1.10.8.18" class="ltx_td ltx_align_right">68.7</td>
<td id="A1.T18.1.1.10.8.19" class="ltx_td ltx_align_right">64.3</td>
<td id="A1.T18.1.1.10.8.20" class="ltx_td ltx_align_right">74.5</td>
<td id="A1.T18.1.1.10.8.21" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.10.8.21.1" class="ltx_text" style="background-color:#E6E6E6;">66.3</span></td>
</tr>
<tr id="A1.T18.1.1.11.9" class="ltx_tr">
<th id="A1.T18.1.1.11.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8</th>
<td id="A1.T18.1.1.11.9.2" class="ltx_td ltx_align_right">99.9</td>
<td id="A1.T18.1.1.11.9.3" class="ltx_td ltx_align_right">98.0</td>
<td id="A1.T18.1.1.11.9.4" class="ltx_td ltx_align_right">97.1</td>
<td id="A1.T18.1.1.11.9.5" class="ltx_td ltx_align_right">98.9</td>
<td id="A1.T18.1.1.11.9.6" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.11.9.7" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.11.9.8" class="ltx_td ltx_align_right">100.0</td>
<td id="A1.T18.1.1.11.9.9" class="ltx_td ltx_align_right">99.0</td>
<td id="A1.T18.1.1.11.9.10" class="ltx_td ltx_align_right">98.4</td>
<td id="A1.T18.1.1.11.9.11" class="ltx_td ltx_align_center ltx_border_rr" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.11.9.11.1" class="ltx_text" style="background-color:#E6E6E6;">99.0</span></td>
<td id="A1.T18.1.1.11.9.12" class="ltx_td ltx_align_right">61.0</td>
<td id="A1.T18.1.1.11.9.13" class="ltx_td ltx_align_right">63.9</td>
<td id="A1.T18.1.1.11.9.14" class="ltx_td ltx_align_right">72.1</td>
<td id="A1.T18.1.1.11.9.15" class="ltx_td ltx_align_right">75.1</td>
<td id="A1.T18.1.1.11.9.16" class="ltx_td ltx_align_right">66.2</td>
<td id="A1.T18.1.1.11.9.17" class="ltx_td ltx_align_right">68.3</td>
<td id="A1.T18.1.1.11.9.18" class="ltx_td ltx_align_right">70.1</td>
<td id="A1.T18.1.1.11.9.19" class="ltx_td ltx_align_right">67.4</td>
<td id="A1.T18.1.1.11.9.20" class="ltx_td ltx_align_right">76.3</td>
<td id="A1.T18.1.1.11.9.21" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.11.9.21.1" class="ltx_text" style="background-color:#E6E6E6;">68.9</span></td>
</tr>
<tr id="A1.T18.1.1.12.10" class="ltx_tr">
<th id="A1.T18.1.1.12.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">W4-g</th>
<td id="A1.T18.1.1.12.10.2" class="ltx_td ltx_align_right ltx_border_bb">99.4</td>
<td id="A1.T18.1.1.12.10.3" class="ltx_td ltx_align_right ltx_border_bb">95.0</td>
<td id="A1.T18.1.1.12.10.4" class="ltx_td ltx_align_right ltx_border_bb">96.5</td>
<td id="A1.T18.1.1.12.10.5" class="ltx_td ltx_align_right ltx_border_bb">99.9</td>
<td id="A1.T18.1.1.12.10.6" class="ltx_td ltx_align_right ltx_border_bb">100.0</td>
<td id="A1.T18.1.1.12.10.7" class="ltx_td ltx_align_right ltx_border_bb">99.8</td>
<td id="A1.T18.1.1.12.10.8" class="ltx_td ltx_align_right ltx_border_bb">97.0</td>
<td id="A1.T18.1.1.12.10.9" class="ltx_td ltx_align_right ltx_border_bb">98.3</td>
<td id="A1.T18.1.1.12.10.10" class="ltx_td ltx_align_right ltx_border_bb">98.6</td>
<td id="A1.T18.1.1.12.10.11" class="ltx_td ltx_align_center ltx_border_bb ltx_border_rr" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.12.10.11.1" class="ltx_text" style="background-color:#E6E6E6;">98.3</span></td>
<td id="A1.T18.1.1.12.10.12" class="ltx_td ltx_align_right ltx_border_bb">60.4</td>
<td id="A1.T18.1.1.12.10.13" class="ltx_td ltx_align_right ltx_border_bb">59.3</td>
<td id="A1.T18.1.1.12.10.14" class="ltx_td ltx_align_right ltx_border_bb">72.8</td>
<td id="A1.T18.1.1.12.10.15" class="ltx_td ltx_align_right ltx_border_bb">73.3</td>
<td id="A1.T18.1.1.12.10.16" class="ltx_td ltx_align_right ltx_border_bb">64.8</td>
<td id="A1.T18.1.1.12.10.17" class="ltx_td ltx_align_right ltx_border_bb">65.4</td>
<td id="A1.T18.1.1.12.10.18" class="ltx_td ltx_align_right ltx_border_bb">70.4</td>
<td id="A1.T18.1.1.12.10.19" class="ltx_td ltx_align_right ltx_border_bb">64.6</td>
<td id="A1.T18.1.1.12.10.20" class="ltx_td ltx_align_right ltx_border_bb">73.0</td>
<td id="A1.T18.1.1.12.10.21" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;"><span id="A1.T18.1.1.12.10.21.1" class="ltx_text" style="background-color:#E6E6E6;">67.1</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A18: </span><span id="A1.T18.3.1" class="ltx_text ltx_font_bold">Language Confusion scores for Command Models.</span> (Line-level pass rate (LPR))</figcaption>
</figure>
<figure id="A1.T19" class="ltx_table">
<div id="A1.T19.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:261.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-66.7pt,40.2pt) scale(0.764648409620307,0.764648409620307) ;">
<table id="A1.T19.1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T19.1.1.2.1" class="ltx_tr">
<td id="A1.T19.1.1.2.1.1" class="ltx_td ltx_border_tt"></td>
<td id="A1.T19.1.1.2.1.2" class="ltx_td ltx_border_tt"></td>
<td id="A1.T19.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="12"><span id="A1.T19.1.1.2.1.3.1" class="ltx_text ltx_font_bold">Monolingual</span></td>
</tr>
<tr id="A1.T19.1.1.1" class="ltx_tr">
<td id="A1.T19.1.1.1.2" class="ltx_td"></td>
<td id="A1.T19.1.1.1.3" class="ltx_td ltx_border_r"></td>
<td id="A1.T19.1.1.1.4" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.4.1" class="ltx_text ltx_font_bold">ar</span></td>
<td id="A1.T19.1.1.1.5" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.5.1" class="ltx_text ltx_font_bold">de</span></td>
<td id="A1.T19.1.1.1.6" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.6.1" class="ltx_text ltx_font_bold">es</span></td>
<td id="A1.T19.1.1.1.7" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.7.1" class="ltx_text ltx_font_bold">fr</span></td>
<td id="A1.T19.1.1.1.8" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.8.1" class="ltx_text ltx_font_bold">it</span></td>
<td id="A1.T19.1.1.1.9" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.9.1" class="ltx_text ltx_font_bold">ja</span></td>
<td id="A1.T19.1.1.1.10" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.10.1" class="ltx_text ltx_font_bold">ko</span></td>
<td id="A1.T19.1.1.1.11" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.11.1" class="ltx_text ltx_font_bold">pt</span></td>
<td id="A1.T19.1.1.1.12" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T19.1.1.1.12.1" class="ltx_text ltx_font_bold">zh</span></td>
<td id="A1.T19.1.1.1.13" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.13.1" class="ltx_text ltx_font_bold">Avg</span></td>
<td id="A1.T19.1.1.1.14" class="ltx_td ltx_align_center"><span id="A1.T19.1.1.1.14.1" class="ltx_text ltx_font_bold">Ltn/IE</span></td>
<td id="A1.T19.1.1.1.1" class="ltx_td ltx_align_center"><math id="A1.T19.1.1.1.1.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T19.1.1.1.1.m1.1a"><mo id="A1.T19.1.1.1.1.m1.1.1" xref="A1.T19.1.1.1.1.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T19.1.1.1.1.m1.1b"><not id="A1.T19.1.1.1.1.m1.1.1.cmml" xref="A1.T19.1.1.1.1.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T19.1.1.1.1.m1.1c">\neg</annotation></semantics></math></td>
</tr>
<tr id="A1.T19.1.1.3.2" class="ltx_tr">
<td id="A1.T19.1.1.3.2.1" class="ltx_td ltx_align_left ltx_border_t">103B</td>
<td id="A1.T19.1.1.3.2.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T19.1.1.3.2.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F7;"><span id="A1.T19.1.1.3.2.3.1" class="ltx_text" style="background-color:#FDF7F7;">-0.3%</span></td>
<td id="A1.T19.1.1.3.2.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.3.2.4.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.3.2.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAFDFC;"><span id="A1.T19.1.1.3.2.5.1" class="ltx_text" style="background-color:#FAFDFC;">0.2%</span></td>
<td id="A1.T19.1.1.3.2.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFAF9;"><span id="A1.T19.1.1.3.2.6.1" class="ltx_text" style="background-color:#FEFAF9;">-0.2%</span></td>
<td id="A1.T19.1.1.3.2.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F9;"><span id="A1.T19.1.1.3.2.7.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="A1.T19.1.1.3.2.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F0F9F5;"><span id="A1.T19.1.1.3.2.8.1" class="ltx_text" style="background-color:#F0F9F5;">0.6%</span></td>
<td id="A1.T19.1.1.3.2.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F9;"><span id="A1.T19.1.1.3.2.9.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="A1.T19.1.1.3.2.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FCF1F0;"><span id="A1.T19.1.1.3.2.10.1" class="ltx_text" style="background-color:#FCF1F0;">-0.5%</span></td>
<td id="A1.T19.1.1.3.2.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F0F9F4;"><span id="A1.T19.1.1.3.2.11.1" class="ltx_text" style="background-color:#F0F9F4;">0.6%</span></td>
<td id="A1.T19.1.1.3.2.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.3.2.12.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.3.2.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FEFBFB;"><span id="A1.T19.1.1.3.2.13.1" class="ltx_text" style="background-color:#FEFBFB;">-0.1%</span></td>
<td id="A1.T19.1.1.3.2.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBFEFC;"><span id="A1.T19.1.1.3.2.14.1" class="ltx_text" style="background-color:#FBFEFC;">0.2%</span></td>
</tr>
<tr id="A1.T19.1.1.4.3" class="ltx_tr">
<td id="A1.T19.1.1.4.3.1" class="ltx_td"></td>
<td id="A1.T19.1.1.4.3.2" class="ltx_td ltx_align_left ltx_border_r">W8A8-sq</td>
<td id="A1.T19.1.1.4.3.3" class="ltx_td ltx_align_right" style="background-color:#FEFFFE;"><span id="A1.T19.1.1.4.3.3.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="A1.T19.1.1.4.3.4" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.4.3.4.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.4.3.5" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.4.3.5.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.4.3.6" class="ltx_td ltx_align_right" style="background-color:#FEFFFF;"><span id="A1.T19.1.1.4.3.6.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.4.3.7" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.4.3.7.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.4.3.8" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.4.3.8.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.4.3.9" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.4.3.9.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.4.3.10" class="ltx_td ltx_align_right" style="background-color:#FBEFED;"><span id="A1.T19.1.1.4.3.10.1" class="ltx_text" style="background-color:#FBEFED;">-0.6%</span></td>
<td id="A1.T19.1.1.4.3.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F2FAF6;"><span id="A1.T19.1.1.4.3.11.1" class="ltx_text" style="background-color:#F2FAF6;">0.5%</span></td>
<td id="A1.T19.1.1.4.3.12" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.4.3.12.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.4.3.13" class="ltx_td ltx_align_right" style="background-color:#FEFCFB;"><span id="A1.T19.1.1.4.3.13.1" class="ltx_text" style="background-color:#FEFCFB;">-0.1%</span></td>
<td id="A1.T19.1.1.4.3.14" class="ltx_td ltx_align_right" style="background-color:#FCFEFD;"><span id="A1.T19.1.1.4.3.14.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
</tr>
<tr id="A1.T19.1.1.5.4" class="ltx_tr">
<td id="A1.T19.1.1.5.4.1" class="ltx_td"></td>
<td id="A1.T19.1.1.5.4.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T19.1.1.5.4.3" class="ltx_td ltx_align_right" style="background-color:#FEFEFE;"><span id="A1.T19.1.1.5.4.3.1" class="ltx_text" style="background-color:#FEFEFE;">0.0%</span></td>
<td id="A1.T19.1.1.5.4.4" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.5.4.4.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.5.4.5" class="ltx_td ltx_align_right" style="background-color:#FAFDFC;"><span id="A1.T19.1.1.5.4.5.1" class="ltx_text" style="background-color:#FAFDFC;">0.2%</span></td>
<td id="A1.T19.1.1.5.4.6" class="ltx_td ltx_align_right" style="background-color:#FAFDFC;"><span id="A1.T19.1.1.5.4.6.1" class="ltx_text" style="background-color:#FAFDFC;">0.2%</span></td>
<td id="A1.T19.1.1.5.4.7" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.5.4.7.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.5.4.8" class="ltx_td ltx_align_right" style="background-color:#F5FBF8;"><span id="A1.T19.1.1.5.4.8.1" class="ltx_text" style="background-color:#F5FBF8;">0.4%</span></td>
<td id="A1.T19.1.1.5.4.9" class="ltx_td ltx_align_right" style="background-color:#FDF9F9;"><span id="A1.T19.1.1.5.4.9.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="A1.T19.1.1.5.4.10" class="ltx_td ltx_align_right" style="background-color:#FDF9F9;"><span id="A1.T19.1.1.5.4.10.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="A1.T19.1.1.5.4.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E0F3E9;"><span id="A1.T19.1.1.5.4.11.1" class="ltx_text" style="background-color:#E0F3E9;">1.2%</span></td>
<td id="A1.T19.1.1.5.4.12" class="ltx_td ltx_align_right" style="background-color:#FBFEFC;"><span id="A1.T19.1.1.5.4.12.1" class="ltx_text" style="background-color:#FBFEFC;">0.2%</span></td>
<td id="A1.T19.1.1.5.4.13" class="ltx_td ltx_align_right" style="background-color:#FEFFFF;"><span id="A1.T19.1.1.5.4.13.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.5.4.14" class="ltx_td ltx_align_right" style="background-color:#F6FCF9;"><span id="A1.T19.1.1.5.4.14.1" class="ltx_text" style="background-color:#F6FCF9;">0.4%</span></td>
</tr>
<tr id="A1.T19.1.1.6.5" class="ltx_tr">
<td id="A1.T19.1.1.6.5.1" class="ltx_td"></td>
<td id="A1.T19.1.1.6.5.2" class="ltx_td ltx_align_left ltx_border_r">W4-g</td>
<td id="A1.T19.1.1.6.5.3" class="ltx_td ltx_align_right" style="background-color:#FEFAF9;"><span id="A1.T19.1.1.6.5.3.1" class="ltx_text" style="background-color:#FEFAF9;">-0.2%</span></td>
<td id="A1.T19.1.1.6.5.4" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.6.5.4.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.6.5.5" class="ltx_td ltx_align_right" style="background-color:#F8FCFA;"><span id="A1.T19.1.1.6.5.5.1" class="ltx_text" style="background-color:#F8FCFA;">0.3%</span></td>
<td id="A1.T19.1.1.6.5.6" class="ltx_td ltx_align_right" style="background-color:#F6FCF9;"><span id="A1.T19.1.1.6.5.6.1" class="ltx_text" style="background-color:#F6FCF9;">0.4%</span></td>
<td id="A1.T19.1.1.6.5.7" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.6.5.7.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.6.5.8" class="ltx_td ltx_align_right" style="background-color:#F8DFDC;"><span id="A1.T19.1.1.6.5.8.1" class="ltx_text" style="background-color:#F8DFDC;">-1.2%</span></td>
<td id="A1.T19.1.1.6.5.9" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.6.5.9.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.6.5.10" class="ltx_td ltx_align_right" style="background-color:#FDF9F9;"><span id="A1.T19.1.1.6.5.10.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="A1.T19.1.1.6.5.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E5F5ED;"><span id="A1.T19.1.1.6.5.11.1" class="ltx_text" style="background-color:#E5F5ED;">1.0%</span></td>
<td id="A1.T19.1.1.6.5.12" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.6.5.12.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.6.5.13" class="ltx_td ltx_align_right" style="background-color:#FDFEFE;"><span id="A1.T19.1.1.6.5.13.1" class="ltx_text" style="background-color:#FDFEFE;">0.1%</span></td>
<td id="A1.T19.1.1.6.5.14" class="ltx_td ltx_align_right" style="background-color:#FEFCFC;"><span id="A1.T19.1.1.6.5.14.1" class="ltx_text" style="background-color:#FEFCFC;">-0.1%</span></td>
</tr>
<tr id="A1.T19.1.1.7.6" class="ltx_tr">
<td id="A1.T19.1.1.7.6.1" class="ltx_td"></td>
<td id="A1.T19.1.1.7.6.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T19.1.1.7.6.3" class="ltx_td ltx_align_right" style="background-color:#FEFFFF;"><span id="A1.T19.1.1.7.6.3.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.7.6.4" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.7.6.4.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.7.6.5" class="ltx_td ltx_align_right" style="background-color:#FEFFFE;"><span id="A1.T19.1.1.7.6.5.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="A1.T19.1.1.7.6.6" class="ltx_td ltx_align_right" style="background-color:#FCFEFD;"><span id="A1.T19.1.1.7.6.6.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T19.1.1.7.6.7" class="ltx_td ltx_align_right" style="background-color:#FDF9F9;"><span id="A1.T19.1.1.7.6.7.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="A1.T19.1.1.7.6.8" class="ltx_td ltx_align_right" style="background-color:#E5F5ED;"><span id="A1.T19.1.1.7.6.8.1" class="ltx_text" style="background-color:#E5F5ED;">1.0%</span></td>
<td id="A1.T19.1.1.7.6.9" class="ltx_td ltx_align_right" style="background-color:#FAE4E2;"><span id="A1.T19.1.1.7.6.9.1" class="ltx_text" style="background-color:#FAE4E2;">-1.0%</span></td>
<td id="A1.T19.1.1.7.6.10" class="ltx_td ltx_align_right" style="background-color:#F0F9F5;"><span id="A1.T19.1.1.7.6.10.1" class="ltx_text" style="background-color:#F0F9F5;">0.6%</span></td>
<td id="A1.T19.1.1.7.6.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F2FAF6;"><span id="A1.T19.1.1.7.6.11.1" class="ltx_text" style="background-color:#F2FAF6;">0.5%</span></td>
<td id="A1.T19.1.1.7.6.12" class="ltx_td ltx_align_right" style="background-color:#FCFEFD;"><span id="A1.T19.1.1.7.6.12.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T19.1.1.7.6.13" class="ltx_td ltx_align_right" style="background-color:#FCFEFD;"><span id="A1.T19.1.1.7.6.13.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T19.1.1.7.6.14" class="ltx_td ltx_align_right" style="background-color:#FCFEFD;"><span id="A1.T19.1.1.7.6.14.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
</tr>
<tr id="A1.T19.1.1.8.7" class="ltx_tr">
<td id="A1.T19.1.1.8.7.1" class="ltx_td ltx_align_left ltx_border_t">35B</td>
<td id="A1.T19.1.1.8.7.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T19.1.1.8.7.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F1FAF6;"><span id="A1.T19.1.1.8.7.3.1" class="ltx_text" style="background-color:#F1FAF6;">0.5%</span></td>
<td id="A1.T19.1.1.8.7.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.8.7.4.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.8.7.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.8.7.5.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.8.7.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F6;"><span id="A1.T19.1.1.8.7.6.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
<td id="A1.T19.1.1.8.7.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F5FBF8;"><span id="A1.T19.1.1.8.7.7.1" class="ltx_text" style="background-color:#F5FBF8;">0.4%</span></td>
<td id="A1.T19.1.1.8.7.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAFDFC;"><span id="A1.T19.1.1.8.7.8.1" class="ltx_text" style="background-color:#FAFDFC;">0.2%</span></td>
<td id="A1.T19.1.1.8.7.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.8.7.9.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.8.7.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8FCFA;"><span id="A1.T19.1.1.8.7.10.1" class="ltx_text" style="background-color:#F8FCFA;">0.3%</span></td>
<td id="A1.T19.1.1.8.7.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FDF6F6;"><span id="A1.T19.1.1.8.7.11.1" class="ltx_text" style="background-color:#FDF6F6;">-0.3%</span></td>
<td id="A1.T19.1.1.8.7.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDFFFE;"><span id="A1.T19.1.1.8.7.12.1" class="ltx_text" style="background-color:#FDFFFE;">0.1%</span></td>
<td id="A1.T19.1.1.8.7.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDFFFE;"><span id="A1.T19.1.1.8.7.13.1" class="ltx_text" style="background-color:#FDFFFE;">0.1%</span></td>
<td id="A1.T19.1.1.8.7.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDFEFE;"><span id="A1.T19.1.1.8.7.14.1" class="ltx_text" style="background-color:#FDFEFE;">0.1%</span></td>
</tr>
<tr id="A1.T19.1.1.9.8" class="ltx_tr">
<td id="A1.T19.1.1.9.8.1" class="ltx_td"></td>
<td id="A1.T19.1.1.9.8.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T19.1.1.9.8.3" class="ltx_td ltx_align_right" style="background-color:#EDF8F3;"><span id="A1.T19.1.1.9.8.3.1" class="ltx_text" style="background-color:#EDF8F3;">0.7%</span></td>
<td id="A1.T19.1.1.9.8.4" class="ltx_td ltx_align_right" style="background-color:#E5F5ED;"><span id="A1.T19.1.1.9.8.4.1" class="ltx_text" style="background-color:#E5F5ED;">1.0%</span></td>
<td id="A1.T19.1.1.9.8.5" class="ltx_td ltx_align_right" style="background-color:#FAE6E4;"><span id="A1.T19.1.1.9.8.5.1" class="ltx_text" style="background-color:#FAE6E4;">-0.9%</span></td>
<td id="A1.T19.1.1.9.8.6" class="ltx_td ltx_align_right" style="background-color:#FDF7F6;"><span id="A1.T19.1.1.9.8.6.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
<td id="A1.T19.1.1.9.8.7" class="ltx_td ltx_align_right" style="background-color:#F5FBF8;"><span id="A1.T19.1.1.9.8.7.1" class="ltx_text" style="background-color:#F5FBF8;">0.4%</span></td>
<td id="A1.T19.1.1.9.8.8" class="ltx_td ltx_align_right" style="background-color:#F5FBF8;"><span id="A1.T19.1.1.9.8.8.1" class="ltx_text" style="background-color:#F5FBF8;">0.4%</span></td>
<td id="A1.T19.1.1.9.8.9" class="ltx_td ltx_align_right" style="background-color:#E5F5ED;"><span id="A1.T19.1.1.9.8.9.1" class="ltx_text" style="background-color:#E5F5ED;">1.0%</span></td>
<td id="A1.T19.1.1.9.8.10" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.9.8.10.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.9.8.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#EDF8F3;"><span id="A1.T19.1.1.9.8.11.1" class="ltx_text" style="background-color:#EDF8F3;">0.7%</span></td>
<td id="A1.T19.1.1.9.8.12" class="ltx_td ltx_align_right" style="background-color:#F7FCF9;"><span id="A1.T19.1.1.9.8.12.1" class="ltx_text" style="background-color:#F7FCF9;">0.3%</span></td>
<td id="A1.T19.1.1.9.8.13" class="ltx_td ltx_align_right" style="background-color:#FFFFFF;"><span id="A1.T19.1.1.9.8.13.1" class="ltx_text" style="background-color:#FFFFFF;">0.0%</span></td>
<td id="A1.T19.1.1.9.8.14" class="ltx_td ltx_align_right" style="background-color:#EDF8F3;"><span id="A1.T19.1.1.9.8.14.1" class="ltx_text" style="background-color:#EDF8F3;">0.7%</span></td>
</tr>
<tr id="A1.T19.1.1.10.9" class="ltx_tr">
<td id="A1.T19.1.1.10.9.1" class="ltx_td"></td>
<td id="A1.T19.1.1.10.9.2" class="ltx_td ltx_align_left ltx_border_r">W4-g</td>
<td id="A1.T19.1.1.10.9.3" class="ltx_td ltx_align_right" style="background-color:#F9FDFB;"><span id="A1.T19.1.1.10.9.3.1" class="ltx_text" style="background-color:#F9FDFB;">0.2%</span></td>
<td id="A1.T19.1.1.10.9.4" class="ltx_td ltx_align_right" style="background-color:#F4C8C5;"><span id="A1.T19.1.1.10.9.4.1" class="ltx_text" style="background-color:#F4C8C5;">-2.1%</span></td>
<td id="A1.T19.1.1.10.9.5" class="ltx_td ltx_align_right" style="background-color:#F7D5D3;"><span id="A1.T19.1.1.10.9.5.1" class="ltx_text" style="background-color:#F7D5D3;">-1.6%</span></td>
<td id="A1.T19.1.1.10.9.6" class="ltx_td ltx_align_right" style="background-color:#EDF8F3;"><span id="A1.T19.1.1.10.9.6.1" class="ltx_text" style="background-color:#EDF8F3;">0.7%</span></td>
<td id="A1.T19.1.1.10.9.7" class="ltx_td ltx_align_right" style="background-color:#F5FBF8;"><span id="A1.T19.1.1.10.9.7.1" class="ltx_text" style="background-color:#F5FBF8;">0.4%</span></td>
<td id="A1.T19.1.1.10.9.8" class="ltx_td ltx_align_right" style="background-color:#FAFDFC;"><span id="A1.T19.1.1.10.9.8.1" class="ltx_text" style="background-color:#FAFDFC;">0.2%</span></td>
<td id="A1.T19.1.1.10.9.9" class="ltx_td ltx_align_right" style="background-color:#F4CAC6;"><span id="A1.T19.1.1.10.9.9.1" class="ltx_text" style="background-color:#F4CAC6;">-2.0%</span></td>
<td id="A1.T19.1.1.10.9.10" class="ltx_td ltx_align_right" style="background-color:#FBECEB;"><span id="A1.T19.1.1.10.9.10.1" class="ltx_text" style="background-color:#FBECEB;">-0.7%</span></td>
<td id="A1.T19.1.1.10.9.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E8F6EF;"><span id="A1.T19.1.1.10.9.11.1" class="ltx_text" style="background-color:#E8F6EF;">0.9%</span></td>
<td id="A1.T19.1.1.10.9.12" class="ltx_td ltx_align_right" style="background-color:#FCF3F2;"><span id="A1.T19.1.1.10.9.12.1" class="ltx_text" style="background-color:#FCF3F2;">-0.4%</span></td>
<td id="A1.T19.1.1.10.9.13" class="ltx_td ltx_align_right" style="background-color:#FBEEEC;"><span id="A1.T19.1.1.10.9.13.1" class="ltx_text" style="background-color:#FBEEEC;">-0.6%</span></td>
<td id="A1.T19.1.1.10.9.14" class="ltx_td ltx_align_right" style="background-color:#FEFAFA;"><span id="A1.T19.1.1.10.9.14.1" class="ltx_text" style="background-color:#FEFAFA;">-0.2%</span></td>
</tr>
<tr id="A1.T19.1.1.11.10" class="ltx_tr">
<td id="A1.T19.1.1.11.10.1" class="ltx_td ltx_border_tt ltx_border_tt"></td>
<td id="A1.T19.1.1.11.10.2" class="ltx_td ltx_border_tt ltx_border_tt"></td>
<td id="A1.T19.1.1.11.10.3" class="ltx_td ltx_align_center ltx_border_tt ltx_border_tt" colspan="12"><span id="A1.T19.1.1.11.10.3.1" class="ltx_text ltx_font_bold">Cross-lingual</span></td>
</tr>
<tr id="A1.T19.1.1.12.11" class="ltx_tr">
<td id="A1.T19.1.1.12.11.1" class="ltx_td ltx_align_left">103B</td>
<td id="A1.T19.1.1.12.11.2" class="ltx_td ltx_align_left ltx_border_r">W8</td>
<td id="A1.T19.1.1.12.11.3" class="ltx_td ltx_align_right" style="background-color:#FCF3F2;"><span id="A1.T19.1.1.12.11.3.1" class="ltx_text" style="background-color:#FCF3F2;">-0.5%</span></td>
<td id="A1.T19.1.1.12.11.4" class="ltx_td ltx_align_right" style="background-color:#F2FAF6;"><span id="A1.T19.1.1.12.11.4.1" class="ltx_text" style="background-color:#F2FAF6;">0.5%</span></td>
<td id="A1.T19.1.1.12.11.5" class="ltx_td ltx_align_right" style="background-color:#F2FAF6;"><span id="A1.T19.1.1.12.11.5.1" class="ltx_text" style="background-color:#F2FAF6;">0.5%</span></td>
<td id="A1.T19.1.1.12.11.6" class="ltx_td ltx_align_right" style="background-color:#FEF9F9;"><span id="A1.T19.1.1.12.11.6.1" class="ltx_text" style="background-color:#FEF9F9;">-0.2%</span></td>
<td id="A1.T19.1.1.12.11.7" class="ltx_td ltx_align_right" style="background-color:#FEFCFB;"><span id="A1.T19.1.1.12.11.7.1" class="ltx_text" style="background-color:#FEFCFB;">-0.1%</span></td>
<td id="A1.T19.1.1.12.11.8" class="ltx_td ltx_align_right" style="background-color:#FDF6F5;"><span id="A1.T19.1.1.12.11.8.1" class="ltx_text" style="background-color:#FDF6F5;">-0.3%</span></td>
<td id="A1.T19.1.1.12.11.9" class="ltx_td ltx_align_right" style="background-color:#F8FDFA;"><span id="A1.T19.1.1.12.11.9.1" class="ltx_text" style="background-color:#F8FDFA;">0.3%</span></td>
<td id="A1.T19.1.1.12.11.10" class="ltx_td ltx_align_right" style="background-color:#FAE5E3;"><span id="A1.T19.1.1.12.11.10.1" class="ltx_text" style="background-color:#FAE5E3;">-1.0%</span></td>
<td id="A1.T19.1.1.12.11.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F6CFCC;"><span id="A1.T19.1.1.12.11.11.1" class="ltx_text" style="background-color:#F6CFCC;">-1.8%</span></td>
<td id="A1.T19.1.1.12.11.12" class="ltx_td ltx_align_right" style="background-color:#FDF7F7;"><span id="A1.T19.1.1.12.11.12.1" class="ltx_text" style="background-color:#FDF7F7;">-0.3%</span></td>
<td id="A1.T19.1.1.12.11.13" class="ltx_td ltx_align_right" style="background-color:#FEFDFD;"><span id="A1.T19.1.1.12.11.13.1" class="ltx_text" style="background-color:#FEFDFD;">0.0%</span></td>
<td id="A1.T19.1.1.12.11.14" class="ltx_td ltx_align_right" style="background-color:#FCF0EF;"><span id="A1.T19.1.1.12.11.14.1" class="ltx_text" style="background-color:#FCF0EF;">-0.6%</span></td>
</tr>
<tr id="A1.T19.1.1.13.12" class="ltx_tr">
<td id="A1.T19.1.1.13.12.1" class="ltx_td"></td>
<td id="A1.T19.1.1.13.12.2" class="ltx_td ltx_align_left ltx_border_r">W8A8-sq</td>
<td id="A1.T19.1.1.13.12.3" class="ltx_td ltx_align_right" style="background-color:#F8FDFA;"><span id="A1.T19.1.1.13.12.3.1" class="ltx_text" style="background-color:#F8FDFA;">0.3%</span></td>
<td id="A1.T19.1.1.13.12.4" class="ltx_td ltx_align_right" style="background-color:#E5F5ED;"><span id="A1.T19.1.1.13.12.4.1" class="ltx_text" style="background-color:#E5F5ED;">1.0%</span></td>
<td id="A1.T19.1.1.13.12.5" class="ltx_td ltx_align_right" style="background-color:#FCFEFD;"><span id="A1.T19.1.1.13.12.5.1" class="ltx_text" style="background-color:#FCFEFD;">0.2%</span></td>
<td id="A1.T19.1.1.13.12.6" class="ltx_td ltx_align_right" style="background-color:#E9F6F0;"><span id="A1.T19.1.1.13.12.6.1" class="ltx_text" style="background-color:#E9F6F0;">0.9%</span></td>
<td id="A1.T19.1.1.13.12.7" class="ltx_td ltx_align_right" style="background-color:#FAE6E4;"><span id="A1.T19.1.1.13.12.7.1" class="ltx_text" style="background-color:#FAE6E4;">-0.9%</span></td>
<td id="A1.T19.1.1.13.12.8" class="ltx_td ltx_align_right" style="background-color:#FBFDFC;"><span id="A1.T19.1.1.13.12.8.1" class="ltx_text" style="background-color:#FBFDFC;">0.2%</span></td>
<td id="A1.T19.1.1.13.12.9" class="ltx_td ltx_align_right" style="background-color:#E3F4EC;"><span id="A1.T19.1.1.13.12.9.1" class="ltx_text" style="background-color:#E3F4EC;">1.1%</span></td>
<td id="A1.T19.1.1.13.12.10" class="ltx_td ltx_align_right" style="background-color:#FBEAE8;"><span id="A1.T19.1.1.13.12.10.1" class="ltx_text" style="background-color:#FBEAE8;">-0.8%</span></td>
<td id="A1.T19.1.1.13.12.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F7DAD7;"><span id="A1.T19.1.1.13.12.11.1" class="ltx_text" style="background-color:#F7DAD7;">-1.4%</span></td>
<td id="A1.T19.1.1.13.12.12" class="ltx_td ltx_align_right" style="background-color:#FEFFFF;"><span id="A1.T19.1.1.13.12.12.1" class="ltx_text" style="background-color:#FEFFFF;">0.1%</span></td>
<td id="A1.T19.1.1.13.12.13" class="ltx_td ltx_align_right" style="background-color:#FEFFFE;"><span id="A1.T19.1.1.13.12.13.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="A1.T19.1.1.13.12.14" class="ltx_td ltx_align_right" style="background-color:#FEFFFF;"><span id="A1.T19.1.1.13.12.14.1" class="ltx_text" style="background-color:#FEFFFF;">0.0%</span></td>
</tr>
<tr id="A1.T19.1.1.14.13" class="ltx_tr">
<td id="A1.T19.1.1.14.13.1" class="ltx_td"></td>
<td id="A1.T19.1.1.14.13.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T19.1.1.14.13.3" class="ltx_td ltx_align_right" style="background-color:#F5CECB;"><span id="A1.T19.1.1.14.13.3.1" class="ltx_text" style="background-color:#F5CECB;">-1.8%</span></td>
<td id="A1.T19.1.1.14.13.4" class="ltx_td ltx_align_right" style="background-color:#F7D8D6;"><span id="A1.T19.1.1.14.13.4.1" class="ltx_text" style="background-color:#F7D8D6;">-1.5%</span></td>
<td id="A1.T19.1.1.14.13.5" class="ltx_td ltx_align_right" style="background-color:#FDF8F8;"><span id="A1.T19.1.1.14.13.5.1" class="ltx_text" style="background-color:#FDF8F8;">-0.2%</span></td>
<td id="A1.T19.1.1.14.13.6" class="ltx_td ltx_align_right" style="background-color:#FBEEED;"><span id="A1.T19.1.1.14.13.6.1" class="ltx_text" style="background-color:#FBEEED;">-0.6%</span></td>
<td id="A1.T19.1.1.14.13.7" class="ltx_td ltx_align_right" style="background-color:#F8DEDC;"><span id="A1.T19.1.1.14.13.7.1" class="ltx_text" style="background-color:#F8DEDC;">-1.2%</span></td>
<td id="A1.T19.1.1.14.13.8" class="ltx_td ltx_align_right" style="background-color:#FEFBFA;"><span id="A1.T19.1.1.14.13.8.1" class="ltx_text" style="background-color:#FEFBFA;">-0.2%</span></td>
<td id="A1.T19.1.1.14.13.9" class="ltx_td ltx_align_right" style="background-color:#F5CCC8;"><span id="A1.T19.1.1.14.13.9.1" class="ltx_text" style="background-color:#F5CCC8;">-1.9%</span></td>
<td id="A1.T19.1.1.14.13.10" class="ltx_td ltx_align_right" style="background-color:#F9E2E0;"><span id="A1.T19.1.1.14.13.10.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="A1.T19.1.1.14.13.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F3C3BF;"><span id="A1.T19.1.1.14.13.11.1" class="ltx_text" style="background-color:#F3C3BF;">-2.3%</span></td>
<td id="A1.T19.1.1.14.13.12" class="ltx_td ltx_align_right" style="background-color:#F8DFDD;"><span id="A1.T19.1.1.14.13.12.1" class="ltx_text" style="background-color:#F8DFDD;">-1.2%</span></td>
<td id="A1.T19.1.1.14.13.13" class="ltx_td ltx_align_right" style="background-color:#FAE6E4;"><span id="A1.T19.1.1.14.13.13.1" class="ltx_text" style="background-color:#FAE6E4;">-0.9%</span></td>
<td id="A1.T19.1.1.14.13.14" class="ltx_td ltx_align_right" style="background-color:#F7D6D3;"><span id="A1.T19.1.1.14.13.14.1" class="ltx_text" style="background-color:#F7D6D3;">-1.6%</span></td>
</tr>
<tr id="A1.T19.1.1.15.14" class="ltx_tr">
<td id="A1.T19.1.1.15.14.1" class="ltx_td"></td>
<td id="A1.T19.1.1.15.14.2" class="ltx_td ltx_align_left ltx_border_r">W4-g</td>
<td id="A1.T19.1.1.15.14.3" class="ltx_td ltx_align_right" style="background-color:#F1BAB6;"><span id="A1.T19.1.1.15.14.3.1" class="ltx_text" style="background-color:#F1BAB6;">-2.6%</span></td>
<td id="A1.T19.1.1.15.14.4" class="ltx_td ltx_align_right" style="background-color:#FAE9E8;"><span id="A1.T19.1.1.15.14.4.1" class="ltx_text" style="background-color:#FAE9E8;">-0.8%</span></td>
<td id="A1.T19.1.1.15.14.5" class="ltx_td ltx_align_right" style="background-color:#FCEFEE;"><span id="A1.T19.1.1.15.14.5.1" class="ltx_text" style="background-color:#FCEFEE;">-0.6%</span></td>
<td id="A1.T19.1.1.15.14.6" class="ltx_td ltx_align_right" style="background-color:#FEFFFE;"><span id="A1.T19.1.1.15.14.6.1" class="ltx_text" style="background-color:#FEFFFE;">0.1%</span></td>
<td id="A1.T19.1.1.15.14.7" class="ltx_td ltx_align_right" style="background-color:#FCFEFD;"><span id="A1.T19.1.1.15.14.7.1" class="ltx_text" style="background-color:#FCFEFD;">0.1%</span></td>
<td id="A1.T19.1.1.15.14.8" class="ltx_td ltx_align_right" style="background-color:#FDF5F5;"><span id="A1.T19.1.1.15.14.8.1" class="ltx_text" style="background-color:#FDF5F5;">-0.3%</span></td>
<td id="A1.T19.1.1.15.14.9" class="ltx_td ltx_align_right" style="background-color:#FCF0EF;"><span id="A1.T19.1.1.15.14.9.1" class="ltx_text" style="background-color:#FCF0EF;">-0.5%</span></td>
<td id="A1.T19.1.1.15.14.10" class="ltx_td ltx_align_right" style="background-color:#EEA9A3;"><span id="A1.T19.1.1.15.14.10.1" class="ltx_text" style="background-color:#EEA9A3;">-3.3%</span></td>
<td id="A1.T19.1.1.15.14.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F5CECB;"><span id="A1.T19.1.1.15.14.11.1" class="ltx_text" style="background-color:#F5CECB;">-1.8%</span></td>
<td id="A1.T19.1.1.15.14.12" class="ltx_td ltx_align_right" style="background-color:#F9E2E0;"><span id="A1.T19.1.1.15.14.12.1" class="ltx_text" style="background-color:#F9E2E0;">-1.1%</span></td>
<td id="A1.T19.1.1.15.14.13" class="ltx_td ltx_align_right" style="background-color:#FAE7E5;"><span id="A1.T19.1.1.15.14.13.1" class="ltx_text" style="background-color:#FAE7E5;">-0.9%</span></td>
<td id="A1.T19.1.1.15.14.14" class="ltx_td ltx_align_right" style="background-color:#F8DCD9;"><span id="A1.T19.1.1.15.14.14.1" class="ltx_text" style="background-color:#F8DCD9;">-1.3%</span></td>
</tr>
<tr id="A1.T19.1.1.16.15" class="ltx_tr">
<td id="A1.T19.1.1.16.15.1" class="ltx_td"></td>
<td id="A1.T19.1.1.16.15.2" class="ltx_td ltx_align_left ltx_border_r">W4</td>
<td id="A1.T19.1.1.16.15.3" class="ltx_td ltx_align_right" style="background-color:#B1E0C9;"><span id="A1.T19.1.1.16.15.3.1" class="ltx_text" style="background-color:#B1E0C9;">3.0%</span></td>
<td id="A1.T19.1.1.16.15.4" class="ltx_td ltx_align_right" style="background-color:#97D5B7;"><span id="A1.T19.1.1.16.15.4.1" class="ltx_text" style="background-color:#97D5B7;">4.0%</span></td>
<td id="A1.T19.1.1.16.15.5" class="ltx_td ltx_align_right" style="background-color:#7BCAA4;"><span id="A1.T19.1.1.16.15.5.1" class="ltx_text" style="background-color:#7BCAA4;">5.1%</span></td>
<td id="A1.T19.1.1.16.15.6" class="ltx_td ltx_align_right" style="background-color:#C3E7D5;"><span id="A1.T19.1.1.16.15.6.1" class="ltx_text" style="background-color:#C3E7D5;">2.3%</span></td>
<td id="A1.T19.1.1.16.15.7" class="ltx_td ltx_align_right" style="background-color:#EFF9F4;"><span id="A1.T19.1.1.16.15.7.1" class="ltx_text" style="background-color:#EFF9F4;">0.6%</span></td>
<td id="A1.T19.1.1.16.15.8" class="ltx_td ltx_align_right" style="background-color:#FCF2F1;"><span id="A1.T19.1.1.16.15.8.1" class="ltx_text" style="background-color:#FCF2F1;">-0.5%</span></td>
<td id="A1.T19.1.1.16.15.9" class="ltx_td ltx_align_right" style="background-color:#F2BFBB;"><span id="A1.T19.1.1.16.15.9.1" class="ltx_text" style="background-color:#F2BFBB;">-2.4%</span></td>
<td id="A1.T19.1.1.16.15.10" class="ltx_td ltx_align_right" style="background-color:#BEE5D2;"><span id="A1.T19.1.1.16.15.10.1" class="ltx_text" style="background-color:#BEE5D2;">2.5%</span></td>
<td id="A1.T19.1.1.16.15.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F5CFCC;"><span id="A1.T19.1.1.16.15.11.1" class="ltx_text" style="background-color:#F5CFCC;">-1.8%</span></td>
<td id="A1.T19.1.1.16.15.12" class="ltx_td ltx_align_right" style="background-color:#DAF0E6;"><span id="A1.T19.1.1.16.15.12.1" class="ltx_text" style="background-color:#DAF0E6;">1.4%</span></td>
<td id="A1.T19.1.1.16.15.13" class="ltx_td ltx_align_right" style="background-color:#B4E1CB;"><span id="A1.T19.1.1.16.15.13.1" class="ltx_text" style="background-color:#B4E1CB;">2.9%</span></td>
<td id="A1.T19.1.1.16.15.14" class="ltx_td ltx_align_right" style="background-color:#FCF3F3;"><span id="A1.T19.1.1.16.15.14.1" class="ltx_text" style="background-color:#FCF3F3;">-0.4%</span></td>
</tr>
<tr id="A1.T19.1.1.17.16" class="ltx_tr">
<td id="A1.T19.1.1.17.16.1" class="ltx_td ltx_align_left ltx_border_t">35B</td>
<td id="A1.T19.1.1.17.16.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">W8</td>
<td id="A1.T19.1.1.17.16.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#D4EEE1;"><span id="A1.T19.1.1.17.16.3.1" class="ltx_text" style="background-color:#D4EEE1;">1.7%</span></td>
<td id="A1.T19.1.1.17.16.4" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F6D4D1;"><span id="A1.T19.1.1.17.16.4.1" class="ltx_text" style="background-color:#F6D4D1;">-1.6%</span></td>
<td id="A1.T19.1.1.17.16.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBFEFC;"><span id="A1.T19.1.1.17.16.5.1" class="ltx_text" style="background-color:#FBFEFC;">0.2%</span></td>
<td id="A1.T19.1.1.17.16.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF7F6;"><span id="A1.T19.1.1.17.16.6.1" class="ltx_text" style="background-color:#FDF7F6;">-0.3%</span></td>
<td id="A1.T19.1.1.17.16.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F4C9C5;"><span id="A1.T19.1.1.17.16.7.1" class="ltx_text" style="background-color:#F4C9C5;">-2.0%</span></td>
<td id="A1.T19.1.1.17.16.8" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#EAF7F0;"><span id="A1.T19.1.1.17.16.8.1" class="ltx_text" style="background-color:#EAF7F0;">0.8%</span></td>
<td id="A1.T19.1.1.17.16.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBEBE9;"><span id="A1.T19.1.1.17.16.9.1" class="ltx_text" style="background-color:#FBEBE9;">-0.8%</span></td>
<td id="A1.T19.1.1.17.16.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F8FDFA;"><span id="A1.T19.1.1.17.16.10.1" class="ltx_text" style="background-color:#F8FDFA;">0.3%</span></td>
<td id="A1.T19.1.1.17.16.11" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FEFAF9;"><span id="A1.T19.1.1.17.16.11.1" class="ltx_text" style="background-color:#FEFAF9;">-0.2%</span></td>
<td id="A1.T19.1.1.17.16.12" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FDF9F9;"><span id="A1.T19.1.1.17.16.12.1" class="ltx_text" style="background-color:#FDF9F9;">-0.2%</span></td>
<td id="A1.T19.1.1.17.16.13" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FBECEB;"><span id="A1.T19.1.1.17.16.13.1" class="ltx_text" style="background-color:#FBECEB;">-0.7%</span></td>
<td id="A1.T19.1.1.17.16.14" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F5FBF8;"><span id="A1.T19.1.1.17.16.14.1" class="ltx_text" style="background-color:#F5FBF8;">0.4%</span></td>
</tr>
<tr id="A1.T19.1.1.18.17" class="ltx_tr">
<td id="A1.T19.1.1.18.17.1" class="ltx_td"></td>
<td id="A1.T19.1.1.18.17.2" class="ltx_td ltx_align_left ltx_border_r">W8A8</td>
<td id="A1.T19.1.1.18.17.3" class="ltx_td ltx_align_right" style="background-color:#9ED8BC;"><span id="A1.T19.1.1.18.17.3.1" class="ltx_text" style="background-color:#9ED8BC;">3.8%</span></td>
<td id="A1.T19.1.1.18.17.4" class="ltx_td ltx_align_right" style="background-color:#57BB8A;"><span id="A1.T19.1.1.18.17.4.1" class="ltx_text" style="background-color:#57BB8A;">7.2%</span></td>
<td id="A1.T19.1.1.18.17.5" class="ltx_td ltx_align_right" style="background-color:#8ED2B1;"><span id="A1.T19.1.1.18.17.5.1" class="ltx_text" style="background-color:#8ED2B1;">4.4%</span></td>
<td id="A1.T19.1.1.18.17.6" class="ltx_td ltx_align_right" style="background-color:#B4E1CB;"><span id="A1.T19.1.1.18.17.6.1" class="ltx_text" style="background-color:#B4E1CB;">2.9%</span></td>
<td id="A1.T19.1.1.18.17.7" class="ltx_td ltx_align_right" style="background-color:#97D5B7;"><span id="A1.T19.1.1.18.17.7.1" class="ltx_text" style="background-color:#97D5B7;">4.1%</span></td>
<td id="A1.T19.1.1.18.17.8" class="ltx_td ltx_align_right" style="background-color:#AFDFC8;"><span id="A1.T19.1.1.18.17.8.1" class="ltx_text" style="background-color:#AFDFC8;">3.1%</span></td>
<td id="A1.T19.1.1.18.17.9" class="ltx_td ltx_align_right" style="background-color:#DFF2E9;"><span id="A1.T19.1.1.18.17.9.1" class="ltx_text" style="background-color:#DFF2E9;">1.2%</span></td>
<td id="A1.T19.1.1.18.17.10" class="ltx_td ltx_align_right" style="background-color:#7ECBA5;"><span id="A1.T19.1.1.18.17.10.1" class="ltx_text" style="background-color:#7ECBA5;">5.0%</span></td>
<td id="A1.T19.1.1.18.17.11" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#C5E8D7;"><span id="A1.T19.1.1.18.17.11.1" class="ltx_text" style="background-color:#C5E8D7;">2.3%</span></td>
<td id="A1.T19.1.1.18.17.12" class="ltx_td ltx_align_right" style="background-color:#9ED8BC;"><span id="A1.T19.1.1.18.17.12.1" class="ltx_text" style="background-color:#9ED8BC;">3.8%</span></td>
<td id="A1.T19.1.1.18.17.13" class="ltx_td ltx_align_right" style="background-color:#86CEAB;"><span id="A1.T19.1.1.18.17.13.1" class="ltx_text" style="background-color:#86CEAB;">4.7%</span></td>
<td id="A1.T19.1.1.18.17.14" class="ltx_td ltx_align_right" style="background-color:#BCE4D1;"><span id="A1.T19.1.1.18.17.14.1" class="ltx_text" style="background-color:#BCE4D1;">2.6%</span></td>
</tr>
<tr id="A1.T19.1.1.19.18" class="ltx_tr">
<td id="A1.T19.1.1.19.18.1" class="ltx_td ltx_border_bb"></td>
<td id="A1.T19.1.1.19.18.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">W4-g</td>
<td id="A1.T19.1.1.19.18.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#B7E2CD;"><span id="A1.T19.1.1.19.18.3.1" class="ltx_text" style="background-color:#B7E2CD;">2.8%</span></td>
<td id="A1.T19.1.1.19.18.4" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FCF2F1;"><span id="A1.T19.1.1.19.18.4.1" class="ltx_text" style="background-color:#FCF2F1;">-0.5%</span></td>
<td id="A1.T19.1.1.19.18.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#74C79F;"><span id="A1.T19.1.1.19.18.5.1" class="ltx_text" style="background-color:#74C79F;">5.4%</span></td>
<td id="A1.T19.1.1.19.18.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3FAF7;"><span id="A1.T19.1.1.19.18.6.1" class="ltx_text" style="background-color:#F3FAF7;">0.5%</span></td>
<td id="A1.T19.1.1.19.18.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#D0ECDE;"><span id="A1.T19.1.1.19.18.7.1" class="ltx_text" style="background-color:#D0ECDE;">1.9%</span></td>
<td id="A1.T19.1.1.19.18.8" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F8DDDB;"><span id="A1.T19.1.1.19.18.8.1" class="ltx_text" style="background-color:#F8DDDB;">-1.3%</span></td>
<td id="A1.T19.1.1.19.18.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#D3EDE0;"><span id="A1.T19.1.1.19.18.9.1" class="ltx_text" style="background-color:#D3EDE0;">1.7%</span></td>
<td id="A1.T19.1.1.19.18.10" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#EFF9F4;"><span id="A1.T19.1.1.19.18.10.1" class="ltx_text" style="background-color:#EFF9F4;">0.6%</span></td>
<td id="A1.T19.1.1.19.18.11" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F3C4C0;"><span id="A1.T19.1.1.19.18.11.1" class="ltx_text" style="background-color:#F3C4C0;">-2.2%</span></td>
<td id="A1.T19.1.1.19.18.12" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#E6F5EE;"><span id="A1.T19.1.1.19.18.12.1" class="ltx_text" style="background-color:#E6F5EE;">1.0%</span></td>
<td id="A1.T19.1.1.19.18.13" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#D7EFE3;"><span id="A1.T19.1.1.19.18.13.1" class="ltx_text" style="background-color:#D7EFE3;">1.6%</span></td>
<td id="A1.T19.1.1.19.18.14" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9FDFB;"><span id="A1.T19.1.1.19.18.14.1" class="ltx_text" style="background-color:#F9FDFB;">0.3%</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A19: </span><span id="A1.T19.4.1" class="ltx_text ltx_font_bold">Relative performance (%<math id="A1.T19.4.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="A1.T19.4.1.m1.1b"><mi mathvariant="normal" id="A1.T19.4.1.m1.1.1" xref="A1.T19.4.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="A1.T19.4.1.m1.1c"><ci id="A1.T19.4.1.m1.1.1.cmml" xref="A1.T19.4.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T19.4.1.m1.1d">\Delta</annotation></semantics></math>) vs. FP16 for Command Models on Language Confusion metrics</span>. Ltn/IE are Latin-script/Indo-European languages: de, es, fr, it, pt. <math id="A1.T19.5.m1.1" class="ltx_Math" alttext="\neg" display="inline"><semantics id="A1.T19.5.m1.1b"><mo id="A1.T19.5.m1.1.1" xref="A1.T19.5.m1.1.1.cmml">¬</mo><annotation-xml encoding="MathML-Content" id="A1.T19.5.m1.1c"><not id="A1.T19.5.m1.1.1.cmml" xref="A1.T19.5.m1.1.1"></not></annotation-xml><annotation encoding="application/x-tex" id="A1.T19.5.m1.1d">\neg</annotation></semantics></math> are the rest: ar, ja, ko, zh.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>RM/LLM-as-a-Judge and Human Evaluation - Full Results</h3>

<figure id="A1.T20" class="ltx_table">
<div id="A1.T20.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:209.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(30.5pt,-14.7pt) scale(1.16381850903297,1.16381850903297) ;">
<table id="A1.T20.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T20.1.1.1.1" class="ltx_tr">
<th id="A1.T20.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt"></th>
<th id="A1.T20.1.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt"></th>
<th id="A1.T20.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="2"><span id="A1.T20.1.1.1.1.3.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T20.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="2"><span id="A1.T20.1.1.1.1.4.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T20.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="2"><span id="A1.T20.1.1.1.1.5.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T20.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="A1.T20.1.1.1.1.6.1" class="ltx_text ltx_font_bold">ko</span></th>
</tr>
<tr id="A1.T20.1.1.2.2" class="ltx_tr">
<th id="A1.T20.1.1.2.2.1" class="ltx_td ltx_th ltx_th_column ltx_th_row"></th>
<th id="A1.T20.1.1.2.2.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r"></th>
<th id="A1.T20.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T20.1.1.2.2.3.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="A1.T20.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="A1.T20.1.1.2.2.4.1" class="ltx_text ltx_font_bold">RM</span></th>
<th id="A1.T20.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T20.1.1.2.2.5.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="A1.T20.1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="A1.T20.1.1.2.2.6.1" class="ltx_text ltx_font_bold">RM</span></th>
<th id="A1.T20.1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T20.1.1.2.2.7.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="A1.T20.1.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="A1.T20.1.1.2.2.8.1" class="ltx_text ltx_font_bold">RM</span></th>
<th id="A1.T20.1.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T20.1.1.2.2.9.1" class="ltx_text ltx_font_bold">LLM</span></th>
<th id="A1.T20.1.1.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T20.1.1.2.2.10.1" class="ltx_text ltx_font_bold">RM</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T20.1.1.3.1" class="ltx_tr">
<th id="A1.T20.1.1.3.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="A1.T20.1.1.3.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">W8</th>
<td id="A1.T20.1.1.3.1.3" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F3FAF7;"><span id="A1.T20.1.1.3.1.3.1" class="ltx_text" style="background-color:#F3FAF7;">50.5</span></td>
<td id="A1.T20.1.1.3.1.4" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#FEFCFC;"><span id="A1.T20.1.1.3.1.4.1" class="ltx_text" style="background-color:#FEFCFC;">49.7</span></td>
<td id="A1.T20.1.1.3.1.5" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#F7D6D3;"><span id="A1.T20.1.1.3.1.5.1" class="ltx_text" style="background-color:#F7D6D3;">44.9</span></td>
<td id="A1.T20.1.1.3.1.6" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#A1D9BE;"><span id="A1.T20.1.1.3.1.6.1" class="ltx_text" style="background-color:#A1D9BE;">53.7</span></td>
<td id="A1.T20.1.1.3.1.7" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE9E8;"><span id="A1.T20.1.1.3.1.7.1" class="ltx_text" style="background-color:#FAE9E8;">47.3</span></td>
<td id="A1.T20.1.1.3.1.8" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#BBE4D0;"><span id="A1.T20.1.1.3.1.8.1" class="ltx_text" style="background-color:#BBE4D0;">52.7</span></td>
<td id="A1.T20.1.1.3.1.9" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#A2D9BE;"><span id="A1.T20.1.1.3.1.9.1" class="ltx_text" style="background-color:#A2D9BE;">53.7</span></td>
<td id="A1.T20.1.1.3.1.10" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FAE8E6;"><span id="A1.T20.1.1.3.1.10.1" class="ltx_text" style="background-color:#FAE8E6;">47.1</span></td>
</tr>
<tr id="A1.T20.1.1.4.2" class="ltx_tr">
<th id="A1.T20.1.1.4.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T20.1.1.4.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="A1.T20.1.1.4.2.3" class="ltx_td ltx_align_right" style="background-color:#F1B6B1;"><span id="A1.T20.1.1.4.2.3.1" class="ltx_text" style="background-color:#F1B6B1;">40.8</span></td>
<td id="A1.T20.1.1.4.2.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FBEAE9;"><span id="A1.T20.1.1.4.2.4.1" class="ltx_text" style="background-color:#FBEAE9;">47.5</span></td>
<td id="A1.T20.1.1.4.2.5" class="ltx_td ltx_align_right" style="background-color:#FCF0EF;"><span id="A1.T20.1.1.4.2.5.1" class="ltx_text" style="background-color:#FCF0EF;">48.1</span></td>
<td id="A1.T20.1.1.4.2.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#CCEBDC;"><span id="A1.T20.1.1.4.2.6.1" class="ltx_text" style="background-color:#CCEBDC;">52.0</span></td>
<td id="A1.T20.1.1.4.2.7" class="ltx_td ltx_align_right" style="background-color:#E6F5EE;"><span id="A1.T20.1.1.4.2.7.1" class="ltx_text" style="background-color:#E6F5EE;">51.0</span></td>
<td id="A1.T20.1.1.4.2.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#C4E7D6;"><span id="A1.T20.1.1.4.2.8.1" class="ltx_text" style="background-color:#C4E7D6;">52.4</span></td>
<td id="A1.T20.1.1.4.2.9" class="ltx_td ltx_align_right" style="background-color:#D0ECDF;"><span id="A1.T20.1.1.4.2.9.1" class="ltx_text" style="background-color:#D0ECDF;">51.9</span></td>
<td id="A1.T20.1.1.4.2.10" class="ltx_td ltx_align_right" style="background-color:#FBEAE9;"><span id="A1.T20.1.1.4.2.10.1" class="ltx_text" style="background-color:#FBEAE9;">47.5</span></td>
</tr>
<tr id="A1.T20.1.1.5.3" class="ltx_tr">
<th id="A1.T20.1.1.5.3.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T20.1.1.5.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4-g</th>
<td id="A1.T20.1.1.5.3.3" class="ltx_td ltx_align_right" style="background-color:#F7D5D2;"><span id="A1.T20.1.1.5.3.3.1" class="ltx_text" style="background-color:#F7D5D2;">44.8</span></td>
<td id="A1.T20.1.1.5.3.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F2BBB6;"><span id="A1.T20.1.1.5.3.4.1" class="ltx_text" style="background-color:#F2BBB6;">41.5</span></td>
<td id="A1.T20.1.1.5.3.5" class="ltx_td ltx_align_right" style="background-color:#F2BDB8;"><span id="A1.T20.1.1.5.3.5.1" class="ltx_text" style="background-color:#F2BDB8;">41.7</span></td>
<td id="A1.T20.1.1.5.3.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E6F5EE;"><span id="A1.T20.1.1.5.3.6.1" class="ltx_text" style="background-color:#E6F5EE;">51.0</span></td>
<td id="A1.T20.1.1.5.3.7" class="ltx_td ltx_align_right" style="background-color:#F3C2BE;"><span id="A1.T20.1.1.5.3.7.1" class="ltx_text" style="background-color:#F3C2BE;">42.4</span></td>
<td id="A1.T20.1.1.5.3.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FFFFFF;"><span id="A1.T20.1.1.5.3.8.1" class="ltx_text" style="background-color:#FFFFFF;">50.0</span></td>
<td id="A1.T20.1.1.5.3.9" class="ltx_td ltx_align_right" style="background-color:#FAE8E6;"><span id="A1.T20.1.1.5.3.9.1" class="ltx_text" style="background-color:#FAE8E6;">47.1</span></td>
<td id="A1.T20.1.1.5.3.10" class="ltx_td ltx_align_right" style="background-color:#F3C0BC;"><span id="A1.T20.1.1.5.3.10.1" class="ltx_text" style="background-color:#F3C0BC;">42.2</span></td>
</tr>
<tr id="A1.T20.1.1.6.4" class="ltx_tr">
<th id="A1.T20.1.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T20.1.1.6.4.1.1" class="ltx_text">Internal</span></th>
<th id="A1.T20.1.1.6.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4</th>
<td id="A1.T20.1.1.6.4.3" class="ltx_td ltx_align_right" style="background-color:#E8877F;"><span id="A1.T20.1.1.6.4.3.1" class="ltx_text" style="background-color:#E8877F;">34.9</span></td>
<td id="A1.T20.1.1.6.4.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#EFAEA8;"><span id="A1.T20.1.1.6.4.4.1" class="ltx_text" style="background-color:#EFAEA8;">39.8</span></td>
<td id="A1.T20.1.1.6.4.5" class="ltx_td ltx_align_right" style="background-color:#E67C73;"><span id="A1.T20.1.1.6.4.5.1" class="ltx_text" style="background-color:#E67C73;">33.5</span></td>
<td id="A1.T20.1.1.6.4.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F2BBB6;"><span id="A1.T20.1.1.6.4.6.1" class="ltx_text" style="background-color:#F2BBB6;">41.5</span></td>
<td id="A1.T20.1.1.6.4.7" class="ltx_td ltx_align_right" style="background-color:#EEA8A2;"><span id="A1.T20.1.1.6.4.7.1" class="ltx_text" style="background-color:#EEA8A2;">39.2</span></td>
<td id="A1.T20.1.1.6.4.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#EFAFAA;"><span id="A1.T20.1.1.6.4.8.1" class="ltx_text" style="background-color:#EFAFAA;">40.0</span></td>
<td id="A1.T20.1.1.6.4.9" class="ltx_td ltx_align_right" style="background-color:#F0B5AF;"><span id="A1.T20.1.1.6.4.9.1" class="ltx_text" style="background-color:#F0B5AF;">40.7</span></td>
<td id="A1.T20.1.1.6.4.10" class="ltx_td ltx_align_right" style="background-color:#EA918A;"><span id="A1.T20.1.1.6.4.10.1" class="ltx_text" style="background-color:#EA918A;">36.2</span></td>
</tr>
<tr id="A1.T20.1.1.7.5" class="ltx_tr">
<th id="A1.T20.1.1.7.5.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A1.T20.1.1.7.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_tt">W8</th>
<td id="A1.T20.1.1.7.5.3" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FDF9F9;"><span id="A1.T20.1.1.7.5.3.1" class="ltx_text" style="background-color:#FDF9F9;">49.3</span></td>
<td id="A1.T20.1.1.7.5.4" class="ltx_td ltx_align_right ltx_border_r ltx_border_tt" style="background-color:#E6F5EE;"><span id="A1.T20.1.1.7.5.4.1" class="ltx_text" style="background-color:#E6F5EE;">51.0</span></td>
<td id="A1.T20.1.1.7.5.5" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#A3DABF;"><span id="A1.T20.1.1.7.5.5.1" class="ltx_text" style="background-color:#A3DABF;">53.7</span></td>
<td id="A1.T20.1.1.7.5.6" class="ltx_td ltx_align_right ltx_border_r ltx_border_tt" style="background-color:#FBEFEE;"><span id="A1.T20.1.1.7.5.6.1" class="ltx_text" style="background-color:#FBEFEE;">48.0</span></td>
<td id="A1.T20.1.1.7.5.7" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#FAE7E5;"><span id="A1.T20.1.1.7.5.7.1" class="ltx_text" style="background-color:#FAE7E5;">47.0</span></td>
<td id="A1.T20.1.1.7.5.8" class="ltx_td ltx_align_right ltx_border_r ltx_border_tt" style="background-color:#FAE9E8;"><span id="A1.T20.1.1.7.5.8.1" class="ltx_text" style="background-color:#FAE9E8;">47.3</span></td>
<td id="A1.T20.1.1.7.5.9" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#DEF2E8;"><span id="A1.T20.1.1.7.5.9.1" class="ltx_text" style="background-color:#DEF2E8;">51.3</span></td>
<td id="A1.T20.1.1.7.5.10" class="ltx_td ltx_align_right ltx_border_tt" style="background-color:#E6F5EE;"><span id="A1.T20.1.1.7.5.10.1" class="ltx_text" style="background-color:#E6F5EE;">51.0</span></td>
</tr>
<tr id="A1.T20.1.1.8.6" class="ltx_tr">
<th id="A1.T20.1.1.8.6.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T20.1.1.8.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="A1.T20.1.1.8.6.3" class="ltx_td ltx_align_right" style="background-color:#F3C2BD;"><span id="A1.T20.1.1.8.6.3.1" class="ltx_text" style="background-color:#F3C2BD;">42.3</span></td>
<td id="A1.T20.1.1.8.6.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F8DCDA;"><span id="A1.T20.1.1.8.6.4.1" class="ltx_text" style="background-color:#F8DCDA;">45.7</span></td>
<td id="A1.T20.1.1.8.6.5" class="ltx_td ltx_align_right" style="background-color:#92D3B4;"><span id="A1.T20.1.1.8.6.5.1" class="ltx_text" style="background-color:#92D3B4;">54.3</span></td>
<td id="A1.T20.1.1.8.6.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F8DFDD;"><span id="A1.T20.1.1.8.6.6.1" class="ltx_text" style="background-color:#F8DFDD;">46.0</span></td>
<td id="A1.T20.1.1.8.6.7" class="ltx_td ltx_align_right" style="background-color:#FDF9F9;"><span id="A1.T20.1.1.8.6.7.1" class="ltx_text" style="background-color:#FDF9F9;">49.3</span></td>
<td id="A1.T20.1.1.8.6.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#EFF9F4;"><span id="A1.T20.1.1.8.6.8.1" class="ltx_text" style="background-color:#EFF9F4;">50.7</span></td>
<td id="A1.T20.1.1.8.6.9" class="ltx_td ltx_align_right" style="background-color:#F8DFDD;"><span id="A1.T20.1.1.8.6.9.1" class="ltx_text" style="background-color:#F8DFDD;">46.0</span></td>
<td id="A1.T20.1.1.8.6.10" class="ltx_td ltx_align_right" style="background-color:#FBECEB;"><span id="A1.T20.1.1.8.6.10.1" class="ltx_text" style="background-color:#FBECEB;">47.7</span></td>
</tr>
<tr id="A1.T20.1.1.9.7" class="ltx_tr">
<th id="A1.T20.1.1.9.7.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T20.1.1.9.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8</th>
<td id="A1.T20.1.1.9.7.3" class="ltx_td ltx_align_right" style="background-color:#FCF1F0;"><span id="A1.T20.1.1.9.7.3.1" class="ltx_text" style="background-color:#FCF1F0;">48.3</span></td>
<td id="A1.T20.1.1.9.7.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#DEF2E8;"><span id="A1.T20.1.1.9.7.4.1" class="ltx_text" style="background-color:#DEF2E8;">51.3</span></td>
<td id="A1.T20.1.1.9.7.5" class="ltx_td ltx_align_right" style="background-color:#57BB8A;"><span id="A1.T20.1.1.9.7.5.1" class="ltx_text" style="background-color:#57BB8A;">56.7</span></td>
<td id="A1.T20.1.1.9.7.6" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FCF1F0;"><span id="A1.T20.1.1.9.7.6.1" class="ltx_text" style="background-color:#FCF1F0;">48.3</span></td>
<td id="A1.T20.1.1.9.7.7" class="ltx_td ltx_align_right" style="background-color:#DEF2E8;"><span id="A1.T20.1.1.9.7.7.1" class="ltx_text" style="background-color:#DEF2E8;">51.3</span></td>
<td id="A1.T20.1.1.9.7.8" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FDF9F9;"><span id="A1.T20.1.1.9.7.8.1" class="ltx_text" style="background-color:#FDF9F9;">49.3</span></td>
<td id="A1.T20.1.1.9.7.9" class="ltx_td ltx_align_right" style="background-color:#BCE4D1;"><span id="A1.T20.1.1.9.7.9.1" class="ltx_text" style="background-color:#BCE4D1;">52.7</span></td>
<td id="A1.T20.1.1.9.7.10" class="ltx_td ltx_align_right" style="background-color:#FCF1F0;"><span id="A1.T20.1.1.9.7.10.1" class="ltx_text" style="background-color:#FCF1F0;">48.3</span></td>
</tr>
<tr id="A1.T20.1.1.10.8" class="ltx_tr">
<th id="A1.T20.1.1.10.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A1.T20.1.1.10.8.1.1" class="ltx_text">Dolly</span></th>
<th id="A1.T20.1.1.10.8.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">W4-g</th>
<td id="A1.T20.1.1.10.8.3" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F9E1DF;"><span id="A1.T20.1.1.10.8.3.1" class="ltx_text" style="background-color:#F9E1DF;">46.3</span></td>
<td id="A1.T20.1.1.10.8.4" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#FCF4F3;"><span id="A1.T20.1.1.10.8.4.1" class="ltx_text" style="background-color:#FCF4F3;">48.7</span></td>
<td id="A1.T20.1.1.10.8.5" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FBEFEE;"><span id="A1.T20.1.1.10.8.5.1" class="ltx_text" style="background-color:#FBEFEE;">48.0</span></td>
<td id="A1.T20.1.1.10.8.6" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#C5E8D7;"><span id="A1.T20.1.1.10.8.6.1" class="ltx_text" style="background-color:#C5E8D7;">52.3</span></td>
<td id="A1.T20.1.1.10.8.7" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F3C2BD;"><span id="A1.T20.1.1.10.8.7.1" class="ltx_text" style="background-color:#F3C2BD;">42.3</span></td>
<td id="A1.T20.1.1.10.8.8" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F3C2BD;"><span id="A1.T20.1.1.10.8.8.1" class="ltx_text" style="background-color:#F3C2BD;">42.3</span></td>
<td id="A1.T20.1.1.10.8.9" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F6D1CE;"><span id="A1.T20.1.1.10.8.9.1" class="ltx_text" style="background-color:#F6D1CE;">44.3</span></td>
<td id="A1.T20.1.1.10.8.10" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#FAE9E8;"><span id="A1.T20.1.1.10.8.10.1" class="ltx_text" style="background-color:#FAE9E8;">47.3</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A20: </span><span id="A1.T20.5.1" class="ltx_text ltx_font_bold ltx_font_italic">LLM/RM-as-a-Judge<span id="A1.T20.5.1.1" class="ltx_text ltx_font_upright"> Raw win-rates of 103B quantized models vs. FP16</span></span> over <span id="A1.T20.6.2" class="ltx_text ltx_font_italic">Internal</span> and <span id="A1.T20.7.3" class="ltx_text ltx_font_italic">Aya Dolly</span> subsampled test sets.</figcaption>
</figure>
<figure id="A1.T21" class="ltx_table">
<div id="A1.T21.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:229.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(97.9pt,-51.8pt) scale(1.82256841400994,1.82256841400994) ;">
<table id="A1.T21.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T21.1.1.1.1" class="ltx_tr">
<th id="A1.T21.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt"></th>
<th id="A1.T21.1.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt"></th>
<th id="A1.T21.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T21.1.1.1.1.3.1" class="ltx_text ltx_font_bold">fr</span></th>
<th id="A1.T21.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T21.1.1.1.1.4.1" class="ltx_text ltx_font_bold">es</span></th>
<th id="A1.T21.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T21.1.1.1.1.5.1" class="ltx_text ltx_font_bold">ja</span></th>
<th id="A1.T21.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T21.1.1.1.1.6.1" class="ltx_text ltx_font_bold">ko</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T21.1.1.2.1" class="ltx_tr">
<th id="A1.T21.1.1.2.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="A1.T21.1.1.2.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">W8</th>
<td id="A1.T21.1.1.2.1.3" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F3C4C0;"><span id="A1.T21.1.1.2.1.3.1" class="ltx_text" style="background-color:#F3C4C0;">46.3</span></td>
<td id="A1.T21.1.1.2.1.4" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F7FCFA;"><span id="A1.T21.1.1.2.1.4.1" class="ltx_text" style="background-color:#F7FCFA;">50.3</span></td>
<td id="A1.T21.1.1.2.1.5" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#98D6B7;"><span id="A1.T21.1.1.2.1.5.1" class="ltx_text" style="background-color:#98D6B7;">53.7</span></td>
<td id="A1.T21.1.1.2.1.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#ECA099;"><span id="A1.T21.1.1.2.1.6.1" class="ltx_text" style="background-color:#ECA099;">44.0</span></td>
</tr>
<tr id="A1.T21.1.1.3.2" class="ltx_tr">
<th id="A1.T21.1.1.3.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T21.1.1.3.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="A1.T21.1.1.3.2.3" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F0B4AF;"><span id="A1.T21.1.1.3.2.3.1" class="ltx_text" style="background-color:#F0B4AF;">45.3</span></td>
<td id="A1.T21.1.1.3.2.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F3C4C0;"><span id="A1.T21.1.1.3.2.4.1" class="ltx_text" style="background-color:#F3C4C0;">46.3</span></td>
<td id="A1.T21.1.1.3.2.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FBEFEE;"><span id="A1.T21.1.1.3.2.5.1" class="ltx_text" style="background-color:#FBEFEE;">49.0</span></td>
<td id="A1.T21.1.1.3.2.6" class="ltx_td ltx_align_right" style="background-color:#C8E9D9;"><span id="A1.T21.1.1.3.2.6.1" class="ltx_text" style="background-color:#C8E9D9;">52.0</span></td>
</tr>
<tr id="A1.T21.1.1.4.3" class="ltx_tr">
<th id="A1.T21.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T21.1.1.4.3.1.1" class="ltx_text">Internal</span></th>
<th id="A1.T21.1.1.4.3.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W4-g</th>
<td id="A1.T21.1.1.4.3.3" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E67C73;"><span id="A1.T21.1.1.4.3.3.1" class="ltx_text" style="background-color:#E67C73;">41.7</span></td>
<td id="A1.T21.1.1.4.3.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F8DAD8;"><span id="A1.T21.1.1.4.3.4.1" class="ltx_text" style="background-color:#F8DAD8;">47.7</span></td>
<td id="A1.T21.1.1.4.3.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#E68078;"><span id="A1.T21.1.1.4.3.5.1" class="ltx_text" style="background-color:#E68078;">42.0</span></td>
<td id="A1.T21.1.1.4.3.6" class="ltx_td ltx_align_right" style="background-color:#F8DAD8;"><span id="A1.T21.1.1.4.3.6.1" class="ltx_text" style="background-color:#F8DAD8;">47.7</span></td>
</tr>
<tr id="A1.T21.1.1.5.4" class="ltx_tr">
<th id="A1.T21.1.1.5.4.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<th id="A1.T21.1.1.5.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t">W8</th>
<td id="A1.T21.1.1.5.4.3" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F7FCFA;"><span id="A1.T21.1.1.5.4.3.1" class="ltx_text" style="background-color:#F7FCFA;">50.3</span></td>
<td id="A1.T21.1.1.5.4.4" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#F6D4D1;"><span id="A1.T21.1.1.5.4.4.1" class="ltx_text" style="background-color:#F6D4D1;">47.3</span></td>
<td id="A1.T21.1.1.5.4.5" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="background-color:#57BB8A;"><span id="A1.T21.1.1.5.4.5.1" class="ltx_text" style="background-color:#57BB8A;">56.0</span></td>
<td id="A1.T21.1.1.5.4.6" class="ltx_td ltx_align_right ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T21.1.1.5.4.6.1" class="ltx_text" style="background-color:#FFFFFF;">50.0</span></td>
</tr>
<tr id="A1.T21.1.1.6.5" class="ltx_tr">
<th id="A1.T21.1.1.6.5.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="A1.T21.1.1.6.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">W8A8-sq</th>
<td id="A1.T21.1.1.6.5.3" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F3C4C0;"><span id="A1.T21.1.1.6.5.3.1" class="ltx_text" style="background-color:#F3C4C0;">46.3</span></td>
<td id="A1.T21.1.1.6.5.4" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#F2BBB6;"><span id="A1.T21.1.1.6.5.4.1" class="ltx_text" style="background-color:#F2BBB6;">45.7</span></td>
<td id="A1.T21.1.1.6.5.5" class="ltx_td ltx_align_right ltx_border_r" style="background-color:#FFFFFF;"><span id="A1.T21.1.1.6.5.5.1" class="ltx_text" style="background-color:#FFFFFF;">50.0</span></td>
<td id="A1.T21.1.1.6.5.6" class="ltx_td ltx_align_right" style="background-color:#F9E4E2;"><span id="A1.T21.1.1.6.5.6.1" class="ltx_text" style="background-color:#F9E4E2;">48.3</span></td>
</tr>
<tr id="A1.T21.1.1.7.6" class="ltx_tr">
<th id="A1.T21.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A1.T21.1.1.7.6.1.1" class="ltx_text">Dolly</span></th>
<th id="A1.T21.1.1.7.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">W4-g</th>
<td id="A1.T21.1.1.7.6.3" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#F0B4AF;"><span id="A1.T21.1.1.7.6.3.1" class="ltx_text" style="background-color:#F0B4AF;">45.3</span></td>
<td id="A1.T21.1.1.7.6.4" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#FCF3F3;"><span id="A1.T21.1.1.7.6.4.1" class="ltx_text" style="background-color:#FCF3F3;">49.3</span></td>
<td id="A1.T21.1.1.7.6.5" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="background-color:#DBF1E6;"><span id="A1.T21.1.1.7.6.5.1" class="ltx_text" style="background-color:#DBF1E6;">51.3</span></td>
<td id="A1.T21.1.1.7.6.6" class="ltx_td ltx_align_right ltx_border_bb" style="background-color:#F2BFBB;"><span id="A1.T21.1.1.7.6.6.1" class="ltx_text" style="background-color:#F2BFBB;">46.0</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table A21: </span><span id="A1.T21.5.1" class="ltx_text ltx_font_bold">Human evaluation raw win-rates of 103B quantized models vs. FP16</span> over <em id="A1.T21.6.2" class="ltx_emph ltx_font_italic">Internal</em> and <em id="A1.T21.7.3" class="ltx_emph ltx_font_italic">Aya Dolly</em> subsampled test sets.</figcaption>
</figure>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.03210" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.03211" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.03211">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.03211" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.03212" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 16:19:29 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.03211
Quantization techniques are widely used to improve inference speed and deployment of large language models. While a wide body of work examines the impact of quantized LLMs on English tasks, none have examined the effect of quantization across languages. We conduct a thorough analysis of quantized multilingual LLMs, focusing on their performance across languages and at varying scales. We use automatic benchmarks, LLM-as-a-Judge methods, and human evaluation, finding that (1)  harmful effects of quantization are apparent in human evaluation, and automatic metrics severely underestimate the detriment: a 1.7% average drop in Japanese across automatic tasks corresponds to a 16.0% drop reported by human evaluators on realistic prompts; (2)  languages are disparately affected by quantization, with non-Latin script languages impacted worst; and (3)  challenging tasks such as mathematical reasoning degrade fastest. As the ability to serve low-compute models is critical for wide global adoption of NLP technologies, our results urge consideration of multilingual performance as a key evaluation criterion for efficient models.
abstract
How Does Quantization Affect Multilingual LLMs?
Abstract
2407.03211
1IntroductionMultilingual large language models (LLMs)  have the power to bring modern language technology to the world, but only if they are cheap and reliable. Known as thelow-resource double bind, underserved languages and severe compute constraints often geographically co-occur, meaning that for wide adoption, multilingual LLMs must be highly-performantandlightweight.With the shift towards large models, quantization is a widely adopted technique to reduce cost, improve inference speed, and enable wider deployment of LLMs. Work on quantization, however, is by-and-large evaluated in English only. No works to our knowledge have characterized the impact of quantization on the multilingual generation capabilities expected from modern LLMs. Ubiquitous use of compression techniques in the real world drives urgency to the questionhow are multilingual models impacted?Our question is timely, given recent work showing that compression techniques such as quantization and sparsity amplify disparate treatment of long-tail features, which may have implications for under-represented languages in multilingual LLMs.
Indeed, many model designs choices implicitly overfit to a handful of resource rich languages: from tokenizer choice, to weighting of training data, and to widely-used quantization techniques.
Focusing on a small subset of high-resource languages in design degrades model performance for overlooked languages, introduces security vulnerabilities, and unfairly passes high costs to non-English users faced with high latency.We analyze four state-of-the-art multilingual LLMs across 3 different sizes ranging from 8 to 103 billion parameters and covering up to 23 languages, under various quantization techniques. Critically, it is vital that we move beyond automatic evaluation and gatherreal human feedback on performance cost. We thus perform multilingual human evaluation on challenging real-world prompts in addition to LLM-as-a-Judge and evaluation on standard automatic benchmarks such as multilingual MMLU, MGSM, and FLORES-200.
Across experimental set-ups we find that:1.Automatic metrics severely underestimate damage from quantization.Automatic evaluations estimate performance deterioration relative to FP16 across tasks at(French)  and(Japanese)  vsandreported by human evaluators.2.Quantization affects languages differently.Non-Latin script languages are more greatly harmed
on average. Across tasks, Latin-script languages scoredrelative to FP16 for a 103B parameter model while non-Latin scripts scored. For a smaller 8-billion parameter model, scores werevs..3.Challenging tasks degrade fastest.Mathematical reasoning () , performance on real-world challenging prompts judged by humans () , and LLM-as-a-Judge ()  results are severely degraded.4.Occasionally, quantization brings benefits.Similar toandon sparsity, we find that quantizationbenefitsmodel performance in some cases:
e.g., an average 1.3% boost across tasks for a 35B model quantized with W8A8.As the first to broadly study the impact of quantization on multilingual LLMs, our work is part of a wider body of literature that considers the impact of model design choices on downstream performance. Our results urge attention to multilingual performance at all stages of system design.
introduction
How Does Quantization Affect Multilingual LLMs?
1Introduction
2407.03211
2BackgroundQuantization compresses the weights and potentially activations of a neural network to lower-bit representations. Compression can be done by training the model at lower precision, known as Quantization Aware Training (QAT) , or performed on the final model weights, known as Post Training Quantization (PTQ) . Given the difficulties in training LLMs especially at precision lower than 16-bits floating point, PTQ methods which perform the quantization single-shot without needing gradient updates are highly desirable.
Training is completed at higher precision, then weights/activations are quantized without further training. In this work, we focus onpost-training quantizationbecause of its simplicity and applicability at scale. PTQ of LLMs can be further categorized into:Weight-Only QuantizationWeight matrices are quantized offline and the compressed matrices are loaded from memory during inference. Quantized weight matrices have a smaller memory footprint compared to FP16 (smaller for 8-bit and almostsmaller for 4-bit) , enabling inference with less compute. In memory-bound scenarios, it also enables faster inference due to fewer bytes transferred from GPU memory to the compute units.For a weight matrixand input, if only a single scaling factor is used for naive quantization (per-tensor) , then the quantized weights are given by:(1) wheredenotes the scale,the bit precision,the absolute value over each element inandrounding to the nearest integer.A single scaling factor might not be enough if the distribution of parameters in the weight matrix has high variance; thus one could increase the granularity of quantization by using a scale for each output dimension (per-column) , i.e.,. However, whenis aggressively lowered to 4 bits or lower, even per-column granularity might be insufficient to cover the range of values in a column. The granularity can be further increased by using a shared scale for a subset of input dimensions called groups () , thus the scale. A commonly used group size is 128.Equation1gives the simplest way to quantize the weights. Forbits, using more advanced Weight-Only Quantization methods like GPTQor AWQleads to better downstream performance.Weight-and-Activation QuantizationAs the name suggests, Weight-and-Activation Quantization quantizes the model activations alongside the weights. Unlike Weight-Only Quantization where weights can be quantized offline, quantization of activations happens at runtime. One could compute the quantization scales for various activations by using a small slice of training or validation data (static scaling)  but this method typically has large degradation. For minimal degradation, it is preferred to calculate the quantization scaling factor dynamically (dynamic scaling)  for each input on-the-fly. While quantizing activations is more difficult, reducing the precision of the activations alongside the weights enables the usage of specialized low-precision matrix multiplication hardware in modern GPUs leading to up toimprovement in throughput.
For a weight matrixand input, naive Weight-and-Activation Quantization with per-token input granularity and per-column weight granularity generates outputby:(2) (3) (4) where,,anddenotes element-wise multiplication by broadcasting the elements to match the shape of the operands.
body
How Does Quantization Affect Multilingual LLMs?
2Background
2407.03211
3Experiment Set-upModelsWe evaluate Command R+111https://docs.cohere.com/docs/command-r-plus,
Command R222https://docs.cohere.com/docs/command-r, and Aya 23 modelsas representatives of state-of-the-art multilingual LLMs. Command models are 103 and 35 billion parameters, and Aya 23 models are 35 and 8 billion parameters. We quantize the models using the weights available on HuggingFace.QuantizationFor Command R and R+, we evaluate bothweight-only quantizationat 8-bit (W8with per-column scaling)  and 4-bit (W4-gwith group-wise scaling using GPTQ) , as well asweight-and-activation quantizationat 8-bit (W8A8with per-column scaling for weights and per-token scaling for activations) .show that if the model is trained with the right hyper-parameters, naive Weight-and-Activation Quantization has minimal degradation. Otherwise, one may leverage SmoothQuantto smoothen the distribution of activations making them more amenable to quantization. We therefore also exploreW8A8-SmoothQuant(a W8A8 variant with SmoothQuant)  for Command R+ as well as a 4-bit weight-only quantized variant with column-wise scaling (W4)  to understand the impact of scaling granularity at extremely low-bit precision. Following, we use 128 English samples for calibration for SmoothQuant and GPTQ.For Aya 23 8B and 35B, we use bitsandbytes333https://github.com/TimDettmers/bitsandbytesto obtain 8-bit and 4-bit quantized models. Bitsandbytes uses LLM.int8() —similar to W8A8 described above except that it performs certain computations in FP16. Bitsandbytes 4-bit uses the NF4 datatypeto perform Quantile Quantization which limits degradation at the expense of inference speedups.3.1Automatic EvaluationWe evaluate in 10 primary languages:Arabic,French,German,English,Spanish,Italian,Portuguese,Korean,Japanese, andChinese. Quantized models are compared to the originalFP16versions, and we primarily report results asrelative degradationcompared to this FP16 baseline:(5) Raw numeric results are in the Appendix. Results are averaged over 5 runs.444k=0, p=0.75, temp=0.3, except mMMLU, which, as a QA eval, is run deterministically with temp=0.Multilingual MMLU (mMMLU) This multi-domain question answering task consists of 14,000+ multiple-choice questions. We translate MMLUto 9 languages with Google Translate and refer to this version as mMMLU. We measure accuracy in a 5-shot setting. An example is in TableA1.MGSMMGSM is a generative mathematics evaluation set manually translated from GSM8K. Of our target languages, it is available for German, Spanish, French, Japanese, and Chinese. We report accuracy over the 250-item test set for each language.FLORES-200This well-known multi-way parallel test set evaluates translation capabilities. We translate into and out of English, and report SacreBLEU.Language ConfusionThese test sets assess a model’s ability to respond in a user’s desired language. In the monolingual setting, prompts are in languageand the model must respond in language. For instance, a user prompts in Arabic, so implicitly desires an Arabic response. In the cross-lingual variant, a prompt is provided in English but the user requests output in a different language.555An example from theOkapisubsection of the evaluation is: “Reply in Spanish. Explain a common misconception about your topic. Topic: Using AI to Augment Human Capabilities”fastTextlanguage identification is run over the output. We reportline-level pass rate(LPR) , i.e., the percentage of responses for which all lines in the response are in the user’s desired language.Aya EvaluationAya 23 models are evaluated using an extended version of the Aya evaluation setupusing the unseen discriminative tasks (XWinograd, XCOPA, XStoryCloze) , mMMLU, MGSM, and Belebelefromeval-harness.666We follow the setup used by: each evaluation is run once, and for FLORES, no sampling is used and metric is spBLEU.We evaluate models on languages included in the covered 23 languages, except for the unseen tasks where we use all available languages.777mMMLU: ar, de, es, fr, hi, id, it, nl, pt, ro, ru, uk, vi, zh. MGSM: de, es, fr, ja, ru, zh. Belebele: {mMMLU} + cs, fa, el, ja, ko, pl, tr. FLORES: {Belebele} + he.Aya evaluations allow us to add:Czech, Greek, Hebrew, Hindi, Indonesian, Dutch, Persian, Polish, Romanian, Russian, Turkish, Ukrainian, Vietnamese.3.2Human EvaluationWe run human evaluation inSpanish,French,Korean, andJapanese.Internal Evaluation Suite150 diverse prompts designed to be more complex than public evaluation benchmarks. As such, we expect greater degradation with increased quantization given the difficulty of the samples. Prompts for all four languages are translated by humans from an English seed prompt, ensuring that respective language-specific subsets share the same prompts.Aya Dolly-200We use multilingual data from the Aya Evaluation Suite to assess open-ended generation capabilities. For Korean and Japanese, we use prompts from the Aya Dolly-200 test set (dolly-machine-translated) , which are automatically translated from English Dolly-15kthen human-curated to avoid references requiring specific cultural or geographic knowledge.
For French and Spanish, we usedolly-human-edited, a human post-edited version ofdolly-machine-translated. For each language, we evaluate using the first 150 prompts.Annotator StatisticsAnnotations and translations were completed by native-level speakers of the respective languages, each of whom is also fluent in English.
Annotators were paid by the hour, with compensation above the federal minimum wage of the country of employment.Annotation InterfaceWe use a pairwise evaluation setup. Annotators see a prompt and two (shuffled)  completions of the FP16 model and a quantized variant. They rate each response on a 5-point Likert scale, then express a preference between the two model outputs (tie, weak preference, strong preference) . We encourage annotators to avoid tied rankings. Win rates are based on the ranking preferences alone.3.3LLM/RM-as-a-JudgeBecause human evaluation is costly and time-intensive, it is common to use an “LLM-as-a-Judge” to rate model completions. Reward models (RMs)  can also simulate human preference. An RM scores multiple completions given the same prompt, and the prompt-completion pair with the higher score is deemed preferred. We call thisRM-as-a-Judge.We assess quantized model outputs using LLM- and RM-as-a-Judge. In the former, an LLM selects a preferred response from ainstruction, modelA_completion, modelB_completiontuple (see TableA2) . Followingwe use GPT-4888gpt-4-turbo (gpt-4-1106-preview) :https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4as an LLM proxy judge. To minimize bias, we randomize the order of model outputs. For RM-as-a-Judge, a multilingual reward model scoresprompt, completionpairs for each model output, over which we calculate win-rate. We report win-rates of quantized models versus the FP16 baseline.We assess the outputs of quantized models over theInternal Evaluation SuiteandAya Dolly-200described in Section3.2. We use the same prompt and completion pairs as in human evaluation, which provides the ability to relate LLM/RM-as-a-Judge performance with human evaluation.
body
How Does Quantization Affect Multilingual LLMs?
3Experiment Set-up
2407.03211
4ResultsAvg.FLORESLanguage ConfusionRel.mMMLUMGSMEnL2L2EnMonolingualCross-lingualFP16-66.7-70.6-37.7-39.6-99.2-91.5-W8-0.2%66.70.0%69.9-1.0%37.70.0%39.60.0%99.20.0%91.2-0.3%W8A8-sq-0.5%66.3-0.5%69.5-1.6%37.80.2%39.1-1.3%99.20.0%91.50.1%W8A8-0.8%65.6-1.7%69.8-1.1%37.70.0%39.1-1.2%99.40.2%90.4-1.2%W4-g-0.9%65.7-1.4%68.6-2.9%37.80.4%39.4-0.5%99.20.0%90.5-1.1%103BW4-2.5%63.8-4.3%64.4-8.8%37.1-1.6%39.0-1.6%99.30.1%92.81.4%FP16-59.4-49.8-32.4-35.5-98.7-66.5-W8-0.2%59.3-0.1%49.4-0.7%32.3-0.2%35.4-0.2%98.80.1%66.3-0.2%W8A80.2%59.3-0.2%47.1-5.5%32.91.6%35.80.9%99.00.3%68.93.7%35BW4-g-2.8%58.2-2.0%43.3-13.1%31.7-1.9%35.3-0.7%98.3-0.4%67.11.0%Table 1:Per-dataset average performance across non-English languages for 103B and 35B Command models at varying levels of quantization.%the relative performance vs. FP16 [ex., for MGSM at W4-g on the 35B:.] Languages: ar, de, es, fr, it, ja, ko, pt, zh; except MGSM: de, es, fr, ja, zh.
Any discrepancy is due to rounding: raw scores and%were calculated at full precision.Avg.FLORESUnseenRel.mMMLUMGSMEnL2L2EnBelebeleTasksFP16-58.2-51.2-37.8-42.9-77.6-70.8-W80.1%57.9-0.5%52.11.8%37.90.3%43.00.1%77.1-0.6%70.6-0.2%Aya 35BW4-2.9%56.6-2.7%48.1-6.0%37.2-1.4%42.4-1.2%73.0-5.9%70.5-0.3%FP16-48.2-34.7-34.8-39.5-64.8-67.6-W80.3%47.8-0.9%35.42.1%34.80.2%39.70.5%64.6-0.3%67.60.1%Aya 8BW4-3.7%46.7-3.2%32.1-7.5%34.1-1.8%39.1-1.0%59.3-8.5%67.5-0.2%Table 2:Per-dataset average performance across non-English languages for 35B and 8B Aya 23 models at varying levels of quantization.%is relative performance vs. FP16. We follow the evaluation setup ofand evaluate on languages in the 23 languages list. On “Unseen Tasks” (XWinograd, XCOPA, XStoryCloze) , we use all the available languages.
See Section3.1for details and language list.To clearly see the many-faceted impact of quantization, we discuss our results by quantization level (§4.1) , by task (§4.2) , by language (§4.3) , by model size (§4.4) , and by quantization strategy (§4.5) . We then report LLM-as-a-Judge and RM-as-a-Judge (§4.6)  and human evaluation results (§4.7) .4.1By Quantization LevelHow do different levels of quantization affect downstream performance?Command R and R+In Table1, we aggregate results of each metric for each level of quantization. We average scores across languages, then calculate the relative percentage drop from FP16.999Ex. For 103BW4-gMGSM, scores were: {de: 71.2, es: 75.7, fr: 69.0, ja: 58.0, zh: 68.9}, thus the average score was 68.6—a 2.9% drop from FP16 () .We discuss results ofW8,W8A8, andW4-gquantization, which are variants available for both Command model sizes. Most results follow intuition: greater quantization leads to larger performance degradation:forW8,forW8A8, andoverall for the 103B model. An exception isW8A8for the 35B, which experiences a slight boost overall due to higher performance on translation and language confusion evaluations.Aya 23 ModelsTable2shows the aggregated results for Aya 23 models on the extended Aya evaluations atW8, andW4quantization.
We find a similar trend with Command models whereW4often leads to a larger drop compared toW8, consistent across tasks and languages.W8, however, does not substantially drop performance in any task.4.2By TaskAre tasks differently affected by quantization?Results here reference Tables1and2, with full raw and relative results in AppendixA.2.
Mathematical reasoning as measured by MGSM is strikingly affected by quantization. Relative performance of the 35BW4-gmodel is a dismalon average, with as poor asin Chinese. MGSM and Belebele are most greatly degraded for Aya 23 models withW4quantization, droppingandon the 8B model. mMMLU is the next most greatly degraded task.ardeesfritjakoptzhAvgLtn/IEW80.0%0.1%0.0%0.0%0.0%0.1%0.0%-0.4%-0.2%-0.1%-0.1%-0.1%W8A8-sq-0.6%0.2%-0.3%0.1%-0.6%-0.3%-0.1%-0.7%-0.8%-0.3%-0.3%-0.4%W8A8-1.3%-0.9%-0.5%-0.5%-0.8%-0.3%-1.3%-0.8%-0.9%-0.8%-0.7%-0.9%W4-g-0.8%-0.2%-0.4%0.1%-0.4%-0.4%-0.6%-1.2%-0.9%-0.5%-0.4%-0.7%103BW4-1.0%-0.6%0.1%-0.8%-1.2%-1.4%-2.9%-0.8%-2.3%-1.2%-0.7%-1.9%W80.3%-0.5%-0.1%-0.2%-0.4%0.3%-0.1%0.1%-0.3%-0.1%-0.2%0.0%W8A82.0%2.5%0.7%1.0%1.2%1.1%0.9%1.4%1.0%1.3%1.3%1.3%35BW4-g-1.1%-1.1%0.1%-0.3%-0.1%-2.3%-1.4%-0.6%-1.3%-0.9%-0.4%-1.5%Table 3:Per-language relative performance (%)  vs. FP16, averaged over mMMLU, FLORES, and Language Confusion tasks.Ltn/IE are Latin-script/Indo-European languages: de, es, fr, it, pt.are the rest: ar, ja, ko, zh.deesfrjazhAvgLtn/IEW80.1%-0.1%-0.3%-0.4%-0.2%-0.2%-0.1%-0.3%W8A8-sq0.4%-0.9%-0.1%-0.3%-1.2%-0.4%-0.2%-0.8%W8A8-0.4%-1.0%-0.6%-0.1%-1.3%-0.7%-0.6%-0.7%W4-g-0.5%-0.5%-0.3%-1.7%-1.1%-0.8%-0.4%-1.4%103BW4-2.3%-1.1%-1.7%-3.0%-3.5%-2.3%-1.7%-3.3%W8-0.6%-0.3%-0.1%-0.4%0.0%-0.2%-0.3%-0.2%W8A81.3%-0.6%0.3%-0.3%0.0%0.1%0.3%-0.2%35BW4-g-3.7%-1.8%-1.7%-3.8%-4.0%-3.0%-2.4%-3.9%Table 4:Per-language relative performance (%)  vs. FP16, averaged over MGSM, mMMLU, FLORES, and Language Confusion tasks.Ltn/IEare Latin-script/Indo-European: de, es, fr.are the rest: ja, zh.On FLORES, the 103B Command model is more sensitive to quantization for the L2En translation direction than for L2En, though we see the opposite for the smaller 35B and Aya 23 models atW4. Quantization does not noticeably impact unseen discriminative tasks (XWinograd, XCOPA, XStoryCloze: TableA17) .Curiously, there are some fleeting performance boosts: an increase of 1.8–2.1% on MGSM and mild improvements on FLORES withW8on Aya models, and a similar translation boost of the 35B Command model atW8A8. Quantization generally has no effect or causes mild improvement on the monolingual language confusion task, and cross-lingual language confusion performance is boosted with greater quantization in some cases.4.3By LanguageAre languages differently affected by quantization?Table3shows performance averaged over mMMLU, FLORES, and Language Confusion tasks, with Table4further including MGSM for supported languages. Metrics are on different scales, so we average relative change (%)  rather than raw scores.101010Ex. to arrive atfor 103BW8A8in Arabic, we average relative performance for mMMLU, FLORES EnL2, and Language Confusion tasks:.We further separate into languages written in the Latin/Roman script (which also are the subset of Indo-European languages;Ltn/IE)  versus the rest (Ltn/IE) .W4-gcauses considerable degradation across languages for the 35B Command model. A relationship between language and performance is apparent:Ltn/IElanguages typically degrade more. Chinese, Japanese, and Korean are particularly harmed byW4quantization of the 103B. The effect is seen consistently across all automatic metrics for Command, with limited exception. Table5is discussed more thoroughly in Section4.5, but also shows this discrepancy. In the Appendix, we see the same for Aya 23 models atW4.Interestingly,W8A8of the 35B Command modelhelpson average across all languages. The magnitude is primarily due to an increase on cross-lingual language confusion.W8also aids Aya 23 on MGSM (TableA6)  forLtn/IElanguages, and across languages on FLORES (TableA16) .4.4By Model SizeHow do model size and quantization level interact?Across evaluations at the most extreme quantization (W4/W4-g) , smaller models are more sensitive:W4-gvariants of 103B and 35B Command recordandperformance relative to FP16 on average, with a stark difference ofvs.on MGSM. Aya 23 35B/8B recordvs.on average, with their largest gap occurring in Belebele (vs.) . (Refer back to Tables1and2.) 4.5By Quantization StrategyHow do techniques like SmoothQuant and group-wise scaling affect downstream performance?FLORESLanguage ConfusionAvg. Rel. %mMMLUMGSMEnL2L2EnMonolingualCross-lingualLtn/IELtn/IELtn/IELtn/IELtn/IELtn/IELtn/IEW8A8-0.7%-1.0%-1.3%-2.1%-0.9%-1.3%-0.1%0.1%-1.0%-1.6%0.0%0.4%-0.9%-1.6%W8A8-sq-0.4%-0.7%-0.4%-0.8%-1.3%-1.9%0.2%0.0%-1.1%-1.6%-0.1%0.1%0.1%0.0%W4-1.9%-3.3%-3.9%-4.9%-8.0%-10.2%-1.3%-2.0%-1.1%-2.3%0.1%0.1%2.9%-0.4%W4-g-0.6%-1.4%-1.1%-1.9%-1.8%-4.9%0.2%0.7%-0.3%-0.8%0.1%-0.1%-0.9%-1.3%Table 5:Effect of mitigation strategies on W8A8 and W4 quantization on the 103B model.Percentage points off FP16 baseline for W8A8-sq vs. naive W8A8 and W4-g vs. W4, broken down by Latin-script/Indo-European languages (Ltn/IE)  versus others () . Avg. Rel. % reports averaged performance all datasets.fresjakoAvgLtn/IELLMRMLLMRMLLMRMLLMRMLLMRMLLMRMLLMRMW81.0%-0.7%-10.2%7.5%-5.4%5.4%7.5%-5.8%-1.8%1.6%-4.6%3.4%1.0%-0.2%W8A8-sq-18.4%-5.1%-3.7%4.1%2.0%4.7%3.7%-5.1%-4.1%-0.3%-11.0%-0.5%2.9%-0.2%W4-g-10.5%-17.0%-16.6%2.0%-15.3%0.0%-5.8%-15.6%-12.1%-7.7%-13.6%-7.5%-10.5%-7.8%InternalW4-30.2%-20.4%-33.0%-17.0%-21.7%-20.0%-18.6%-27.6%-25.9%-21.2%-31.6%-18.7%-20.2%-23.8%W8-1.3%2.0%7.3%-4.0%-6.0%-5.3%2.7%2.0%0.7%-1.3%3.0%-1.0%-1.7%-1.7%W8A8-sq-15.3%-8.7%8.7%-8.0%-1.3%1.3%-8.0%-4.7%-4.0%-5.0%-3.3%-8.3%-4.7%-1.7%W8A8-3.4%2.7%13.3%-3.3%2.7%-1.3%5.3%-3.3%4.5%-1.3%5.0%-0.3%4.0%-2.3%DollyW4-g-7.4%-2.7%-4.0%4.7%-15.3%-15.3%-11.3%-5.3%-9.5%-4.7%-5.7%1.0%-13.3%-10.3%Table 6:Relative performance vs. FP16 of 103B quantized models according toLLM/RM-as-a-JudgeoverInternalandAya Dollysubsampled test sets. Raw win-rates in TableA20.Table5shows the effect of using SmoothQuant and Group-Wise scaling strategies.
We evaluate variants of the 103B Command model with SmoothQuant (W8A8-sq) , and a more naiveW4variant using per-column quantization instead of group-wise scaling. We compareW8A8-sqtoW8A8, andW4-gtoW4.On average and across mMMLU, MGSM, and FLORES, Group-Wise scaling greatly improves over column-wiseW4, recovering over 6 percentage points lost on MGSM forLtn/IElanguages. SmoothQuant has a similar effect on average and for mMMLU, though to a lesser degree. That said, SmoothQuant harms MGSM scores slightly, and Group-Wise scaling degrades cross-lingual language confusion. We again observe thatlanguages suffer more in nearly all cases.On cross-lingual language confusion, strategies aimed to retain performance have different effects: SmoothQuant recovers all lost from naiveW8A8, but Group-Wise scaling is actively damaging. In contrast,W4benefitsLtn/IEand Arabic on cross-lingual language confusion, but worsens the rest.111111Full results in are TableA19.Thus, while the quantization strategies tend to aid performance overall, there may be adverse effects on specific tasks.
More research is needed to understand this, but it is intriguing to consider the effect that lower-precision might have on the ability to produce output in a desired language, and maintain that language once decoding begins.4.6LLM/RM-as-a-JudgeTable6shows relative performance of quantized variants of the 103B Command model evaluated with LLM- and RM-as-a-Judge.121212Calculation:, as 50 is the expected win-rate of two FP16 models compared.In nearly all cases, the LLM and RM agree thatW4andW4-gseverely harm performance on our challengingInternaltest set. Performance is also severely degraded forLtn/IElanguages onDollywithW4-g, and French withW8A8-sq. On average across languages, the LLM and RM agree on the ranking of model quality overInternal. Results on the easierDollytest set are less clear-cut: The LLM reports greater degradation forInternalthanDollyoverall, but the RM disagrees forW8andW8A8-sq. PerhapsDollyprompts are easy enough that models output similar responses, creating more noise in the judgments; future work could examine this hypothesis. Furthermore, on multiple instances, the LLM and RM disagree on whether performanceimprovesorworsens, given the same setting. Comparisons between the two differing methods of automated evaluation are worthy of further study.4.7Human EvaluationfresjakoAvgLtn/IEW8-7.4%0.6%7.4%-12.0%-2.8%-3.4%-2.3%W8A8-sq-9.4%-7.4%-2.0%4.0%-3.7%-8.4%1.0%InternalW4-g-16.6%-4.6%-16.0%-4.6%-10.5%-10.6%-10.3%W80.6%-5.4%12.0%0.0%1.8%-2.4%6.0%W8A8-sq-7.4%-8.6%0.0%-3.4%-4.8%-8.0%-1.7%DollyW4-g-9.4%-1.4%2.6%-8.0%-4.1%-5.4%-2.7%Table 7:Relative performance vs. FP16 of 103B quantized models according tohuman evaluatorsoverInternalandAya Dollysubsampled test sets.Human evaluation paints a similar picture in Table7, with some outliers. Average performance drops steadily across evaluated languages on theInternaltest set, which has more difficult prompts. The sharpest decline is in French, withatW4-g. Curiously, there is an initialboost for Japanese withW8, but it falls towith more extreme quantization. Interestingly, human annotators generally prefer outputs of quantized models onDollyprompts in Japanese, too, but disprefer those in other languages. We see more pronounced degradation onInternaloverall, with an average relative drop of 5.7% versus 2.4% for Dolly.
body
How Does Quantization Affect Multilingual LLMs?
4Results
2407.03211
5Related WorkImpact of Compression on Multilingual TasksThere is a scarcity of research examining the impact of compression and quantization on multilingual tasks.study the impact of multilingual calibration sets on quantization, but their evaluation is English-only.study the effect of compression on multilingual model fairness in terms of classification accuracy, showing that while monolingual evaluation indicates a negative impact, multilingual evaluation differs across languages and dimensions.show that recovering compression-caused performance loss of LSTMs is harder in a multilingual setting than monolingually. In machine translation,show that distillation has a varied effect by language due to dependence on priors such as amount of synthetic data used and confidence of the teacher models, while quantization exhibits more consistent performance trends across languages. Our work is the first, to our knowledge, to study the effect of quantization on LLMs and for open-ended generation.More broadly, multilingual data is an example of long tail data. Prior work shows that compression techniques like quantization and sparsity amplify disparate treatment of long-tail rare features.show that depending on how out of distribution the task data is, sparsity-based compression can sometimes avoid overfitting to the training data, making a model better suited to the downstream task.find that sparsity preserves machine translation performance on frequent sentences, but disparately impacts infrequent sentences.Quantization of LLMsA recent line of work has emerged on techniques to improve performance of quantized LLMs, with the sole focus on English models and data for tuning and evaluation. Even the most recentomit the multilingual dimension without acknowledging the limitation. Multilinguality and compression are both integral parts of LLMs, and our work explores this new territory.Model design choicesWe consider how design choices such as quantization impact performance for users of different languages. A wider body of work examines how design choices impact performance on underrepresented features or subgroups.andfind that hardware choice incurs disparate impact on underrepresented features.establish that distillation imposes similar trade-offs, but the disproportionate harm to the long-tail could be mitigated by modifying the student-teacher objective.evaluate the positive role of ensembling disproportionately favoring underrepresented attributes.show that differential privacy techniques like gradient clipping and noise injection disproportionately impact underrepresented features.
body
How Does Quantization Affect Multilingual LLMs?
5Related Work
2407.03211
6Conclusion & Future WorkWe examine widely adopted quantization techniques for model compression and ask,How does quantization impact different languages?We perform an extensive study of quantization in state-of-the-art multilingual LLMs—from 8 billion to 103 billion parameters—in 20+ languages using automatic metrics, LLM-as-a-Judge, RM-as-a-Judge, and human evaluation. We find that: (1)  Damage from quantization is much worse than appears from automatic metrics: even when not observed automatically, human evaluators notice it. (2)  Quantization affects languages to varying degrees, with non-Latin script languages more severely affected on automatic benchmarks. (3)  Challenging tasks degrade fast and severely: math performance is strikingly reduced, as are responses on realistic challenging prompts judged by humans. On a bright note, quantization occasionally brings performance benefits.Our results urge attention to multilingual performance at all stages of system design. Researchers might extend our work to consider the impact of other decisions on multilingual performance, including on languages excluded from training and out-of-distribution tasks. By being mindful of the impact on long-tail features, we’ll build better systems to serve the world.
body
How Does Quantization Affect Multilingual LLMs?
6Conclusion & Future Work
2407.03211
7LimitationsGenerality of findingsDue to the number of methods, languages, and benchmarks we examine, we focus our evaluation on models from two families (Command R and Aya) . As we observe similar trends across these models, our findings are likely to generalize to other LLMs. Nevertheless, models that have been optimized differently or trained with a focus on specific tasks such as code or mathematical reasoning may behave differently.Under-represented languagesFor our study, we focused on languages that were supported by the models we evaluated. Performance deterioration is likely even larger for languages that are not or severely under-represented in the pre-training data. For such languages, evaluation is also more challenging due to poor availability of benchmark data and human annotators.
conclusion
How Does Quantization Affect Multilingual LLMs?
7Limitations