<!DOCTYPE html><html prefix="dcterms: http://purl.org/dc/terms/" lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.21075] Apple Intelligence Foundation Language Models</title><meta property="og:description" content="We present foundation language models developed to power Apple Intelligence features, including a 3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Pr…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Apple Intelligence Foundation Language Models">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Apple Intelligence Foundation Language Models">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.21075">

<!--Generated on Mon Aug  5 14:53:26 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<!--Document created on July 2024.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<div id="p1" class="ltx_para">
<span id="p1.1" class="ltx_ERROR undefined">\pretocmd</span><span id="p1.2" class="ltx_ERROR undefined">\NAT@citex</span><span id="p1.3" class="ltx_ERROR undefined">\forcsvlist</span>
<p id="p1.4" class="ltx_p">#3



<span id="p1.4.1" class="ltx_ERROR undefined">\patchcmd</span><span id="p1.4.2" class="ltx_ERROR undefined">\NAT@citex</span>
<span id="p1.4.3" class="ltx_ERROR undefined">\NAT@cmt</span>#2
<span id="p1.4.4" class="ltx_ERROR undefined">\patchcmd</span><span id="p1.4.5" class="ltx_ERROR undefined">\NAT@citex</span>
<span id="p1.4.6" class="ltx_ERROR undefined">\NAT@def@citea</span>
<span id="p1.4.7" class="ltx_ERROR undefined">\NAT@def@citea</span>





















    
  
    






 







</p>
</div>
<h1 class="ltx_title ltx_title_document">Apple Intelligence Foundation Language Models</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Apple
</span></span>
</div>
<div class="ltx_dates">(July 2024)</div>

<div class="ltx_abstract">
<p id="id1.1" class="ltx_p"><span id="id1.1.1" class="ltx_text ltx_font_sansserif">We present foundation language models developed to power Apple Intelligence features, including a <math id="id1.1.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="id1.1.1.m1.1a"><mo id="id1.1.1.m1.1.1" xref="id1.1.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="id1.1.1.m1.1b"><csymbol cd="latexml" id="id1.1.1.m1.1.1.cmml" xref="id1.1.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="id1.1.1.m1.1c">\sim</annotation></semantics></math>3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute <cite class="ltx_cite ltx_citemacro_citep">[Apple, <a href="#bib.bib7" title="" class="ltx_ref">2024b</a>]</cite>.
These models are designed to perform a wide range of tasks efficiently, accurately, and responsibly.
This report describes the model architecture, the data used to train the model, the training process, how the models are optimized for inference, and the evaluation results. We highlight our focus on Responsible AI and how the principles are applied throughout the model development.</span></p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S1.2.1.1" class="ltx_text ltx_font_sansserif">1</span> </span><span id="S1.3.2" class="ltx_text ltx_font_sansserif">Introduction</span>
</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">At the 2024 Worldwide Developers Conference, we introduced Apple Intelligence, a personal intelligence system integrated deeply into iOS 18, iPadOS 18, and macOS Sequoia.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Apple Intelligence consists of multiple highly-capable generative models that are fast, efficient, specialized for our users’ everyday tasks, and can adapt on the fly for their current activity. The foundation models built into Apple Intelligence have been fine-tuned for user experiences such as writing and refining text, prioritizing and summarizing notifications, creating playful images for conversations with family and friends, and taking in-app actions to simplify interactions across apps.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.21075/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_img_landscape" width="553" height="175" alt="Modeling Overview Diagram.">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Modeling overview for the Apple foundation models.</figcaption>
</figure>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this report we will detail how two of these models—AFM-on-device (AFM stands for <em id="S1.p3.1.1" class="ltx_emph ltx_font_italic">Apple Foundation Model</em>), a <math id="S1.p3.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S1.p3.1.m1.1a"><mo id="S1.p3.1.m1.1.1" xref="S1.p3.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S1.p3.1.m1.1b"><csymbol cd="latexml" id="S1.p3.1.m1.1.1.cmml" xref="S1.p3.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.1.m1.1c">\sim</annotation></semantics></math>3 billion parameter language model, and AFM-server, a larger server-based language model—have been built and adapted to perform specialized tasks efficiently, accurately, and responsibly (<a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 1</span></a>). These two foundation models are part of a larger family of generative models created by Apple to support users and developers; this includes a coding model (based on an AFM language model) to build intelligence into Xcode, as well as a diffusion model to help users express themselves visually, for example, in the Messages app.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Apple Intelligence is designed with Apple’s core values at every step and built on a foundation of industry-lead privacy protection. Additionally, we have created Responsible AI principles to guide how we develop AI tools, as well as the models that underpin them:</p>
</div>
<div id="S1.p5" class="ltx_para">
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p"><span id="S1.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Empower users with intelligent tools</span>: We identify areas where AI can be used responsibly to create tools for addressing specific user needs. We respect how our users choose to use these tools to accomplish their goals.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p"><span id="S1.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Represent our users</span>: We build deeply personal products with the goal of representing users around the globe authentically. We work continuously to avoid perpetuating stereotypes and systemic biases across our AI tools and models.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p"><span id="S1.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Design with care</span>: We take precautions at every stage of our process, including design, model training, feature development, and quality evaluation to identify how our AI tools may be misused or lead to potential harm. We will continuously and proactively improve our AI tools with the help of user feedback.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.1" class="ltx_p"><span id="S1.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Protect privacy</span>: We protect our users’ privacy with powerful on-device processing and groundbreaking infrastructure like Private Cloud Compute. We do not use our users’ private personal data or user interactions when training our foundation models.</p>
</div>
</li>
</ol>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">These principles are reflected at every stage of the architecture that enables Apple Intelligence and connects features and tools with specialized models.</p>
</div>
<div id="S1.p7" class="ltx_para">
<p id="S1.p7.1" class="ltx_p">In the remainder of this report, we provide details on decisions such as: how we develop models that are highly capable, fast, and power-efficient; how we approach training these models; how our adapters are fine-tuned for specific user needs; and how we evaluate model performance for both helpfulness and unintended harm.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S2.2.1.1" class="ltx_text ltx_font_sansserif">2</span> </span><span id="S2.3.2" class="ltx_text ltx_font_sansserif">Architecture</span>
</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">The AFM base models are dense decoder-only models that build on the Transformer architecture <cite class="ltx_cite ltx_citemacro_citep">[Vaswani et al., <a href="#bib.bib52" title="" class="ltx_ref">2017</a>]</cite>, with the following design choices:</p>
</div>
<div id="S2.p2" class="ltx_para">
<ul id="S2.I1" class="ltx_itemize">
<li id="S2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i1.p1" class="ltx_para">
<p id="S2.I1.i1.p1.1" class="ltx_p">A shared input/output embedding matrix <cite class="ltx_cite ltx_citemacro_citep">[Press and Wolf, <a href="#bib.bib42" title="" class="ltx_ref">2016</a>]</cite> to reduce memory usage for parameters.</p>
</div>
</li>
<li id="S2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i2.p1" class="ltx_para">
<p id="S2.I1.i2.p1.1" class="ltx_p">Pre-Normalization <cite class="ltx_cite ltx_citemacro_citep">[Nguyen and Salazar, <a href="#bib.bib39" title="" class="ltx_ref">2019</a>]</cite> with RMSNorm <cite class="ltx_cite ltx_citemacro_citep">[Zhang and Sennrich, <a href="#bib.bib62" title="" class="ltx_ref">2019</a>]</cite> for training stability.</p>
</div>
</li>
<li id="S2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i3.p1" class="ltx_para">
<p id="S2.I1.i3.p1.1" class="ltx_p">Query/key normalization <cite class="ltx_cite ltx_citemacro_citep">[Wortsman et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>]</cite> to improve training stability.</p>
</div>
</li>
<li id="S2.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i4.p1" class="ltx_para">
<p id="S2.I1.i4.p1.1" class="ltx_p">Grouped-query attention (GQA) <cite class="ltx_cite ltx_citemacro_citep">[Ainslie et al., <a href="#bib.bib4" title="" class="ltx_ref">2023</a>]</cite> with 8 key-value heads to reduce the KV-cache memory footprint.</p>
</div>
</li>
<li id="S2.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i5.p1" class="ltx_para">
<p id="S2.I1.i5.p1.1" class="ltx_p">The SwiGLU activation <cite class="ltx_cite ltx_citemacro_citep">[Shazeer, <a href="#bib.bib46" title="" class="ltx_ref">2020</a>]</cite> for higher efficiency.</p>
</div>
</li>
<li id="S2.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S2.I1.i6.p1" class="ltx_para">
<p id="S2.I1.i6.p1.1" class="ltx_p">RoPE <cite class="ltx_cite ltx_citemacro_citep">[Su et al., <a href="#bib.bib48" title="" class="ltx_ref">2024</a>]</cite> positional embeddings with the base frequency set to <math id="S2.I1.i6.p1.1.m1.1" class="ltx_Math" alttext="500\mathrm{k}" display="inline"><semantics id="S2.I1.i6.p1.1.m1.1a"><mrow id="S2.I1.i6.p1.1.m1.1.1" xref="S2.I1.i6.p1.1.m1.1.1.cmml"><mn id="S2.I1.i6.p1.1.m1.1.1.2" xref="S2.I1.i6.p1.1.m1.1.1.2.cmml">500</mn><mo lspace="0em" rspace="0em" id="S2.I1.i6.p1.1.m1.1.1.1" xref="S2.I1.i6.p1.1.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="S2.I1.i6.p1.1.m1.1.1.3" xref="S2.I1.i6.p1.1.m1.1.1.3.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i6.p1.1.m1.1b"><apply id="S2.I1.i6.p1.1.m1.1.1.cmml" xref="S2.I1.i6.p1.1.m1.1.1"><times id="S2.I1.i6.p1.1.m1.1.1.1.cmml" xref="S2.I1.i6.p1.1.m1.1.1.1"></times><cn type="integer" id="S2.I1.i6.p1.1.m1.1.1.2.cmml" xref="S2.I1.i6.p1.1.m1.1.1.2">500</cn><ci id="S2.I1.i6.p1.1.m1.1.1.3.cmml" xref="S2.I1.i6.p1.1.m1.1.1.3">k</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i6.p1.1.m1.1c">500\mathrm{k}</annotation></semantics></math> for long-context support.</p>
</div>
</li>
</ul>
</div>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.1" class="ltx_p"><a href="#S2.T1" title="Table 1 ‣ 2 Architecture ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 1</span></a> provides some details about AFM-on-device dimensions.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<table id="S2.T1.7" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S2.T1.1.1" class="ltx_tr">
<td id="S2.T1.1.1.2" class="ltx_td ltx_align_left ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;">Model dimension</td>
<td id="S2.T1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><math id="S2.T1.1.1.1.m1.1" class="ltx_Math" alttext="3072" display="inline"><semantics id="S2.T1.1.1.1.m1.1a"><mn id="S2.T1.1.1.1.m1.1.1" xref="S2.T1.1.1.1.m1.1.1.cmml">3072</mn><annotation-xml encoding="MathML-Content" id="S2.T1.1.1.1.m1.1b"><cn type="integer" id="S2.T1.1.1.1.m1.1.1.cmml" xref="S2.T1.1.1.1.m1.1.1">3072</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.1.1.1.m1.1c">3072</annotation></semantics></math></td>
</tr>
<tr id="S2.T1.2.2" class="ltx_tr">
<td id="S2.T1.2.2.2" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">Head dimension</td>
<td id="S2.T1.2.2.1" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;"><math id="S2.T1.2.2.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S2.T1.2.2.1.m1.1a"><mn id="S2.T1.2.2.1.m1.1.1" xref="S2.T1.2.2.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S2.T1.2.2.1.m1.1b"><cn type="integer" id="S2.T1.2.2.1.m1.1.1.cmml" xref="S2.T1.2.2.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.2.2.1.m1.1c">128</annotation></semantics></math></td>
</tr>
<tr id="S2.T1.3.3" class="ltx_tr">
<td id="S2.T1.3.3.2" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">Num query heads</td>
<td id="S2.T1.3.3.1" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;"><math id="S2.T1.3.3.1.m1.1" class="ltx_Math" alttext="24" display="inline"><semantics id="S2.T1.3.3.1.m1.1a"><mn id="S2.T1.3.3.1.m1.1.1" xref="S2.T1.3.3.1.m1.1.1.cmml">24</mn><annotation-xml encoding="MathML-Content" id="S2.T1.3.3.1.m1.1b"><cn type="integer" id="S2.T1.3.3.1.m1.1.1.cmml" xref="S2.T1.3.3.1.m1.1.1">24</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.3.3.1.m1.1c">24</annotation></semantics></math></td>
</tr>
<tr id="S2.T1.4.4" class="ltx_tr">
<td id="S2.T1.4.4.2" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">Num key/value heads</td>
<td id="S2.T1.4.4.1" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;"><math id="S2.T1.4.4.1.m1.1" class="ltx_Math" alttext="8" display="inline"><semantics id="S2.T1.4.4.1.m1.1a"><mn id="S2.T1.4.4.1.m1.1.1" xref="S2.T1.4.4.1.m1.1.1.cmml">8</mn><annotation-xml encoding="MathML-Content" id="S2.T1.4.4.1.m1.1b"><cn type="integer" id="S2.T1.4.4.1.m1.1.1.cmml" xref="S2.T1.4.4.1.m1.1.1">8</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.4.4.1.m1.1c">8</annotation></semantics></math></td>
</tr>
<tr id="S2.T1.5.5" class="ltx_tr">
<td id="S2.T1.5.5.2" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;">Num layers</td>
<td id="S2.T1.5.5.1" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;"><math id="S2.T1.5.5.1.m1.1" class="ltx_Math" alttext="26" display="inline"><semantics id="S2.T1.5.5.1.m1.1a"><mn id="S2.T1.5.5.1.m1.1.1" xref="S2.T1.5.5.1.m1.1.1.cmml">26</mn><annotation-xml encoding="MathML-Content" id="S2.T1.5.5.1.m1.1b"><cn type="integer" id="S2.T1.5.5.1.m1.1.1.cmml" xref="S2.T1.5.5.1.m1.1.1">26</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.5.5.1.m1.1c">26</annotation></semantics></math></td>
</tr>
<tr id="S2.T1.6.6" class="ltx_tr">
<td id="S2.T1.6.6.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">Num non-embedding params (B)</td>
<td id="S2.T1.6.6.1" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><math id="S2.T1.6.6.1.m1.1" class="ltx_Math" alttext="2.58" display="inline"><semantics id="S2.T1.6.6.1.m1.1a"><mn id="S2.T1.6.6.1.m1.1.1" xref="S2.T1.6.6.1.m1.1.1.cmml">2.58</mn><annotation-xml encoding="MathML-Content" id="S2.T1.6.6.1.m1.1b"><cn type="float" id="S2.T1.6.6.1.m1.1.1.cmml" xref="S2.T1.6.6.1.m1.1.1">2.58</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.6.6.1.m1.1c">2.58</annotation></semantics></math></td>
</tr>
<tr id="S2.T1.7.7" class="ltx_tr">
<td id="S2.T1.7.7.2" class="ltx_td ltx_align_left ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">Num embedding params (B)</td>
<td id="S2.T1.7.7.1" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><math id="S2.T1.7.7.1.m1.1" class="ltx_Math" alttext="0.15" display="inline"><semantics id="S2.T1.7.7.1.m1.1a"><mn id="S2.T1.7.7.1.m1.1.1" xref="S2.T1.7.7.1.m1.1.1.cmml">0.15</mn><annotation-xml encoding="MathML-Content" id="S2.T1.7.7.1.m1.1b"><cn type="float" id="S2.T1.7.7.1.m1.1.1.cmml" xref="S2.T1.7.7.1.m1.1.1">0.15</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.7.7.1.m1.1c">0.15</annotation></semantics></math></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>AFM-on-device dimensions.</figcaption>
</figure>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S3.2.1.1" class="ltx_text ltx_font_sansserif">3</span> </span><span id="S3.3.2" class="ltx_text ltx_font_sansserif">Pre-training</span>
</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">Our AFM pre-training process plays a critical role in developing highly capable language models to power a host of Apple Intelligence features that can help and support users. We focus on efficiency and data quality at every step in order to pre-train for a high-quality end-to-end user experience with efficient and low-latency models.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S3.SS1.2.1.1" class="ltx_text ltx_font_sansserif">3.1</span> </span><span id="S3.SS1.3.2" class="ltx_text ltx_font_sansserif">Data</span>
</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">The AFM pre-training dataset consists of a diverse and high quality data mixture.
This includes data we have licensed from publishers, curated publicly-available or open-sourced datasets, and publicly available information crawled by our web-crawler, Applebot <cite class="ltx_cite ltx_citemacro_citep">[Apple, <a href="#bib.bib6" title="" class="ltx_ref">2024a</a>]</cite>. We respect the right of webpages to opt out of being crawled by Applebot, using standard robots.txt directives.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p">Given our focus on protecting user privacy, we note that no private Apple user data is included in the data mixture. Additionally, extensive efforts have been made to exclude profanity, unsafe material, and personally identifiable information from publicly available data (see <a href="#S7" title="7 Responsible AI ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 7</span></a> for more details). Rigorous decontamination is also performed against many common evaluation benchmarks.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.1" class="ltx_p">We find that data quality, much more so than quantity, is the key determining factor of downstream model performance. In the following, we provide more details about key components of the data mixture.</p>
</div>
<section id="S3.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS1.SSS1.2.1.1" class="ltx_text ltx_font_sansserif">3.1.1</span> </span><span id="S3.SS1.SSS1.3.2" class="ltx_text ltx_font_sansserif">Web pages</span>
</h4>

<div id="S3.SS1.SSS1.p1" class="ltx_para">
<p id="S3.SS1.SSS1.p1.1" class="ltx_p">We crawl publicly available information using our web crawler, Applebot <cite class="ltx_cite ltx_citemacro_citep">[Apple, <a href="#bib.bib6" title="" class="ltx_ref">2024a</a>]</cite>, and respect the rights of web publishers to opt out of Applebot using standard robots.txt directives.
Plus, we take steps to exclude pages containing profanity and apply filters to remove certain categories of personally identifiable information (PII).
The remaining documents are then processed by a pipeline which performs quality filtering and plain-text extraction, more specifically:</p>
</div>
<div id="S3.SS1.SSS1.p2" class="ltx_para">
<ol id="S3.I1" class="ltx_enumerate">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p">Body extraction is performed using a combination of Safari’s reader mode and the Boilerpipe <cite class="ltx_cite ltx_citemacro_citep">[Kohlschütter et al., <a href="#bib.bib26" title="" class="ltx_ref">2010</a>]</cite> algorithm.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p">Safety and profanity filtering, using heuristics and model-based classifiers.</p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S3.I1.i3.p1" class="ltx_para">
<p id="S3.I1.i3.p1.1" class="ltx_p">Global fuzzy de-duplication using locality-sensitive n-gram hashing.</p>
</div>
</li>
<li id="S3.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4.</span> 
<div id="S3.I1.i4.p1" class="ltx_para">
<p id="S3.I1.i4.p1.1" class="ltx_p">Extensive quality filtering using heuristics and model-based classifiers <cite class="ltx_cite ltx_citemacro_citep">[Kong et al., <a href="#bib.bib27" title="" class="ltx_ref">2024</a>; Li et al., <a href="#bib.bib32" title="" class="ltx_ref">2024a</a>]</cite>.</p>
</div>
</li>
<li id="S3.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">5.</span> 
<div id="S3.I1.i5.p1" class="ltx_para">
<p id="S3.I1.i5.p1.1" class="ltx_p">Decontamination against 811 common pre-training benchmarks, filtering entire documents upon 4-13 gram collisions with any of the benchmark datasets, unless the collision-count for a given n-gram reaches a “common-usage" threshold of 1000.</p>
</div>
</li>
</ol>
</div>
</section>
<section id="S3.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS1.SSS2.2.1.1" class="ltx_text ltx_font_sansserif">3.1.2</span> </span><span id="S3.SS1.SSS2.3.2" class="ltx_text ltx_font_sansserif">Licensed datasets</span>
</h4>

<div id="S3.SS1.SSS2.p1" class="ltx_para">
<p id="S3.SS1.SSS2.p1.1" class="ltx_p">We go to lengths to identify and license a limited amount of high-quality data from publishers.
These licensed datasets provide a natural source of diverse and high quality long-context data, so we include them as part of the data mixture for the continued and context-lengthening stages of pre-training (see <a href="#S3.SS2.SSS2" title="3.2.2 Continued pre-training ‣ 3.2 Recipe ‣ 3 Pre-training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3.2.2</span></a> and <a href="#S3.SS2.SSS3" title="3.2.3 Context lengthening ‣ 3.2 Recipe ‣ 3 Pre-training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2.3</span></a> for more details).
We decontaminate sections of publisher licensed data the same way we decontaminate web pages (<a href="#S3.SS1.SSS1" title="3.1.1 Web pages ‣ 3.1 Data ‣ 3 Pre-training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3.1.1</span></a>).</p>
</div>
</section>
<section id="S3.SS1.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS1.SSS3.2.1.1" class="ltx_text ltx_font_sansserif">3.1.3</span> </span><span id="S3.SS1.SSS3.3.2" class="ltx_text ltx_font_sansserif">Code</span>
</h4>

<div id="S3.SS1.SSS3.p1" class="ltx_para">
<p id="S3.SS1.SSS3.p1.1" class="ltx_p">Code data is obtained from license-filtered<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>
Using MIT, Apache, BSD, CC0, CC-BY, Unlicensed, ISC, and Artistic Licenses.
</span></span></span> open source repositories on GitHub. The bulk of the code data covers 14 common programming languages, including: Swift, Python, C, Objective-C, C++, JavaScript, Java, and Go.
The data is de-duplicated, further filtered for PII and quality, and decontaminated in the same fashion as in <a href="#S3.SS1.SSS1" title="3.1.1 Web pages ‣ 3.1 Data ‣ 3 Pre-training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3.1.1</span></a>.</p>
</div>
</section>
<section id="S3.SS1.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS1.SSS4.2.1.1" class="ltx_text ltx_font_sansserif">3.1.4</span> </span><span id="S3.SS1.SSS4.3.2" class="ltx_text ltx_font_sansserif">Math</span>
</h4>

<div id="S3.SS1.SSS4.p1" class="ltx_para">
<p id="S3.SS1.SSS4.p1.1" class="ltx_p">We integrate two categories of high-quality data sourced from the web. The first category is a Math Q&amp;A dataset, comprising 3 billion tokens from 20 web domains rich in math content. We extract the questions and answers by identifying relevant tags from HTML pages. The second category is a collection of 14 billion tokens from web pages such as math forums, blogs, tutorials, and seminars. To filter these web pages, we used a specialized pipeline that includes a math tag filter with a collection of 40 strings to identify mathematical templates, a math symbol filter with a collection of 350 Unicode and LaTeX symbols to identify math content, a quality filter powered by a language model classifier specifically designed for math <cite class="ltx_cite ltx_citemacro_citep">[Kong et al., <a href="#bib.bib27" title="" class="ltx_ref">2024</a>]</cite>, and a domain filter that processes all web pages from domains manually labeled by humans. We applied these filters, followed by deduplication, decontamination, and PII removal to produce the final dataset.</p>
</div>
</section>
<section id="S3.SS1.SSS5" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS1.SSS5.2.1.1" class="ltx_text ltx_font_sansserif">3.1.5</span> </span><span id="S3.SS1.SSS5.3.2" class="ltx_text ltx_font_sansserif">Public datasets</span>
</h4>

<div id="S3.SS1.SSS5.p1" class="ltx_para">
<p id="S3.SS1.SSS5.p1.1" class="ltx_p">We evaluated and selected a number of high-quality publicly-available datasets with licenses that permit use for training language models.
Then, we filtered the datasets to remove personally identifiable information before including them in the pre-training mixture.</p>
</div>
</section>
<section id="S3.SS1.SSS6" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS1.SSS6.2.1.1" class="ltx_text ltx_font_sansserif">3.1.6</span> </span><span id="S3.SS1.SSS6.3.2" class="ltx_text ltx_font_sansserif">Tokenizer</span>
</h4>

<div id="S3.SS1.SSS6.p1" class="ltx_para">
<p id="S3.SS1.SSS6.p1.1" class="ltx_p">We use a <em id="S3.SS1.SSS6.p1.1.1" class="ltx_emph ltx_font_italic">byte-pair encoding</em> (BPE) tokenizer, following the implementation from SentencePiece. All numbers are split into individual digits and we use byte-fallback to decompose unknown UTF-8 characters into byte tokens. We do not enable Unicode normalization.
The total vocabulary size is 100k and 49k tokens for AFM-server and AFM-on-device, respectively.</p>
</div>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S3.SS2.2.1.1" class="ltx_text ltx_font_sansserif">3.2</span> </span><span id="S3.SS2.3.2" class="ltx_text ltx_font_sansserif">Recipe</span>
</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">We break AFM pre-training into three distinct stages: 1. <em id="S3.SS2.p1.1.1" class="ltx_emph ltx_font_italic">core</em> which consumes most of the compute budget, 2. <em id="S3.SS2.p1.1.2" class="ltx_emph ltx_font_italic">continued</em>, where we down-weight the lower-quality bulk web-crawl data, favoring a higher code and math weight instead combined with inclusion of the licensed data described in <a href="#S3.SS1.SSS2" title="3.1.2 Licensed datasets ‣ 3.1 Data ‣ 3 Pre-training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3.1.2</span></a>, 3. <em id="S3.SS2.p1.1.3" class="ltx_emph ltx_font_italic">context-lengthening</em> which is similar to another <em id="S3.SS2.p1.1.4" class="ltx_emph ltx_font_italic">continued</em> pre-training stage, but conducted at longer sequence length and with synthetic long-context data included in the mixture.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">Details about model quality after each of the three pre-training stages (alongside additional metrics for AFM derived from our internal benchmark implementations) are in <a href="#A3" title="Appendix C Pre-training stage-breakdown evaluations ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix C</span></a>, and <a href="#A4" title="Appendix D Long-context evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix D</span></a> examines AFM-server’s long-context capabilities.</p>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.2" class="ltx_p">All three stages use decoupled weight decay <cite class="ltx_cite ltx_citemacro_citep">[Loshchilov and Hutter, <a href="#bib.bib37" title="" class="ltx_ref">2019</a>]</cite> for regularization, as well as a simplified version of <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mu" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><mi id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml">μ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><ci id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1">𝜇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">\mu</annotation></semantics></math>Param <cite class="ltx_cite ltx_citemacro_citep">[Yang et al., <a href="#bib.bib59" title="" class="ltx_ref">2022</a>]</cite>, similar to what is described as <math id="S3.SS2.p3.2.m2.1" class="ltx_Math" alttext="\mu" display="inline"><semantics id="S3.SS2.p3.2.m2.1a"><mi id="S3.SS2.p3.2.m2.1.1" xref="S3.SS2.p3.2.m2.1.1.cmml">μ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.2.m2.1b"><ci id="S3.SS2.p3.2.m2.1.1.cmml" xref="S3.SS2.p3.2.m2.1.1">𝜇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.2.m2.1c">\mu</annotation></semantics></math>Param (simple) in <cite class="ltx_cite ltx_citemacro_citep">[Wortsman et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>]</cite>. Thus far we have not found more sophisticated parameter norm controls to be necessary at these scales. All stages maintain sharded model and optimizer states in <span id="S3.SS2.p3.2.1" class="ltx_text ltx_font_typewriter">float32</span>, casting to <span id="S3.SS2.p3.2.2" class="ltx_text ltx_font_typewriter">bfloat16</span> for the forward and backward passes for efficiency.</p>
</div>
<section id="S3.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS2.SSS1.2.1.1" class="ltx_text ltx_font_sansserif">3.2.1</span> </span><span id="S3.SS2.SSS1.3.2" class="ltx_text ltx_font_sansserif">Core pre-training</span>
</h4>

<div id="S3.SS2.SSS1.p1" class="ltx_para">
<p id="S3.SS2.SSS1.p1.1" class="ltx_p">AFM-server core training is conducted from scratch, while AFM-on-device is distilled and pruned from a larger model.</p>
</div>
<section id="S3.SS2.SSS1.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_paragraph">AFM-server<span id="S3.SS2.SSS1.Px1.1.1" class="ltx_text ltx_font_medium">:</span>
</h5>

<div id="S3.SS2.SSS1.Px1.p1" class="ltx_para">
<p id="S3.SS2.SSS1.Px1.p1.5" class="ltx_p">We train AFM-server from scratch for 6.3T tokens on 8192 TPUv4 chips, using a sequence length of 4096 and a batch-size of 4096 sequences. The batch size was determined using a scaling law fit to model size and compute budget, however we find that downstream results are relatively insensitive to a fairly wide range of batch sizes, and expect that any value between <math id="S3.SS2.SSS1.Px1.p1.1.m1.1" class="ltx_math_unparsed" alttext="0.5\times" display="inline"><semantics id="S3.SS2.SSS1.Px1.p1.1.m1.1a"><mrow id="S3.SS2.SSS1.Px1.p1.1.m1.1b"><mn id="S3.SS2.SSS1.Px1.p1.1.m1.1.1">0.5</mn><mo lspace="0.222em" id="S3.SS2.SSS1.Px1.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p1.1.m1.1c">0.5\times</annotation></semantics></math> and <math id="S3.SS2.SSS1.Px1.p1.2.m2.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S3.SS2.SSS1.Px1.p1.2.m2.1a"><mrow id="S3.SS2.SSS1.Px1.p1.2.m2.1b"><mn id="S3.SS2.SSS1.Px1.p1.2.m2.1.1">2</mn><mo lspace="0.222em" id="S3.SS2.SSS1.Px1.p1.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p1.2.m2.1c">2\times</annotation></semantics></math> the predicted batch size would have yielded similar results (the predicted optimum was in fact <math id="S3.SS2.SSS1.Px1.p1.3.m3.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S3.SS2.SSS1.Px1.p1.3.m3.1a"><mo id="S3.SS2.SSS1.Px1.p1.3.m3.1.1" xref="S3.SS2.SSS1.Px1.p1.3.m3.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p1.3.m3.1b"><csymbol cd="latexml" id="S3.SS2.SSS1.Px1.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS1.Px1.p1.3.m3.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p1.3.m3.1c">\sim</annotation></semantics></math>3072, but 4096 allowed for better chip utilization).
We perform a learning rate sweep using a proxy model with a model dimension of 768, finding that the optimum learning rate spans 0.01-0.02, so we choose 0.01 to be conservative. Linear layers will have an effective learning rate scaled by <math id="S3.SS2.SSS1.Px1.p1.4.m4.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S3.SS2.SSS1.Px1.p1.4.m4.1a"><mo id="S3.SS2.SSS1.Px1.p1.4.m4.1.1" xref="S3.SS2.SSS1.Px1.p1.4.m4.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p1.4.m4.1b"><csymbol cd="latexml" id="S3.SS2.SSS1.Px1.p1.4.m4.1.1.cmml" xref="S3.SS2.SSS1.Px1.p1.4.m4.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p1.4.m4.1c">\sim</annotation></semantics></math>0.1 due to the use of <math id="S3.SS2.SSS1.Px1.p1.5.m5.1" class="ltx_Math" alttext="\mu" display="inline"><semantics id="S3.SS2.SSS1.Px1.p1.5.m5.1a"><mi id="S3.SS2.SSS1.Px1.p1.5.m5.1.1" xref="S3.SS2.SSS1.Px1.p1.5.m5.1.1.cmml">μ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p1.5.m5.1b"><ci id="S3.SS2.SSS1.Px1.p1.5.m5.1.1.cmml" xref="S3.SS2.SSS1.Px1.p1.5.m5.1.1">𝜇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p1.5.m5.1c">\mu</annotation></semantics></math>Param (simple).<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>In scaling law experiments we find that <math id="footnote2.m1.1" class="ltx_Math" alttext="\mu" display="inline"><semantics id="footnote2.m1.1b"><mi id="footnote2.m1.1.1" xref="footnote2.m1.1.1.cmml">μ</mi><annotation-xml encoding="MathML-Content" id="footnote2.m1.1c"><ci id="footnote2.m1.1.1.cmml" xref="footnote2.m1.1.1">𝜇</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m1.1d">\mu</annotation></semantics></math>Param (simple) stabilizes the optimal learning rate as model size increases, although extrapolating to very significantly deeper and/or larger models does exhibit a slight left-shift beyond what is accounted for.</span></span></span></p>
</div>
<div id="S3.SS2.SSS1.Px1.p2" class="ltx_para">
<p id="S3.SS2.SSS1.Px1.p2.1" class="ltx_p">We use a tuned decoupled weight decay of <math id="S3.SS2.SSS1.Px1.p2.1.m1.1" class="ltx_Math" alttext="3.16\mathrm{e}{-4}" display="inline"><semantics id="S3.SS2.SSS1.Px1.p2.1.m1.1a"><mrow id="S3.SS2.SSS1.Px1.p2.1.m1.1.1" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.cmml"><mrow id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.cmml"><mn id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.2" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.2.cmml">3.16</mn><mo lspace="0em" rspace="0em" id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.1" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.3" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.3.cmml">e</mi></mrow><mo id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.1" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.1.cmml">−</mo><mn id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.3" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.Px1.p2.1.m1.1b"><apply id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1"><minus id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.1.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.1"></minus><apply id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2"><times id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.1"></times><cn type="float" id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.2">3.16</cn><ci id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.2.3">e</ci></apply><cn type="integer" id="S3.SS2.SSS1.Px1.p2.1.m1.1.1.3.cmml" xref="S3.SS2.SSS1.Px1.p2.1.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.Px1.p2.1.m1.1c">3.16\mathrm{e}{-4}</annotation></semantics></math>, finding that it works well across all tested model sizes and compute budgets. The learning rate schedule includes a linear warm-up for 5000 steps, followed by cosine decay to 0.005 of the peak over the remainder of training. For further details on the optimizer, see <a href="#S3.SS2.SSS4" title="3.2.4 Optimizer ‣ 3.2 Recipe ‣ 3 Pre-training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3.2.4</span></a>. <a href="#A1" title="Appendix A Core pre-training recipe ablation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix A</span></a> compares the AFM core pre-training recipe to a more typical configuration.</p>
</div>
</section>
<section id="S3.SS2.SSS1.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_font_bold ltx_title_paragraph">AFM-on-device<span id="S3.SS2.SSS1.Px2.1.1" class="ltx_text ltx_font_medium">:</span>
</h5>

<div id="S3.SS2.SSS1.Px2.p1" class="ltx_para">
<p id="S3.SS2.SSS1.Px2.p1.1" class="ltx_p">For the on-device model, we found that knowledge distillation <cite class="ltx_cite ltx_citemacro_citep">[Hinton et al., <a href="#bib.bib20" title="" class="ltx_ref">2015</a>]</cite> and structural pruning are effective ways to improve model performance and training efficiency. These two methods are complementary to each other and work in different ways. More specifically, before training AFM-on-device, we initialize it from a pruned 6.4B model (trained from scratch using the same recipe as AFM-server), using pruning masks that are learned through a method similar to what is described in <cite class="ltx_cite ltx_citemacro_citep">[Wang et al., <a href="#bib.bib53" title="" class="ltx_ref">2020</a>; Xia et al., <a href="#bib.bib57" title="" class="ltx_ref">2023</a>]</cite>. The key differences are: (1) we only prune the hidden dimension in the feed-forward layers; (2) we use Soft-Top-K masking <cite class="ltx_cite ltx_citemacro_citep">[Lei et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>]</cite> instead of HardConcrete masking <cite class="ltx_cite ltx_citemacro_citep">[Louizos et al., <a href="#bib.bib38" title="" class="ltx_ref">2018</a>]</cite>; (3) we employ the same pre-training data mixture as the core phase to learn the mask, training for 188B tokens. Then, during the core pre-training of AFM-on-device, a distillation loss is used by replacing the target labels with a convex combination of the true labels and the teacher model’s top-1 predictions, (with 0.9 weight assigned to the teacher’s labels), training for a full 6.3T tokens. We observe that initializing from a pruned model improves both data efficiency and the final benchmark results by 0-2%, whilst adding distillation boosts MMLU and GSM8K by about 5% and 3% respectively. More detailed ablation results can be found in <a href="#A2" title="Appendix B Ablations on pruning and distillation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix B</span></a>. All training hyper-parameters except for batch-size are kept the same as AFM-server.</p>
</div>
</section>
</section>
<section id="S3.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS2.SSS2.2.1.1" class="ltx_text ltx_font_sansserif">3.2.2</span> </span><span id="S3.SS2.SSS2.3.2" class="ltx_text ltx_font_sansserif">Continued pre-training</span>
</h4>

<div id="S3.SS2.SSS2.p1" class="ltx_para">
<p id="S3.SS2.SSS2.p1.2" class="ltx_p">For both models we perform continued pre-training at a sequence length of 8192, with another 1T tokens from a mixture that upweights math and code, and down-weights the bulk web-crawl. We also include the licensed data described in <a href="#S3.SS1.SSS2" title="3.1.2 Licensed datasets ‣ 3.1 Data ‣ 3 Pre-training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3.1.2</span></a>. We use a peak learning rate of <math id="S3.SS2.SSS2.p1.1.m1.1" class="ltx_Math" alttext="3\mathrm{e}{-4}" display="inline"><semantics id="S3.SS2.SSS2.p1.1.m1.1a"><mrow id="S3.SS2.SSS2.p1.1.m1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.cmml"><mrow id="S3.SS2.SSS2.p1.1.m1.1.1.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.cmml"><mn id="S3.SS2.SSS2.p1.1.m1.1.1.2.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="S3.SS2.SSS2.p1.1.m1.1.1.2.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="S3.SS2.SSS2.p1.1.m1.1.1.2.3" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.3.cmml">e</mi></mrow><mo id="S3.SS2.SSS2.p1.1.m1.1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml">−</mo><mn id="S3.SS2.SSS2.p1.1.m1.1.1.3" xref="S3.SS2.SSS2.p1.1.m1.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.1.m1.1b"><apply id="S3.SS2.SSS2.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1"><minus id="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1"></minus><apply id="S3.SS2.SSS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.2"><times id="S3.SS2.SSS2.p1.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.1"></times><cn type="integer" id="S3.SS2.SSS2.p1.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.2">3</cn><ci id="S3.SS2.SSS2.p1.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.3">e</ci></apply><cn type="integer" id="S3.SS2.SSS2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.1.m1.1c">3\mathrm{e}{-4}</annotation></semantics></math> and decoupled weight decay of <math id="S3.SS2.SSS2.p1.2.m2.1" class="ltx_Math" alttext="1\mathrm{e}{-5}" display="inline"><semantics id="S3.SS2.SSS2.p1.2.m2.1a"><mrow id="S3.SS2.SSS2.p1.2.m2.1.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.cmml"><mrow id="S3.SS2.SSS2.p1.2.m2.1.1.2" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.cmml"><mn id="S3.SS2.SSS2.p1.2.m2.1.1.2.2" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="S3.SS2.SSS2.p1.2.m2.1.1.2.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="S3.SS2.SSS2.p1.2.m2.1.1.2.3" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.3.cmml">e</mi></mrow><mo id="S3.SS2.SSS2.p1.2.m2.1.1.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.1.cmml">−</mo><mn id="S3.SS2.SSS2.p1.2.m2.1.1.3" xref="S3.SS2.SSS2.p1.2.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.2.m2.1b"><apply id="S3.SS2.SSS2.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1"><minus id="S3.SS2.SSS2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.1"></minus><apply id="S3.SS2.SSS2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2"><times id="S3.SS2.SSS2.p1.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.1"></times><cn type="integer" id="S3.SS2.SSS2.p1.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.2">1</cn><ci id="S3.SS2.SSS2.p1.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.3">e</ci></apply><cn type="integer" id="S3.SS2.SSS2.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.2.m2.1c">1\mathrm{e}{-5}</annotation></semantics></math>, and 1000 warm-up steps with a final learning rate decay to 0.001 of peak, differently to core pre-training. Other settings (batch size, etc) are carried over. We did not find a distillation loss to be helpful here for AFM-on-device, unlike in core pre-training, so the recipe is identical to that used for AFM-server.</p>
</div>
</section>
<section id="S3.SS2.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS2.SSS3.2.1.1" class="ltx_text ltx_font_sansserif">3.2.3</span> </span><span id="S3.SS2.SSS3.3.2" class="ltx_text ltx_font_sansserif">Context lengthening</span>
</h4>

<div id="S3.SS2.SSS3.p1" class="ltx_para">
<p id="S3.SS2.SSS3.p1.2" class="ltx_p">Finally, we conduct a further 100B tokens of continued pre-training at a sequence length of <math id="S3.SS2.SSS3.p1.1.m1.1" class="ltx_Math" alttext="32768" display="inline"><semantics id="S3.SS2.SSS3.p1.1.m1.1a"><mn id="S3.SS2.SSS3.p1.1.m1.1.1" xref="S3.SS2.SSS3.p1.1.m1.1.1.cmml">32768</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p1.1.m1.1b"><cn type="integer" id="S3.SS2.SSS3.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS3.p1.1.m1.1.1">32768</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p1.1.m1.1c">32768</annotation></semantics></math> tokens, using the data mixture from the continued pre-training stage, augmented with synthetic long-context Q&amp;A data. We also increase the RoPE base frequency from 500k to <math id="S3.SS2.SSS3.p1.2.m2.1" class="ltx_Math" alttext="6315089" display="inline"><semantics id="S3.SS2.SSS3.p1.2.m2.1a"><mn id="S3.SS2.SSS3.p1.2.m2.1.1" xref="S3.SS2.SSS3.p1.2.m2.1.1.cmml">6315089</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p1.2.m2.1b"><cn type="integer" id="S3.SS2.SSS3.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS3.p1.2.m2.1.1">6315089</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p1.2.m2.1c">6315089</annotation></semantics></math>, following the scaling laws described in <cite class="ltx_cite ltx_citemacro_citep">[Liu et al., <a href="#bib.bib36" title="" class="ltx_ref">2024</a>]</cite>, with the expectation that this will allow for better short-to-long generalization—which is desirable given that the majority of our pre-training data is comprised of documents that are significantly shorter than 32k tokens long. The recipe is similar to that used for continued pre-training. We examine the long-context performance of AFM-server in <a href="#A4" title="Appendix D Long-context evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix D</span></a>.</p>
</div>
</section>
<section id="S3.SS2.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S3.SS2.SSS4.2.1.1" class="ltx_text ltx_font_sansserif">3.2.4</span> </span><span id="S3.SS2.SSS4.3.2" class="ltx_text ltx_font_sansserif">Optimizer</span>
</h4>

<div id="S3.SS2.SSS4.p1" class="ltx_para">
<p id="S3.SS2.SSS4.p1.5" class="ltx_p">We choose to use a variant of RMSProp <cite class="ltx_cite ltx_citemacro_citep">[Hinton, <a href="#bib.bib19" title="" class="ltx_ref">2012</a>]</cite> with momentum for AFM pre-training. In particular, we divide the raw gradient by the square-root of a bias-corrected exponential moving average of the squared gradient to produce an instantaneous update, which is clipped to a maximum norm of <math id="S3.SS2.SSS4.p1.1.m1.1" class="ltx_Math" alttext="1.0" display="inline"><semantics id="S3.SS2.SSS4.p1.1.m1.1a"><mn id="S3.SS2.SSS4.p1.1.m1.1.1" xref="S3.SS2.SSS4.p1.1.m1.1.1.cmml">1.0</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS4.p1.1.m1.1b"><cn type="float" id="S3.SS2.SSS4.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS4.p1.1.m1.1.1">1.0</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS4.p1.1.m1.1c">1.0</annotation></semantics></math> per parameter block, before then further smoothing this estimate over steps with an exponential moving average without bias-correction to produce the net update. Unless otherwise noted, the smoothing constants for both the squared gradient (<math id="S3.SS2.SSS4.p1.2.m2.1" class="ltx_Math" alttext="\beta_{2}" display="inline"><semantics id="S3.SS2.SSS4.p1.2.m2.1a"><msub id="S3.SS2.SSS4.p1.2.m2.1.1" xref="S3.SS2.SSS4.p1.2.m2.1.1.cmml"><mi id="S3.SS2.SSS4.p1.2.m2.1.1.2" xref="S3.SS2.SSS4.p1.2.m2.1.1.2.cmml">β</mi><mn id="S3.SS2.SSS4.p1.2.m2.1.1.3" xref="S3.SS2.SSS4.p1.2.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS4.p1.2.m2.1b"><apply id="S3.SS2.SSS4.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS4.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS4.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.SSS4.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS4.p1.2.m2.1.1.2">𝛽</ci><cn type="integer" id="S3.SS2.SSS4.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS4.p1.2.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS4.p1.2.m2.1c">\beta_{2}</annotation></semantics></math>) and the update (<math id="S3.SS2.SSS4.p1.3.m3.1" class="ltx_Math" alttext="\beta_{1}" display="inline"><semantics id="S3.SS2.SSS4.p1.3.m3.1a"><msub id="S3.SS2.SSS4.p1.3.m3.1.1" xref="S3.SS2.SSS4.p1.3.m3.1.1.cmml"><mi id="S3.SS2.SSS4.p1.3.m3.1.1.2" xref="S3.SS2.SSS4.p1.3.m3.1.1.2.cmml">β</mi><mn id="S3.SS2.SSS4.p1.3.m3.1.1.3" xref="S3.SS2.SSS4.p1.3.m3.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS4.p1.3.m3.1b"><apply id="S3.SS2.SSS4.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS4.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS4.p1.3.m3.1.1.1.cmml" xref="S3.SS2.SSS4.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.SSS4.p1.3.m3.1.1.2.cmml" xref="S3.SS2.SSS4.p1.3.m3.1.1.2">𝛽</ci><cn type="integer" id="S3.SS2.SSS4.p1.3.m3.1.1.3.cmml" xref="S3.SS2.SSS4.p1.3.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS4.p1.3.m3.1c">\beta_{1}</annotation></semantics></math>) are set to <math id="S3.SS2.SSS4.p1.4.m4.1" class="ltx_Math" alttext="0.95" display="inline"><semantics id="S3.SS2.SSS4.p1.4.m4.1a"><mn id="S3.SS2.SSS4.p1.4.m4.1.1" xref="S3.SS2.SSS4.p1.4.m4.1.1.cmml">0.95</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS4.p1.4.m4.1b"><cn type="float" id="S3.SS2.SSS4.p1.4.m4.1.1.cmml" xref="S3.SS2.SSS4.p1.4.m4.1.1">0.95</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS4.p1.4.m4.1c">0.95</annotation></semantics></math>. A small constant <math id="S3.SS2.SSS4.p1.5.m5.1" class="ltx_Math" alttext="\epsilon=1\mathrm{e}{-30}" display="inline"><semantics id="S3.SS2.SSS4.p1.5.m5.1a"><mrow id="S3.SS2.SSS4.p1.5.m5.1.1" xref="S3.SS2.SSS4.p1.5.m5.1.1.cmml"><mi id="S3.SS2.SSS4.p1.5.m5.1.1.2" xref="S3.SS2.SSS4.p1.5.m5.1.1.2.cmml">ϵ</mi><mo id="S3.SS2.SSS4.p1.5.m5.1.1.1" xref="S3.SS2.SSS4.p1.5.m5.1.1.1.cmml">=</mo><mrow id="S3.SS2.SSS4.p1.5.m5.1.1.3" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.cmml"><mrow id="S3.SS2.SSS4.p1.5.m5.1.1.3.2" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.2.cmml"><mn id="S3.SS2.SSS4.p1.5.m5.1.1.3.2.2" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.2.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="S3.SS2.SSS4.p1.5.m5.1.1.3.2.1" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.2.1.cmml">​</mo><mi mathvariant="normal" id="S3.SS2.SSS4.p1.5.m5.1.1.3.2.3" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.2.3.cmml">e</mi></mrow><mo id="S3.SS2.SSS4.p1.5.m5.1.1.3.1" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.1.cmml">−</mo><mn id="S3.SS2.SSS4.p1.5.m5.1.1.3.3" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.3.cmml">30</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS4.p1.5.m5.1b"><apply id="S3.SS2.SSS4.p1.5.m5.1.1.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1"><eq id="S3.SS2.SSS4.p1.5.m5.1.1.1.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1.1"></eq><ci id="S3.SS2.SSS4.p1.5.m5.1.1.2.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1.2">italic-ϵ</ci><apply id="S3.SS2.SSS4.p1.5.m5.1.1.3.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1.3"><minus id="S3.SS2.SSS4.p1.5.m5.1.1.3.1.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.1"></minus><apply id="S3.SS2.SSS4.p1.5.m5.1.1.3.2.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.2"><times id="S3.SS2.SSS4.p1.5.m5.1.1.3.2.1.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.2.1"></times><cn type="integer" id="S3.SS2.SSS4.p1.5.m5.1.1.3.2.2.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.2.2">1</cn><ci id="S3.SS2.SSS4.p1.5.m5.1.1.3.2.3.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.2.3">e</ci></apply><cn type="integer" id="S3.SS2.SSS4.p1.5.m5.1.1.3.3.cmml" xref="S3.SS2.SSS4.p1.5.m5.1.1.3.3">30</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS4.p1.5.m5.1c">\epsilon=1\mathrm{e}{-30}</annotation></semantics></math> is added to the instantaneous squared gradient prior to smoothing, for numerical stability.</p>
</div>
<section id="S3.SS2.SSS4.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="S3.SS2.SSS4.Px1.p1" class="ltx_para">
<p id="S3.SS2.SSS4.Px1.p1.1" class="ltx_p">The smoothed updates are scaled by the learning rate, weight-decay is added, and then scheduled decay is applied to form the final weight delta. As an additional guard for stability, prior to the optimizer we clip the global gradient norm to <math id="S3.SS2.SSS4.Px1.p1.1.m1.1" class="ltx_Math" alttext="1.0" display="inline"><semantics id="S3.SS2.SSS4.Px1.p1.1.m1.1a"><mn id="S3.SS2.SSS4.Px1.p1.1.m1.1.1" xref="S3.SS2.SSS4.Px1.p1.1.m1.1.1.cmml">1.0</mn><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS4.Px1.p1.1.m1.1b"><cn type="float" id="S3.SS2.SSS4.Px1.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS4.Px1.p1.1.m1.1.1">1.0</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS4.Px1.p1.1.m1.1c">1.0</annotation></semantics></math>. For a recipe ablation against a more typical configuration, see <a href="#A1" title="Appendix A Core pre-training recipe ablation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix A</span></a>.</p>
</div>
</section>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S3.SS3.2.1.1" class="ltx_text ltx_font_sansserif">3.3</span> </span><span id="S3.SS3.3.2" class="ltx_text ltx_font_sansserif">Training infrastructure</span>
</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">The AFM models are pre-trained on v4 and v5p Cloud TPU clusters with the AXLearn framework <cite class="ltx_cite ltx_citemacro_citep">[Apple, <a href="#bib.bib5" title="" class="ltx_ref">2023</a>]</cite>, a JAX <cite class="ltx_cite ltx_citemacro_citep">[Bradbury et al., <a href="#bib.bib9" title="" class="ltx_ref">2018</a>]</cite> based deep learning library designed for the public cloud.
Training is conducted using a combination of tensor, fully-sharded-data-parallel, and sequence parallelism, allowing training to scale to a large number of model parameters and sequence lengths at high utilization.
This system allows us to train the AFM models efficiently and scalably, including AFM-on-device, AFM-server, and larger models.</p>
</div>
<section id="S3.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="S3.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS3.SSS0.Px1.p1.1" class="ltx_p">AFM-server was trained on 8192 TPUv4 chips provisioned as <math id="S3.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="8\times 1024" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.1.m1.1a"><mrow id="S3.SS3.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml"><mn id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml">8</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml">×</mo><mn id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml">1024</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.1.m1.1b"><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1"><times id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1"></times><cn type="integer" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2">8</cn><cn type="integer" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3">1024</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.1.m1.1c">8\times 1024</annotation></semantics></math> chip slices, where slices are connected together by the data-center network (DCN) <cite class="ltx_cite ltx_citemacro_citep">[Chowdhery et al., <a href="#bib.bib11" title="" class="ltx_ref">2022</a>]</cite>. Only data-parallelism crosses the slice boundary, other types of state sharding are within-slice only as the within-slice interconnect bandwidth is orders of magnitude higher than the DCN. The sustained model-flop-utilization (MFU) for this training run was approximately 52%. AFM-on-device was trained on one slice of 2048 TPUv5p chips.</p>
</div>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S4.2.1.1" class="ltx_text ltx_font_sansserif">4</span> </span><span id="S4.3.2" class="ltx_text ltx_font_sansserif">Post-Training</span>
</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">While Apple Intelligence features are powered through adapters on top of the base model (see <a href="#S5" title="5 Powering Apple Intelligence features ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 5</span></a> for a deep-dive on the adapter architecture), empirically we found that improving the general-purpose post-training lifts the performance of all features, as the models have stronger capabilities on instruction following, reasoning, and writing.</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p">We conduct extensive research in post-training methods to instill general-purpose instruction following and conversation capabilities to the pre-trained AFM models.
Our goal is to ensure these model capabilities are aligned with Apple’s core values and principles, including our commitment to protecting user privacy, and our Responsible AI principles.
Our post-training efforts include a series of data collection and generation, instruction tuning, and alignment innovations.
Our post-training process contains two stages: supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF).
We present two new post-training algorithms: (1) a rejection sampling fine-tuning algorithm with teacher committee (iTeC), and (2) a reinforcement learning from human feedback (RLHF) algorithm with mirror descent policy optimization and a leave-one-out advantage estimator (MDLOO) that are used on our reinforcement learning iterations and lead to significant model quality improvements.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S4.SS1.2.1.1" class="ltx_text ltx_font_sansserif">4.1</span> </span><span id="S4.SS1.3.2" class="ltx_text ltx_font_sansserif">Data</span>
</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We use a hybrid data strategy in our post-training pipeline, which consists of both human annotated and synthetic data.
Throughout our data collection and experiment process, we have found data quality to be the key to model success and thus have conducted extensive data curation and filtering procedures.</p>
</div>
<section id="S4.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S4.SS1.SSS1.2.1.1" class="ltx_text ltx_font_sansserif">4.1.1</span> </span><span id="S4.SS1.SSS1.3.2" class="ltx_text ltx_font_sansserif">Human annotations</span>
</h4>

<section id="S4.SS1.SSS1.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Demonstration data</h5>

<div id="S4.SS1.SSS1.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS1.Px1.p1.1" class="ltx_p">To fuel the instruction fine-tuning of AFM,
we collect high-quality human annotated demonstration datasets from various sources.
This dialogue-style data consists of both system-level and task-level instructions (a.k.a. prompts),
as well as their corresponding responses.
Similar to <cite class="ltx_cite ltx_citemacro_citep">[Zhou et al., <a href="#bib.bib64" title="" class="ltx_ref">2024</a>]</cite>,
we observe quality to weigh more importantly than quantity in our experiments.
As a result,
we focus on key data quality criteria including helpfulness, harmlessness, presentation, and response accuracy,
in addition to targeting a diverse task distribution covering Apple Intelligence features.
To protect user privacy,
we take steps to verify no personally identifiable information is present in our data, and we do not include any personal data stored by users with Apple.</p>
</div>
</section>
<section id="S4.SS1.SSS1.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Human preference feedback</h5>

<div id="S4.SS1.SSS1.Px2.p1" class="ltx_para">
<p id="S4.SS1.SSS1.Px2.p1.1" class="ltx_p">To iteratively improve AFM’s capabilities,
we further collect human feedback for reinforcement learning.
In particular,
we instruct human annotators to compare and rank two model responses for the same prompt to collect side-by-side preference labels.
In addition, we also use single-side questions to guide this process.
These questions inform raters to grade the model response quality of various aspects including instruction following, safety, factuality, and presentation,
and we also retain these labels for model training.
We emphasize Apple values and standards in the process.
Similar to demonstration data,
we find data quality to be crucial for feedback data,
and thus we iterate data and model qualities jointly to improve them in a unified flywheel.</p>
</div>
</section>
</section>
<section id="S4.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S4.SS1.SSS2.2.1.1" class="ltx_text ltx_font_sansserif">4.1.2</span> </span><span id="S4.SS1.SSS2.3.2" class="ltx_text ltx_font_sansserif">Synthetic data</span>
</h4>

<div id="S4.SS1.SSS2.p1" class="ltx_para">
<p id="S4.SS1.SSS2.p1.1" class="ltx_p">In addition to human annotations, we delve into enhancing data quality and diversity through synthetic data generation. Our findings suggest that when guided by our robust reward models, AFMs are capable of generating high quality responses and for some specific domains, these responses are found to be on par with, or even superior to, human annotations. Therefore, we extend our prompt set to increase the diversity and find that those generated responses can benefit AFMs themselves. In the following, we discuss three domains where we generate synthetic data for AFM post-training: mathematics, tool use, and coding.</p>
</div>
<section id="S4.SS1.SSS2.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Mathematics</h5>

<div id="S4.SS1.SSS2.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS2.Px1.p1.1" class="ltx_p">In the field of mathematics, the wide-ranging subjects and difficulty level make it exceptionally resource-intensive for collecting human demonstrations, since it requires expert knowledge from the human writers. It also becomes impractical to solely rely on human-written content as the model continuously improves. As a consequence, exploring the potential of synthetic data becomes essential to effectively address the challenges.</p>
</div>
<div id="S4.SS1.SSS2.Px1.p2" class="ltx_para">
<p id="S4.SS1.SSS2.Px1.p2.1" class="ltx_p">The creation of synthetic data for mathematics involves two primary stages: generating synthetic math problems and producing their corresponding solutions. For math problem synthesis, we employ several “evolution" strategies where a seed set of prompts are transformed into a much larger set of diverse prompts:</p>
</div>
<div id="S4.SS1.SSS2.Px1.p3" class="ltx_para">
<ul id="S4.I1" class="ltx_itemize">
<li id="S4.I1.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"></span>
<div id="S4.I1.ix1.p1" class="ltx_para">
<p id="S4.I1.ix1.p1.1" class="ltx_p"><span id="S4.I1.ix1.p1.1.1" class="ltx_text ltx_font_bold">Problem rephrase and reversion.</span> Following the approach in <cite class="ltx_cite ltx_citemacro_citep">[Yu et al., <a href="#bib.bib60" title="" class="ltx_ref">2023</a>]</cite>, we prompt AFM to rephrase seed math questions, and curate reverse questions to derive a specific number in a raw problem statement when provided with the final answer.</p>
</div>
</li>
<li id="S4.I1.ix2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"></span>
<div id="S4.I1.ix2.p1" class="ltx_para">
<p id="S4.I1.ix2.p1.6" class="ltx_p"><span id="S4.I1.ix2.p1.6.1" class="ltx_text ltx_font_bold">Problem evolution.</span> Inspired by the instruction evolving technique <cite class="ltx_cite ltx_citemacro_citep">[Xu et al., <a href="#bib.bib58" title="" class="ltx_ref">2023</a>]</cite>, given a seed problem set <math id="S4.I1.ix2.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{D}_{\text{seed}}" display="inline"><semantics id="S4.I1.ix2.p1.1.m1.1a"><msub id="S4.I1.ix2.p1.1.m1.1.1" xref="S4.I1.ix2.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.ix2.p1.1.m1.1.1.2" xref="S4.I1.ix2.p1.1.m1.1.1.2.cmml">𝒟</mi><mtext id="S4.I1.ix2.p1.1.m1.1.1.3" xref="S4.I1.ix2.p1.1.m1.1.1.3a.cmml">seed</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.I1.ix2.p1.1.m1.1b"><apply id="S4.I1.ix2.p1.1.m1.1.1.cmml" xref="S4.I1.ix2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.I1.ix2.p1.1.m1.1.1.1.cmml" xref="S4.I1.ix2.p1.1.m1.1.1">subscript</csymbol><ci id="S4.I1.ix2.p1.1.m1.1.1.2.cmml" xref="S4.I1.ix2.p1.1.m1.1.1.2">𝒟</ci><ci id="S4.I1.ix2.p1.1.m1.1.1.3a.cmml" xref="S4.I1.ix2.p1.1.m1.1.1.3"><mtext mathsize="70%" id="S4.I1.ix2.p1.1.m1.1.1.3.cmml" xref="S4.I1.ix2.p1.1.m1.1.1.3">seed</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.ix2.p1.1.m1.1c">\mathcal{D}_{\text{seed}}</annotation></semantics></math> we prompt AFM to generate two distinct sets of math problems, i.e.
<math id="S4.I1.ix2.p1.2.m2.1" class="ltx_Math" alttext="F(\mathcal{D}_{\text{seed}})\xrightarrow[\text{}]{\scriptstyle\text{depth}}\mathcal{D}_{\text{depth}}" display="inline"><semantics id="S4.I1.ix2.p1.2.m2.1a"><mrow id="S4.I1.ix2.p1.2.m2.1.1" xref="S4.I1.ix2.p1.2.m2.1.1.cmml"><mrow id="S4.I1.ix2.p1.2.m2.1.1.1" xref="S4.I1.ix2.p1.2.m2.1.1.1.cmml"><mi id="S4.I1.ix2.p1.2.m2.1.1.1.3" xref="S4.I1.ix2.p1.2.m2.1.1.1.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S4.I1.ix2.p1.2.m2.1.1.1.2" xref="S4.I1.ix2.p1.2.m2.1.1.1.2.cmml">​</mo><mrow id="S4.I1.ix2.p1.2.m2.1.1.1.1.1" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.2" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.cmml">(</mo><msub id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.2" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.2.cmml">𝒟</mi><mtext id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.3" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.3a.cmml">seed</mtext></msub><mo stretchy="false" id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.3" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><munderover accent="true" accentunder="true" id="S4.I1.ix2.p1.2.m2.1.1.2" xref="S4.I1.ix2.p1.2.m2.1.1.2.cmml"><mo stretchy="false" id="S4.I1.ix2.p1.2.m2.1.1.2.2.2" xref="S4.I1.ix2.p1.2.m2.1.1.2.2.2.cmml">→</mo><mrow id="S4.I1.ix2.p1.2.m2.1.1.2.1" xref="S4.I1.ix2.p1.2.m2.1.1.2.1a.cmml"></mrow><mtext mathsize="70%" id="S4.I1.ix2.p1.2.m2.1.1.2.2.1" xref="S4.I1.ix2.p1.2.m2.1.1.2.2.1a.cmml">depth</mtext></munderover><msub id="S4.I1.ix2.p1.2.m2.1.1.3" xref="S4.I1.ix2.p1.2.m2.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" mathsize="70%" id="S4.I1.ix2.p1.2.m2.1.1.3.2" xref="S4.I1.ix2.p1.2.m2.1.1.3.2.cmml">𝒟</mi><mtext mathsize="71%" id="S4.I1.ix2.p1.2.m2.1.1.3.3" xref="S4.I1.ix2.p1.2.m2.1.1.3.3a.cmml">depth</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.ix2.p1.2.m2.1b"><apply id="S4.I1.ix2.p1.2.m2.1.1.cmml" xref="S4.I1.ix2.p1.2.m2.1.1"><apply id="S4.I1.ix2.p1.2.m2.1.1.2.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.2"><ci id="S4.I1.ix2.p1.2.m2.1.1.2.1a.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.2.1"><mrow id="S4.I1.ix2.p1.2.m2.1.1.2.1.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.2.1"></mrow></ci><apply id="S4.I1.ix2.p1.2.m2.1.1.2.2.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.2"><ci id="S4.I1.ix2.p1.2.m2.1.1.2.2.1a.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.2.2.1"><mtext mathsize="70%" id="S4.I1.ix2.p1.2.m2.1.1.2.2.1.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.2.2.1">depth</mtext></ci><ci id="S4.I1.ix2.p1.2.m2.1.1.2.2.2.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.2.2.2">→</ci></apply></apply><apply id="S4.I1.ix2.p1.2.m2.1.1.1.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.1"><times id="S4.I1.ix2.p1.2.m2.1.1.1.2.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.1.2"></times><ci id="S4.I1.ix2.p1.2.m2.1.1.1.3.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.1.3">𝐹</ci><apply id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1">subscript</csymbol><ci id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.2.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.2">𝒟</ci><ci id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.3a.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.3"><mtext mathsize="70%" id="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.3.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.1.1.1.1.3">seed</mtext></ci></apply></apply><apply id="S4.I1.ix2.p1.2.m2.1.1.3.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S4.I1.ix2.p1.2.m2.1.1.3.1.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.3">subscript</csymbol><ci id="S4.I1.ix2.p1.2.m2.1.1.3.2.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.3.2">𝒟</ci><ci id="S4.I1.ix2.p1.2.m2.1.1.3.3a.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.3.3"><mtext mathsize="50%" id="S4.I1.ix2.p1.2.m2.1.1.3.3.cmml" xref="S4.I1.ix2.p1.2.m2.1.1.3.3">depth</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.ix2.p1.2.m2.1c">F(\mathcal{D}_{\text{seed}})\xrightarrow[\text{}]{\scriptstyle\text{depth}}\mathcal{D}_{\text{depth}}</annotation></semantics></math>, and <math id="S4.I1.ix2.p1.3.m3.1" class="ltx_Math" alttext="F(\mathcal{D}_{\text{seed}})\xrightarrow[\text{}]{\scriptstyle\text{breadth}}\mathcal{D}_{\text{breadth}}" display="inline"><semantics id="S4.I1.ix2.p1.3.m3.1a"><mrow id="S4.I1.ix2.p1.3.m3.1.1" xref="S4.I1.ix2.p1.3.m3.1.1.cmml"><mrow id="S4.I1.ix2.p1.3.m3.1.1.1" xref="S4.I1.ix2.p1.3.m3.1.1.1.cmml"><mi id="S4.I1.ix2.p1.3.m3.1.1.1.3" xref="S4.I1.ix2.p1.3.m3.1.1.1.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S4.I1.ix2.p1.3.m3.1.1.1.2" xref="S4.I1.ix2.p1.3.m3.1.1.1.2.cmml">​</mo><mrow id="S4.I1.ix2.p1.3.m3.1.1.1.1.1" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.2" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.cmml">(</mo><msub id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.2" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.2.cmml">𝒟</mi><mtext id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.3" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.3a.cmml">seed</mtext></msub><mo stretchy="false" id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.3" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><munderover accent="true" accentunder="true" id="S4.I1.ix2.p1.3.m3.1.1.2" xref="S4.I1.ix2.p1.3.m3.1.1.2.cmml"><mo stretchy="false" id="S4.I1.ix2.p1.3.m3.1.1.2.2.2" xref="S4.I1.ix2.p1.3.m3.1.1.2.2.2.cmml">→</mo><mrow id="S4.I1.ix2.p1.3.m3.1.1.2.1" xref="S4.I1.ix2.p1.3.m3.1.1.2.1a.cmml"></mrow><mtext mathsize="70%" id="S4.I1.ix2.p1.3.m3.1.1.2.2.1" xref="S4.I1.ix2.p1.3.m3.1.1.2.2.1a.cmml">breadth</mtext></munderover><msub id="S4.I1.ix2.p1.3.m3.1.1.3" xref="S4.I1.ix2.p1.3.m3.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" mathsize="70%" id="S4.I1.ix2.p1.3.m3.1.1.3.2" xref="S4.I1.ix2.p1.3.m3.1.1.3.2.cmml">𝒟</mi><mtext mathsize="71%" id="S4.I1.ix2.p1.3.m3.1.1.3.3" xref="S4.I1.ix2.p1.3.m3.1.1.3.3a.cmml">breadth</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.ix2.p1.3.m3.1b"><apply id="S4.I1.ix2.p1.3.m3.1.1.cmml" xref="S4.I1.ix2.p1.3.m3.1.1"><apply id="S4.I1.ix2.p1.3.m3.1.1.2.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.2"><ci id="S4.I1.ix2.p1.3.m3.1.1.2.1a.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.2.1"><mrow id="S4.I1.ix2.p1.3.m3.1.1.2.1.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.2.1"></mrow></ci><apply id="S4.I1.ix2.p1.3.m3.1.1.2.2.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.2"><ci id="S4.I1.ix2.p1.3.m3.1.1.2.2.1a.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.2.2.1"><mtext mathsize="70%" id="S4.I1.ix2.p1.3.m3.1.1.2.2.1.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.2.2.1">breadth</mtext></ci><ci id="S4.I1.ix2.p1.3.m3.1.1.2.2.2.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.2.2.2">→</ci></apply></apply><apply id="S4.I1.ix2.p1.3.m3.1.1.1.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.1"><times id="S4.I1.ix2.p1.3.m3.1.1.1.2.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.1.2"></times><ci id="S4.I1.ix2.p1.3.m3.1.1.1.3.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.1.3">𝐹</ci><apply id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.1.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1">subscript</csymbol><ci id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.2.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.2">𝒟</ci><ci id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.3a.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.3"><mtext mathsize="70%" id="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.3.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.1.1.1.1.3">seed</mtext></ci></apply></apply><apply id="S4.I1.ix2.p1.3.m3.1.1.3.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S4.I1.ix2.p1.3.m3.1.1.3.1.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.3">subscript</csymbol><ci id="S4.I1.ix2.p1.3.m3.1.1.3.2.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.3.2">𝒟</ci><ci id="S4.I1.ix2.p1.3.m3.1.1.3.3a.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.3.3"><mtext mathsize="50%" id="S4.I1.ix2.p1.3.m3.1.1.3.3.cmml" xref="S4.I1.ix2.p1.3.m3.1.1.3.3">breadth</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.ix2.p1.3.m3.1c">F(\mathcal{D}_{\text{seed}})\xrightarrow[\text{}]{\scriptstyle\text{breadth}}\mathcal{D}_{\text{breadth}}</annotation></semantics></math>. The in-depth evolution enhances instructions by adding complexities while the in-breadth evolution improves the topic coverage.
For both <math id="S4.I1.ix2.p1.4.m4.1" class="ltx_Math" alttext="\mathcal{D}_{\text{breadth}}" display="inline"><semantics id="S4.I1.ix2.p1.4.m4.1a"><msub id="S4.I1.ix2.p1.4.m4.1.1" xref="S4.I1.ix2.p1.4.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.ix2.p1.4.m4.1.1.2" xref="S4.I1.ix2.p1.4.m4.1.1.2.cmml">𝒟</mi><mtext id="S4.I1.ix2.p1.4.m4.1.1.3" xref="S4.I1.ix2.p1.4.m4.1.1.3a.cmml">breadth</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.I1.ix2.p1.4.m4.1b"><apply id="S4.I1.ix2.p1.4.m4.1.1.cmml" xref="S4.I1.ix2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.I1.ix2.p1.4.m4.1.1.1.cmml" xref="S4.I1.ix2.p1.4.m4.1.1">subscript</csymbol><ci id="S4.I1.ix2.p1.4.m4.1.1.2.cmml" xref="S4.I1.ix2.p1.4.m4.1.1.2">𝒟</ci><ci id="S4.I1.ix2.p1.4.m4.1.1.3a.cmml" xref="S4.I1.ix2.p1.4.m4.1.1.3"><mtext mathsize="70%" id="S4.I1.ix2.p1.4.m4.1.1.3.cmml" xref="S4.I1.ix2.p1.4.m4.1.1.3">breadth</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.ix2.p1.4.m4.1c">\mathcal{D}_{\text{breadth}}</annotation></semantics></math> and <math id="S4.I1.ix2.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{D}_{\text{depth}}" display="inline"><semantics id="S4.I1.ix2.p1.5.m5.1a"><msub id="S4.I1.ix2.p1.5.m5.1.1" xref="S4.I1.ix2.p1.5.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.ix2.p1.5.m5.1.1.2" xref="S4.I1.ix2.p1.5.m5.1.1.2.cmml">𝒟</mi><mtext id="S4.I1.ix2.p1.5.m5.1.1.3" xref="S4.I1.ix2.p1.5.m5.1.1.3a.cmml">depth</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.I1.ix2.p1.5.m5.1b"><apply id="S4.I1.ix2.p1.5.m5.1.1.cmml" xref="S4.I1.ix2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S4.I1.ix2.p1.5.m5.1.1.1.cmml" xref="S4.I1.ix2.p1.5.m5.1.1">subscript</csymbol><ci id="S4.I1.ix2.p1.5.m5.1.1.2.cmml" xref="S4.I1.ix2.p1.5.m5.1.1.2">𝒟</ci><ci id="S4.I1.ix2.p1.5.m5.1.1.3a.cmml" xref="S4.I1.ix2.p1.5.m5.1.1.3"><mtext mathsize="70%" id="S4.I1.ix2.p1.5.m5.1.1.3.cmml" xref="S4.I1.ix2.p1.5.m5.1.1.3">depth</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.ix2.p1.5.m5.1c">\mathcal{D}_{\text{depth}}</annotation></semantics></math>, we first perform de-duplication with an embedding model, and subsequently prompt LLMs to ensure the coherence and solvability of the math problems. In addition, for <math id="S4.I1.ix2.p1.6.m6.1" class="ltx_Math" alttext="\mathcal{D}_{\text{depth}}" display="inline"><semantics id="S4.I1.ix2.p1.6.m6.1a"><msub id="S4.I1.ix2.p1.6.m6.1.1" xref="S4.I1.ix2.p1.6.m6.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.I1.ix2.p1.6.m6.1.1.2" xref="S4.I1.ix2.p1.6.m6.1.1.2.cmml">𝒟</mi><mtext id="S4.I1.ix2.p1.6.m6.1.1.3" xref="S4.I1.ix2.p1.6.m6.1.1.3a.cmml">depth</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.I1.ix2.p1.6.m6.1b"><apply id="S4.I1.ix2.p1.6.m6.1.1.cmml" xref="S4.I1.ix2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S4.I1.ix2.p1.6.m6.1.1.1.cmml" xref="S4.I1.ix2.p1.6.m6.1.1">subscript</csymbol><ci id="S4.I1.ix2.p1.6.m6.1.1.2.cmml" xref="S4.I1.ix2.p1.6.m6.1.1.2">𝒟</ci><ci id="S4.I1.ix2.p1.6.m6.1.1.3a.cmml" xref="S4.I1.ix2.p1.6.m6.1.1.3"><mtext mathsize="70%" id="S4.I1.ix2.p1.6.m6.1.1.3.cmml" xref="S4.I1.ix2.p1.6.m6.1.1.3">depth</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.ix2.p1.6.m6.1c">\mathcal{D}_{\text{depth}}</annotation></semantics></math> a difficulty level is assigned and we only select math problems that score above a specified threshold.</p>
</div>
</li>
</ul>
</div>
<div id="S4.SS1.SSS2.Px1.p4" class="ltx_para">
<p id="S4.SS1.SSS2.Px1.p4.1" class="ltx_p">With an augmented set of math questions, we then prompt AFM to synthesize <math id="S4.SS1.SSS2.Px1.p4.1.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S4.SS1.SSS2.Px1.p4.1.m1.1a"><mi id="S4.SS1.SSS2.Px1.p4.1.m1.1.1" xref="S4.SS1.SSS2.Px1.p4.1.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.Px1.p4.1.m1.1b"><ci id="S4.SS1.SSS2.Px1.p4.1.m1.1.1.cmml" xref="S4.SS1.SSS2.Px1.p4.1.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.Px1.p4.1.m1.1c">N</annotation></semantics></math> responses with chain-of-thought per question. If the initial seed data has ground truth, they can be used as an “outcome reward signal” to filter synthesized answers.
For problems that require less reasoning steps, we observe that a correct final answer often gets associated with correct intermediate steps. If direct answer checking is unsuccessful or ground truth is unavailable, we instead assess the response correctness by querying an LLM judge. We find that the filtered answers, when fed into the training data, boost our models’ math capabilities by a large margin.</p>
</div>
</section>
<section id="S4.SS1.SSS2.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Tool use</h5>

<div id="S4.SS1.SSS2.Px2.p1" class="ltx_para">
<p id="S4.SS1.SSS2.Px2.p1.1" class="ltx_p">We develop tool-use capabilities such as function call, code interpreter, and browsing through a mixture of synthetic and human data.
The model capabilities are first bootstrapped with synthetic data, which focuses on single-tool use cases.
We then collect human annotations to improve model capabilities that involve multi-tool and multi-step scenarios.
We further augment the human curated function call data by mixing the oracle tool with other similar tools to increase the difficulty of tool selection.
In addition, we synthesize parallel function call from human curated function call data to enable the new capability and tool intent detection data based on human curated function call and general SFT data to mitigate tool call over-triggering issues.</p>
</div>
</section>
<section id="S4.SS1.SSS2.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Coding</h5>

<div id="S4.SS1.SSS2.Px3.p1" class="ltx_para">
<p id="S4.SS1.SSS2.Px3.p1.1" class="ltx_p">The generation of a synthetic coding dataset involves a self-instruct method with rejection sampling. This approach enables the model to learn and generate data autonomously. Starting with 71 different programming topics as the seeds, the model is prompted to generate an initial pool of coding interview-like questions. For each question, the model generates a set of unit tests and a number of potential solutions. We then use an execution-based rejection sampling method to select the best solution. This involves compiling each potential solution with every unit test and executing them. The solution with the highest number of successful executions is chosen. This results in a collection of (question, test cases, solution) triplets. At the end, we validate the quality of the dataset by filtering the triplets using the number of passed unit tests, resulting in 12K high quality triplets used in the SFT.</p>
</div>
</section>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S4.SS2.2.1.1" class="ltx_text ltx_font_sansserif">4.2</span> </span><span id="S4.SS2.3.2" class="ltx_text ltx_font_sansserif">Supervised fine-tuning (SFT)</span>
</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">It has been shown <cite class="ltx_cite ltx_citemacro_citep">[Chung et al., <a href="#bib.bib12" title="" class="ltx_ref">2024</a>]</cite> that scaling multi-task instruction tuning dramatically enhances model performance on a wide variety of tasks. Similarly, we attempt to scale supervised fine-tuning data to achieve a strong base model for subsequent alignment. During SFT, we collect and train models on demonstration data of a given prompt<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>A prompt may consist of the most recent user instruction as well as all previous user-model-system interactions.</span></span></span>.
We carefully select and combine both human data and synthetic data to form a high quality mixture that covers various natural language use cases.</p>
</div>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Data selection</h5>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px1.p1.1" class="ltx_p">We establish a series of quality guards of the data before onboarding them for model training, including ratings from in-house human labelers, automatic model-based filtering techniques, and deduplication with text embeddings. We also scale up the mixture size by a variety of synthetic data generation methods, as described in <a href="#S4.SS1.SSS2" title="4.1.2 Synthetic data ‣ 4.1 Data ‣ 4 Post-Training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.1.2</span></a>, and rejection sampling as described in <a href="#S4.SS3.SSS2" title="4.3.2 Iterative teaching committee (iTeC) ‣ 4.3 Reinforcement learning from human feedback (RLHF) ‣ 4 Post-Training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.3.2</span></a>.</p>
</div>
</section>
<section id="S4.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Tuning the mixture ratio</h5>

<div id="S4.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px2.p1.3" class="ltx_p">In order to tune the mixture weight, we treat it as an optimization problem. Specifically, given a set of weights <math id="S4.SS2.SSS0.Px2.p1.1.m1.4" class="ltx_Math" alttext="(w_{1},w_{2},...,w_{n})" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.1.m1.4a"><mrow id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.4.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.4" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.4.cmml">(</mo><msub id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.2" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.2.cmml">w</mi><mn id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.3" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.5" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.4.cmml">,</mo><msub id="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2" xref="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.cmml"><mi id="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.2" xref="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.2.cmml">w</mi><mn id="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.3" xref="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.3.cmml">2</mn></msub><mo id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.6" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.4.cmml">,</mo><mi mathvariant="normal" id="S4.SS2.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.cmml">…</mi><mo id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.7" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.4.cmml">,</mo><msub id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.cmml"><mi id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.2" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.2.cmml">w</mi><mi id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.3" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.3.cmml">n</mi></msub><mo stretchy="false" id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.8" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.4.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.1.m1.4b"><vector id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.4.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3"><apply id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.2">𝑤</ci><cn type="integer" id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.1.1.3">1</cn></apply><apply id="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.2.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.2">𝑤</ci><cn type="integer" id="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.3.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.3.3.2.2.3">2</cn></apply><ci id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1">…</ci><apply id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.2.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.2">𝑤</ci><ci id="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.3.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.4.4.3.3.3">𝑛</ci></apply></vector></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.1.m1.4c">(w_{1},w_{2},...,w_{n})</annotation></semantics></math> where <math id="S4.SS2.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="w_{i}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.2.m2.1a"><msub id="S4.SS2.SSS0.Px2.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.2" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.2.cmml">w</mi><mi id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.3" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.2.m2.1b"><apply id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.2">𝑤</ci><ci id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.2.m2.1c">w_{i}</annotation></semantics></math> represents the ratio of a specific component in the mixture, we train a model with <math id="S4.SS2.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="w_{i}\rightarrow w_{i}\pm\Delta w_{i}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.3.m3.1a"><mrow id="S4.SS2.SSS0.Px2.p1.3.m3.1.1" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.cmml"><msub id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml"><mi id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.2" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.2.cmml">w</mi><mi id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3.cmml">i</mi></msub><mo stretchy="false" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.1" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml">→</mo><mrow id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml"><msub id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.cmml"><mi id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.2" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.2.cmml">w</mi><mi id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.3" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.3.cmml">i</mi></msub><mo id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.1" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.1.cmml">±</mo><mrow id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.cmml"><mi mathvariant="normal" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.2" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.1" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.1.cmml">​</mo><msub id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.cmml"><mi id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.2" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.2.cmml">w</mi><mi id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.3" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.cmml">i</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.3.m3.1b"><apply id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1"><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.1">→</ci><apply id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.2.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.2">𝑤</ci><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3">𝑖</ci></apply><apply id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3"><csymbol cd="latexml" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.1">plus-or-minus</csymbol><apply id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.2.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.2">𝑤</ci><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.3.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.2.3">𝑖</ci></apply><apply id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3"><times id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.1"></times><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.2.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.2">Δ</ci><apply id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.2.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.2">𝑤</ci><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.3.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.3.m3.1c">w_{i}\rightarrow w_{i}\pm\Delta w_{i}</annotation></semantics></math> and evaluate the quality change on a set of benchmarks.
We find that extensively running such experiments can effectively identify the best mixture and remove the least impactful data components.</p>
</div>
</section>
<section id="S4.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Training hyperparameters</h5>

<div id="S4.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px3.p1.3" class="ltx_p">The model is trained with a constant learning rate <math id="S4.SS2.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="5\mathrm{e}{-6}" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.1.m1.1a"><mrow id="S4.SS2.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.cmml"><mrow id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.cmml"><mn id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.2" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.2.cmml">5</mn><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.1" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.3" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.3.cmml">e</mi></mrow><mo id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1.cmml">−</mo><mn id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3.cmml">6</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.1.m1.1b"><apply id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1"><minus id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1"></minus><apply id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2"><times id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.1"></times><cn type="integer" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.2">5</cn><ci id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.3.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.3">e</ci></apply><cn type="integer" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3">6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.1.m1.1c">5\mathrm{e}{-6}</annotation></semantics></math> for AFM-server and <math id="S4.SS2.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="2\mathrm{e}{-5}" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.2.m2.1a"><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml"><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.cmml"><mn id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.3.cmml">e</mi></mrow><mo id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1.cmml">−</mo><mn id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.2.m2.1b"><apply id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1"><minus id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1"></minus><apply id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2"><times id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.1"></times><cn type="integer" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.2">2</cn><ci id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.3">e</ci></apply><cn type="integer" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.2.m2.1c">2\mathrm{e}{-5}</annotation></semantics></math> for AFM-device models, as well as a drop out rate <math id="S4.SS2.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="0.1" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.3.m3.1a"><mn id="S4.SS2.SSS0.Px3.p1.3.m3.1.1" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.cmml">0.1</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.3.m3.1b"><cn type="float" id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1">0.1</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.3.m3.1c">0.1</annotation></semantics></math>. Since the evaluation metrics fluctuate across different checkpoints, we run checkpoint selection based on automatic evaluation benchmarks and best-of-N selection with reward models to test the headroom for RL.</p>
</div>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S4.SS3.2.1.1" class="ltx_text ltx_font_sansserif">4.3</span> </span><span id="S4.SS3.3.2" class="ltx_text ltx_font_sansserif">Reinforcement learning from human feedback (RLHF)</span>
</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We further use reinforcement learning with collected human preference data to improve model performance and quality.
This involves training a robust reward model and applying it in two algorithms of iTeC and MDLOO that we discuss below. We describe more details of our RLHF pipeline in <a href="#A5" title="Appendix E Technical details for RLHF ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix E</span></a>.</p>
</div>
<section id="S4.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S4.SS3.SSS1.2.1.1" class="ltx_text ltx_font_sansserif">4.3.1</span> </span><span id="S4.SS3.SSS1.3.2" class="ltx_text ltx_font_sansserif">Reward modeling</span>
</h4>

<div id="S4.SS3.SSS1.p1" class="ltx_para">
<p id="S4.SS3.SSS1.p1.1" class="ltx_p">We train reward models using the human preference data collected with the method in <a href="#S4.SS1.SSS1" title="4.1.1 Human annotations ‣ 4.1 Data ‣ 4 Post-Training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.1.1</span></a>.
Each human preference data item contains one prompt and two responses along with human labels including:</p>
</div>
<div id="S4.SS3.SSS1.p2" class="ltx_para">
<ul id="S4.I2" class="ltx_itemize">
<li id="S4.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I2.i1.p1" class="ltx_para">
<p id="S4.I2.i1.p1.1" class="ltx_p">The preferred response between the two and the preference level, i.e., whether the preferred response is significantly better, better, slightly better, or negligibly better than the rejected response.</p>
</div>
</li>
<li id="S4.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I2.i2.p1" class="ltx_para">
<p id="S4.I2.i2.p1.1" class="ltx_p">The single-sided grading of each response, measuring the instruction following property, the conciseness, truthfulness, and harmlessness of each of the responses.</p>
</div>
</li>
</ul>
</div>
<div id="S4.SS3.SSS1.p3" class="ltx_para">
<p id="S4.SS3.SSS1.p3.1" class="ltx_p">Our reward model training follows the standard practice of reward modeling in RLHF with two main innovations:</p>
</div>
<div id="S4.SS3.SSS1.p4" class="ltx_para">
<ul id="S4.I3" class="ltx_itemize">
<li id="S4.I3.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I3.i1.p1" class="ltx_para">
<p id="S4.I3.i1.p1.1" class="ltx_p">We design a soft label loss function that takes the level of human preference into account.</p>
</div>
</li>
<li id="S4.I3.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I3.i2.p1" class="ltx_para">
<p id="S4.I3.i2.p1.1" class="ltx_p">We incorporate single-sided gradings as regularization terms in reward modeling.</p>
</div>
</li>
</ul>
</div>
<div id="S4.SS3.SSS1.p5" class="ltx_para">
<p id="S4.SS3.SSS1.p5.1" class="ltx_p">We employ the commonly used Bradley-Terry-Luce (BTL) model <cite class="ltx_cite ltx_citemacro_citep">[Bradley and Terry, <a href="#bib.bib10" title="" class="ltx_ref">1952</a>]</cite> for reward modeling in RLHF. In this model, the probability that a human annotator prefers one response over another is modeled as the sigmoid function of the difference of the rewards. Our <em id="S4.SS3.SSS1.p5.1.1" class="ltx_emph ltx_font_italic">soft label</em> loss function encourages that this probability is high when the preference level is high, e.g., when one response is significantly better than the other, and vice versa. We note that this is different from the <em id="S4.SS3.SSS1.p5.1.2" class="ltx_emph ltx_font_italic">margin</em>-based loss function in Llama 2 <cite class="ltx_cite ltx_citemacro_citep">[Touvron et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>]</cite>, which also leverages the preference level. Empirically, we find that our method works better than the margin-based loss function. Moreover, we also find that using the single-sided gradings as regularization terms can effectively improve the accuracy of the reward model. More details of our reward modeling techniques can be found in <a href="#A5.SS1" title="E.1 Reward modeling ‣ Appendix E Technical details for RLHF ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section E.1</span></a>.</p>
</div>
</section>
<section id="S4.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S4.SS3.SSS2.2.1.1" class="ltx_text ltx_font_sansserif">4.3.2</span> </span><span id="S4.SS3.SSS2.3.2" class="ltx_text ltx_font_sansserif">Iterative teaching committee (iTeC)</span>
</h4>

<div id="S4.SS3.SSS2.p1" class="ltx_para">
<p id="S4.SS3.SSS2.p1.1" class="ltx_p">To fully unlock the ability of our model with multiple rounds of RLHF,
we propose a novel iterative RLHF framework which effectively combines various preference optimization algorithms,
including rejection sampling (RS), Direct Preference Optimization (DPO) <cite class="ltx_cite ltx_citemacro_citep">[Rafailov et al., <a href="#bib.bib43" title="" class="ltx_ref">2024</a>]</cite> and its variants such as IPO <cite class="ltx_cite ltx_citemacro_citep">[Azar et al., <a href="#bib.bib8" title="" class="ltx_ref">2024</a>]</cite>, and online reinforcement learning (RL).
This enables us to bring the benefit of RLHF to AFM models across all sizes and improve their alignment at the same time.</p>
</div>
<section id="S4.SS3.SSS2.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Iterative committee</h5>

<div id="S4.SS3.SSS2.Px1.p1" class="ltx_para">
<p id="S4.SS3.SSS2.Px1.p1.1" class="ltx_p">One of the most important lessons we learned from developing AFM RLHF
is to refresh online human preference data collection using a diverse set of the best performing models.
Specifically, for each batch of human preference data collection,
we set up a collection of latest promising models trained from SFT, RS, DPO/IPO, and RL,
as well as best models from the previous iterations,
which we refer to as “model committee”.
We collect pairwise human preference on responses sampled from the latest model committee.</p>
</div>
<div id="S4.SS3.SSS2.Px1.p2" class="ltx_para">
<p id="S4.SS3.SSS2.Px1.p2.1" class="ltx_p">After acquiring each batch of human preference data,
we refresh our reward model, and further train a new set of models
using the collection of preference optimization algorithms.
We then continue the next round of iterative RLHF data collection with a new model committee.</p>
</div>
</section>
<section id="S4.SS3.SSS2.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Committee distillation</h5>

<div id="S4.SS3.SSS2.Px2.p1" class="ltx_para">
<p id="S4.SS3.SSS2.Px2.p1.1" class="ltx_p">We further run rejection sampling (distillation) from the model committee with the latest reward model as a reranker.
Instead of reranking at global-level, i.e., picking a single best performing model from the committee and using it as a teacher model,
we rerank model responses at the prompt-level.
Specifically, for each prompt,
we sample multiple responses from each model in the committee,
and use the latest reward model to select the best response for each prompt.
This allows us to combine the advantages of models trained by different preference optimization algorithms. For instance,
we find that algorithms that leverage negative examples, e.g., online RLHF, DPO, IPO, to be better in improving reasoning skills such as math, while rejection sampling fine-tuning learns instruction following and writing skills more effectively.</p>
</div>
</section>
<section id="S4.SS3.SSS2.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Scaling up distillation</h5>

<div id="S4.SS3.SSS2.Px3.p1" class="ltx_para">
<p id="S4.SS3.SSS2.Px3.p1.1" class="ltx_p">In order to bring the RLHF improvements to AFM models across all sizes,
we scale up distillation from the model committee.
Different from larger models, where carefully iterating data and model quality matters much more than data quantity,
we find smaller models can achieve tremendous improvement when we scale up the number of prompts for distillation.
Our final AFM-on-device model is trained on more than 1M high quality responses generated from the model committee.</p>
</div>
</section>
</section>
<section id="S4.SS3.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S4.SS3.SSS3.2.1.1" class="ltx_text ltx_font_sansserif">4.3.3</span> </span><span id="S4.SS3.SSS3.3.2" class="ltx_text ltx_font_sansserif">Online RLHF algorithm: MDLOO</span>
</h4>

<div id="S4.SS3.SSS3.p1" class="ltx_para">
<p id="S4.SS3.SSS3.p1.1" class="ltx_p">In this section, we introduce our online reinforcement learning algorithm MDLOO,
where we decode responses during model training and apply RL algorithms to maximize the reward.</p>
</div>
<div id="S4.SS3.SSS3.p2" class="ltx_para">
<p id="S4.SS3.SSS3.p2.6" class="ltx_p">We use the commonly adopted RLHF objective that maximizes the KL-penalized reward function <cite class="ltx_cite ltx_citemacro_citep">[Ouyang et al., <a href="#bib.bib40" title="" class="ltx_ref">2022</a>]</cite>:</p>
<table id="S4.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E1.m1.2" class="ltx_math_unparsed" alttext="\max_{\theta}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(\cdot|x)}\left[r_{\phi}(x,y)-\beta D_{\text{KL}}\left(\pi_{\theta}(\cdot|x)\|\pi_{\text{ref}}(\cdot|x)\right)\right]," display="block"><semantics id="S4.E1.m1.2a"><mrow id="S4.E1.m1.2b"><munder id="S4.E1.m1.2.3"><mi id="S4.E1.m1.2.3.2">max</mi><mi id="S4.E1.m1.2.3.3">θ</mi></munder><msub id="S4.E1.m1.2.4"><mi id="S4.E1.m1.2.4.2">𝔼</mi><mrow id="S4.E1.m1.2.4.3"><mi id="S4.E1.m1.2.4.3.1">x</mi><mo id="S4.E1.m1.2.4.3.2">∼</mo><mi class="ltx_font_mathcaligraphic" id="S4.E1.m1.2.4.3.3">𝒟</mi><mo id="S4.E1.m1.2.4.3.4">,</mo><mi id="S4.E1.m1.2.4.3.5">y</mi><mo id="S4.E1.m1.2.4.3.6">∼</mo><msub id="S4.E1.m1.2.4.3.7"><mi id="S4.E1.m1.2.4.3.7.2">π</mi><mi id="S4.E1.m1.2.4.3.7.3">θ</mi></msub><mrow id="S4.E1.m1.2.4.3.8"><mo stretchy="false" id="S4.E1.m1.2.4.3.8.1">(</mo><mo lspace="0em" rspace="0em" id="S4.E1.m1.2.4.3.8.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="S4.E1.m1.2.4.3.8.3">|</mo><mi id="S4.E1.m1.2.4.3.8.4">x</mi><mo stretchy="false" id="S4.E1.m1.2.4.3.8.5">)</mo></mrow></mrow></msub><mrow id="S4.E1.m1.2.5"><mo id="S4.E1.m1.2.5.1">[</mo><msub id="S4.E1.m1.2.5.2"><mi id="S4.E1.m1.2.5.2.2">r</mi><mi id="S4.E1.m1.2.5.2.3">ϕ</mi></msub><mrow id="S4.E1.m1.2.5.3"><mo stretchy="false" id="S4.E1.m1.2.5.3.1">(</mo><mi id="S4.E1.m1.1.1">x</mi><mo id="S4.E1.m1.2.5.3.2">,</mo><mi id="S4.E1.m1.2.2">y</mi><mo stretchy="false" id="S4.E1.m1.2.5.3.3">)</mo></mrow><mo id="S4.E1.m1.2.5.4">−</mo><mi id="S4.E1.m1.2.5.5">β</mi><msub id="S4.E1.m1.2.5.6"><mi id="S4.E1.m1.2.5.6.2">D</mi><mtext id="S4.E1.m1.2.5.6.3">KL</mtext></msub><mrow id="S4.E1.m1.2.5.7"><mo id="S4.E1.m1.2.5.7.1">(</mo><msub id="S4.E1.m1.2.5.7.2"><mi id="S4.E1.m1.2.5.7.2.2">π</mi><mi id="S4.E1.m1.2.5.7.2.3">θ</mi></msub><mrow id="S4.E1.m1.2.5.7.3"><mo stretchy="false" id="S4.E1.m1.2.5.7.3.1">(</mo><mo lspace="0em" rspace="0em" id="S4.E1.m1.2.5.7.3.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="S4.E1.m1.2.5.7.3.3">|</mo><mi id="S4.E1.m1.2.5.7.3.4">x</mi><mo stretchy="false" id="S4.E1.m1.2.5.7.3.5">)</mo></mrow><mo lspace="0em" rspace="0.167em" id="S4.E1.m1.2.5.7.4">∥</mo><msub id="S4.E1.m1.2.5.7.5"><mi id="S4.E1.m1.2.5.7.5.2">π</mi><mtext id="S4.E1.m1.2.5.7.5.3">ref</mtext></msub><mrow id="S4.E1.m1.2.5.7.6"><mo stretchy="false" id="S4.E1.m1.2.5.7.6.1">(</mo><mo lspace="0em" rspace="0em" id="S4.E1.m1.2.5.7.6.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="S4.E1.m1.2.5.7.6.3">|</mo><mi id="S4.E1.m1.2.5.7.6.4">x</mi><mo stretchy="false" id="S4.E1.m1.2.5.7.6.5">)</mo></mrow><mo id="S4.E1.m1.2.5.7.7">)</mo></mrow><mo id="S4.E1.m1.2.5.8">]</mo></mrow><mo id="S4.E1.m1.2.6">,</mo></mrow><annotation encoding="application/x-tex" id="S4.E1.m1.2c">\max_{\theta}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(\cdot|x)}\left[r_{\phi}(x,y)-\beta D_{\text{KL}}\left(\pi_{\theta}(\cdot|x)\|\pi_{\text{ref}}(\cdot|x)\right)\right],</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S4.SS3.SSS3.p2.5" class="ltx_p">where <math id="S4.SS3.SSS3.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="S4.SS3.SSS3.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS3.SSS3.p2.1.m1.1.1" xref="S4.SS3.SSS3.p2.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS3.p2.1.m1.1b"><ci id="S4.SS3.SSS3.p2.1.m1.1.1.cmml" xref="S4.SS3.SSS3.p2.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS3.p2.1.m1.1c">\mathcal{D}</annotation></semantics></math> is the prompt distribution, <math id="S4.SS3.SSS3.p2.2.m2.1" class="ltx_math_unparsed" alttext="D_{\text{KL}}(\cdot\|\cdot)" display="inline"><semantics id="S4.SS3.SSS3.p2.2.m2.1a"><mrow id="S4.SS3.SSS3.p2.2.m2.1b"><msub id="S4.SS3.SSS3.p2.2.m2.1.1"><mi id="S4.SS3.SSS3.p2.2.m2.1.1.2">D</mi><mtext id="S4.SS3.SSS3.p2.2.m2.1.1.3">KL</mtext></msub><mrow id="S4.SS3.SSS3.p2.2.m2.1.2"><mo stretchy="false" id="S4.SS3.SSS3.p2.2.m2.1.2.1">(</mo><mo lspace="0em" rspace="0em" id="S4.SS3.SSS3.p2.2.m2.1.2.2">⋅</mo><mo lspace="0em" rspace="0em" id="S4.SS3.SSS3.p2.2.m2.1.2.3">∥</mo><mo lspace="0em" rspace="0em" id="S4.SS3.SSS3.p2.2.m2.1.2.4">⋅</mo><mo stretchy="false" id="S4.SS3.SSS3.p2.2.m2.1.2.5">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.SS3.SSS3.p2.2.m2.1c">D_{\text{KL}}(\cdot\|\cdot)</annotation></semantics></math> denotes the Kullback-Leibler divergence between two distributions, and <math id="S4.SS3.SSS3.p2.3.m3.1" class="ltx_Math" alttext="\beta" display="inline"><semantics id="S4.SS3.SSS3.p2.3.m3.1a"><mi id="S4.SS3.SSS3.p2.3.m3.1.1" xref="S4.SS3.SSS3.p2.3.m3.1.1.cmml">β</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS3.p2.3.m3.1b"><ci id="S4.SS3.SSS3.p2.3.m3.1.1.cmml" xref="S4.SS3.SSS3.p2.3.m3.1.1">𝛽</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS3.p2.3.m3.1c">\beta</annotation></semantics></math> is the coefficient that controls the divergence between the behavior policy <math id="S4.SS3.SSS3.p2.4.m4.1" class="ltx_Math" alttext="\pi_{\theta}" display="inline"><semantics id="S4.SS3.SSS3.p2.4.m4.1a"><msub id="S4.SS3.SSS3.p2.4.m4.1.1" xref="S4.SS3.SSS3.p2.4.m4.1.1.cmml"><mi id="S4.SS3.SSS3.p2.4.m4.1.1.2" xref="S4.SS3.SSS3.p2.4.m4.1.1.2.cmml">π</mi><mi id="S4.SS3.SSS3.p2.4.m4.1.1.3" xref="S4.SS3.SSS3.p2.4.m4.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS3.p2.4.m4.1b"><apply id="S4.SS3.SSS3.p2.4.m4.1.1.cmml" xref="S4.SS3.SSS3.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS3.SSS3.p2.4.m4.1.1.1.cmml" xref="S4.SS3.SSS3.p2.4.m4.1.1">subscript</csymbol><ci id="S4.SS3.SSS3.p2.4.m4.1.1.2.cmml" xref="S4.SS3.SSS3.p2.4.m4.1.1.2">𝜋</ci><ci id="S4.SS3.SSS3.p2.4.m4.1.1.3.cmml" xref="S4.SS3.SSS3.p2.4.m4.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS3.p2.4.m4.1c">\pi_{\theta}</annotation></semantics></math> and a reference policy <math id="S4.SS3.SSS3.p2.5.m5.1" class="ltx_Math" alttext="\pi_{\text{ref}}" display="inline"><semantics id="S4.SS3.SSS3.p2.5.m5.1a"><msub id="S4.SS3.SSS3.p2.5.m5.1.1" xref="S4.SS3.SSS3.p2.5.m5.1.1.cmml"><mi id="S4.SS3.SSS3.p2.5.m5.1.1.2" xref="S4.SS3.SSS3.p2.5.m5.1.1.2.cmml">π</mi><mtext id="S4.SS3.SSS3.p2.5.m5.1.1.3" xref="S4.SS3.SSS3.p2.5.m5.1.1.3a.cmml">ref</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS3.p2.5.m5.1b"><apply id="S4.SS3.SSS3.p2.5.m5.1.1.cmml" xref="S4.SS3.SSS3.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS3.SSS3.p2.5.m5.1.1.1.cmml" xref="S4.SS3.SSS3.p2.5.m5.1.1">subscript</csymbol><ci id="S4.SS3.SSS3.p2.5.m5.1.1.2.cmml" xref="S4.SS3.SSS3.p2.5.m5.1.1.2">𝜋</ci><ci id="S4.SS3.SSS3.p2.5.m5.1.1.3a.cmml" xref="S4.SS3.SSS3.p2.5.m5.1.1.3"><mtext mathsize="70%" id="S4.SS3.SSS3.p2.5.m5.1.1.3.cmml" xref="S4.SS3.SSS3.p2.5.m5.1.1.3">ref</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS3.p2.5.m5.1c">\pi_{\text{ref}}</annotation></semantics></math>, which is typically a model trained by SFT. In our RL training, we use the reward function</p>
<table id="A6.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E2.m1.7" class="ltx_Math" alttext="\displaystyle R(x,y)=r_{\phi}(x,y)-\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}," display="inline"><semantics id="S4.E2.m1.7a"><mrow id="S4.E2.m1.7.7.1" xref="S4.E2.m1.7.7.1.1.cmml"><mrow id="S4.E2.m1.7.7.1.1" xref="S4.E2.m1.7.7.1.1.cmml"><mrow id="S4.E2.m1.7.7.1.1.2" xref="S4.E2.m1.7.7.1.1.2.cmml"><mi id="S4.E2.m1.7.7.1.1.2.2" xref="S4.E2.m1.7.7.1.1.2.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="S4.E2.m1.7.7.1.1.2.1" xref="S4.E2.m1.7.7.1.1.2.1.cmml">​</mo><mrow id="S4.E2.m1.7.7.1.1.2.3.2" xref="S4.E2.m1.7.7.1.1.2.3.1.cmml"><mo stretchy="false" id="S4.E2.m1.7.7.1.1.2.3.2.1" xref="S4.E2.m1.7.7.1.1.2.3.1.cmml">(</mo><mi id="S4.E2.m1.3.3" xref="S4.E2.m1.3.3.cmml">x</mi><mo id="S4.E2.m1.7.7.1.1.2.3.2.2" xref="S4.E2.m1.7.7.1.1.2.3.1.cmml">,</mo><mi id="S4.E2.m1.4.4" xref="S4.E2.m1.4.4.cmml">y</mi><mo stretchy="false" id="S4.E2.m1.7.7.1.1.2.3.2.3" xref="S4.E2.m1.7.7.1.1.2.3.1.cmml">)</mo></mrow></mrow><mo id="S4.E2.m1.7.7.1.1.1" xref="S4.E2.m1.7.7.1.1.1.cmml">=</mo><mrow id="S4.E2.m1.7.7.1.1.3" xref="S4.E2.m1.7.7.1.1.3.cmml"><mrow id="S4.E2.m1.7.7.1.1.3.2" xref="S4.E2.m1.7.7.1.1.3.2.cmml"><msub id="S4.E2.m1.7.7.1.1.3.2.2" xref="S4.E2.m1.7.7.1.1.3.2.2.cmml"><mi id="S4.E2.m1.7.7.1.1.3.2.2.2" xref="S4.E2.m1.7.7.1.1.3.2.2.2.cmml">r</mi><mi id="S4.E2.m1.7.7.1.1.3.2.2.3" xref="S4.E2.m1.7.7.1.1.3.2.2.3.cmml">ϕ</mi></msub><mo lspace="0em" rspace="0em" id="S4.E2.m1.7.7.1.1.3.2.1" xref="S4.E2.m1.7.7.1.1.3.2.1.cmml">​</mo><mrow id="S4.E2.m1.7.7.1.1.3.2.3.2" xref="S4.E2.m1.7.7.1.1.3.2.3.1.cmml"><mo stretchy="false" id="S4.E2.m1.7.7.1.1.3.2.3.2.1" xref="S4.E2.m1.7.7.1.1.3.2.3.1.cmml">(</mo><mi id="S4.E2.m1.5.5" xref="S4.E2.m1.5.5.cmml">x</mi><mo id="S4.E2.m1.7.7.1.1.3.2.3.2.2" xref="S4.E2.m1.7.7.1.1.3.2.3.1.cmml">,</mo><mi id="S4.E2.m1.6.6" xref="S4.E2.m1.6.6.cmml">y</mi><mo stretchy="false" id="S4.E2.m1.7.7.1.1.3.2.3.2.3" xref="S4.E2.m1.7.7.1.1.3.2.3.1.cmml">)</mo></mrow></mrow><mo id="S4.E2.m1.7.7.1.1.3.1" xref="S4.E2.m1.7.7.1.1.3.1.cmml">−</mo><mrow id="S4.E2.m1.7.7.1.1.3.3" xref="S4.E2.m1.7.7.1.1.3.3.cmml"><mi id="S4.E2.m1.7.7.1.1.3.3.2" xref="S4.E2.m1.7.7.1.1.3.3.2.cmml">β</mi><mo lspace="0.167em" rspace="0em" id="S4.E2.m1.7.7.1.1.3.3.1" xref="S4.E2.m1.7.7.1.1.3.3.1.cmml">​</mo><mrow id="S4.E2.m1.7.7.1.1.3.3.3" xref="S4.E2.m1.7.7.1.1.3.3.3.cmml"><mi id="S4.E2.m1.7.7.1.1.3.3.3.1" xref="S4.E2.m1.7.7.1.1.3.3.3.1.cmml">log</mi><mo lspace="0.167em" id="S4.E2.m1.7.7.1.1.3.3.3a" xref="S4.E2.m1.7.7.1.1.3.3.3.cmml">⁡</mo><mstyle displaystyle="true" id="S4.E2.m1.2.2" xref="S4.E2.m1.2.2.cmml"><mfrac id="S4.E2.m1.2.2a" xref="S4.E2.m1.2.2.cmml"><mrow id="S4.E2.m1.1.1.1" xref="S4.E2.m1.1.1.1.cmml"><msub id="S4.E2.m1.1.1.1.3" xref="S4.E2.m1.1.1.1.3.cmml"><mi id="S4.E2.m1.1.1.1.3.2" xref="S4.E2.m1.1.1.1.3.2.cmml">π</mi><mi id="S4.E2.m1.1.1.1.3.3" xref="S4.E2.m1.1.1.1.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S4.E2.m1.1.1.1.2" xref="S4.E2.m1.1.1.1.2.cmml">​</mo><mrow id="S4.E2.m1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E2.m1.1.1.1.1.1.2" xref="S4.E2.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E2.m1.1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.2" xref="S4.E2.m1.1.1.1.1.1.1.2.cmml">y</mi><mo fence="false" id="S4.E2.m1.1.1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.1.cmml">|</mo><mi id="S4.E2.m1.1.1.1.1.1.1.3" xref="S4.E2.m1.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S4.E2.m1.1.1.1.1.1.3" xref="S4.E2.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mrow id="S4.E2.m1.2.2.2" xref="S4.E2.m1.2.2.2.cmml"><msub id="S4.E2.m1.2.2.2.3" xref="S4.E2.m1.2.2.2.3.cmml"><mi id="S4.E2.m1.2.2.2.3.2" xref="S4.E2.m1.2.2.2.3.2.cmml">π</mi><mtext id="S4.E2.m1.2.2.2.3.3" xref="S4.E2.m1.2.2.2.3.3a.cmml">ref</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E2.m1.2.2.2.2" xref="S4.E2.m1.2.2.2.2.cmml">​</mo><mrow id="S4.E2.m1.2.2.2.1.1" xref="S4.E2.m1.2.2.2.1.1.1.cmml"><mo stretchy="false" id="S4.E2.m1.2.2.2.1.1.2" xref="S4.E2.m1.2.2.2.1.1.1.cmml">(</mo><mrow id="S4.E2.m1.2.2.2.1.1.1" xref="S4.E2.m1.2.2.2.1.1.1.cmml"><mi id="S4.E2.m1.2.2.2.1.1.1.2" xref="S4.E2.m1.2.2.2.1.1.1.2.cmml">y</mi><mo fence="false" id="S4.E2.m1.2.2.2.1.1.1.1" xref="S4.E2.m1.2.2.2.1.1.1.1.cmml">|</mo><mi id="S4.E2.m1.2.2.2.1.1.1.3" xref="S4.E2.m1.2.2.2.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S4.E2.m1.2.2.2.1.1.3" xref="S4.E2.m1.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mfrac></mstyle></mrow></mrow></mrow></mrow><mo id="S4.E2.m1.7.7.1.2" xref="S4.E2.m1.7.7.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.E2.m1.7b"><apply id="S4.E2.m1.7.7.1.1.cmml" xref="S4.E2.m1.7.7.1"><eq id="S4.E2.m1.7.7.1.1.1.cmml" xref="S4.E2.m1.7.7.1.1.1"></eq><apply id="S4.E2.m1.7.7.1.1.2.cmml" xref="S4.E2.m1.7.7.1.1.2"><times id="S4.E2.m1.7.7.1.1.2.1.cmml" xref="S4.E2.m1.7.7.1.1.2.1"></times><ci id="S4.E2.m1.7.7.1.1.2.2.cmml" xref="S4.E2.m1.7.7.1.1.2.2">𝑅</ci><interval closure="open" id="S4.E2.m1.7.7.1.1.2.3.1.cmml" xref="S4.E2.m1.7.7.1.1.2.3.2"><ci id="S4.E2.m1.3.3.cmml" xref="S4.E2.m1.3.3">𝑥</ci><ci id="S4.E2.m1.4.4.cmml" xref="S4.E2.m1.4.4">𝑦</ci></interval></apply><apply id="S4.E2.m1.7.7.1.1.3.cmml" xref="S4.E2.m1.7.7.1.1.3"><minus id="S4.E2.m1.7.7.1.1.3.1.cmml" xref="S4.E2.m1.7.7.1.1.3.1"></minus><apply id="S4.E2.m1.7.7.1.1.3.2.cmml" xref="S4.E2.m1.7.7.1.1.3.2"><times id="S4.E2.m1.7.7.1.1.3.2.1.cmml" xref="S4.E2.m1.7.7.1.1.3.2.1"></times><apply id="S4.E2.m1.7.7.1.1.3.2.2.cmml" xref="S4.E2.m1.7.7.1.1.3.2.2"><csymbol cd="ambiguous" id="S4.E2.m1.7.7.1.1.3.2.2.1.cmml" xref="S4.E2.m1.7.7.1.1.3.2.2">subscript</csymbol><ci id="S4.E2.m1.7.7.1.1.3.2.2.2.cmml" xref="S4.E2.m1.7.7.1.1.3.2.2.2">𝑟</ci><ci id="S4.E2.m1.7.7.1.1.3.2.2.3.cmml" xref="S4.E2.m1.7.7.1.1.3.2.2.3">italic-ϕ</ci></apply><interval closure="open" id="S4.E2.m1.7.7.1.1.3.2.3.1.cmml" xref="S4.E2.m1.7.7.1.1.3.2.3.2"><ci id="S4.E2.m1.5.5.cmml" xref="S4.E2.m1.5.5">𝑥</ci><ci id="S4.E2.m1.6.6.cmml" xref="S4.E2.m1.6.6">𝑦</ci></interval></apply><apply id="S4.E2.m1.7.7.1.1.3.3.cmml" xref="S4.E2.m1.7.7.1.1.3.3"><times id="S4.E2.m1.7.7.1.1.3.3.1.cmml" xref="S4.E2.m1.7.7.1.1.3.3.1"></times><ci id="S4.E2.m1.7.7.1.1.3.3.2.cmml" xref="S4.E2.m1.7.7.1.1.3.3.2">𝛽</ci><apply id="S4.E2.m1.7.7.1.1.3.3.3.cmml" xref="S4.E2.m1.7.7.1.1.3.3.3"><log id="S4.E2.m1.7.7.1.1.3.3.3.1.cmml" xref="S4.E2.m1.7.7.1.1.3.3.3.1"></log><apply id="S4.E2.m1.2.2.cmml" xref="S4.E2.m1.2.2"><divide id="S4.E2.m1.2.2.3.cmml" xref="S4.E2.m1.2.2"></divide><apply id="S4.E2.m1.1.1.1.cmml" xref="S4.E2.m1.1.1.1"><times id="S4.E2.m1.1.1.1.2.cmml" xref="S4.E2.m1.1.1.1.2"></times><apply id="S4.E2.m1.1.1.1.3.cmml" xref="S4.E2.m1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.3.1.cmml" xref="S4.E2.m1.1.1.1.3">subscript</csymbol><ci id="S4.E2.m1.1.1.1.3.2.cmml" xref="S4.E2.m1.1.1.1.3.2">𝜋</ci><ci id="S4.E2.m1.1.1.1.3.3.cmml" xref="S4.E2.m1.1.1.1.3.3">𝜃</ci></apply><apply id="S4.E2.m1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1"><csymbol cd="latexml" id="S4.E2.m1.1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1">conditional</csymbol><ci id="S4.E2.m1.1.1.1.1.1.1.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.2">𝑦</ci><ci id="S4.E2.m1.1.1.1.1.1.1.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.3">𝑥</ci></apply></apply><apply id="S4.E2.m1.2.2.2.cmml" xref="S4.E2.m1.2.2.2"><times id="S4.E2.m1.2.2.2.2.cmml" xref="S4.E2.m1.2.2.2.2"></times><apply id="S4.E2.m1.2.2.2.3.cmml" xref="S4.E2.m1.2.2.2.3"><csymbol cd="ambiguous" id="S4.E2.m1.2.2.2.3.1.cmml" xref="S4.E2.m1.2.2.2.3">subscript</csymbol><ci id="S4.E2.m1.2.2.2.3.2.cmml" xref="S4.E2.m1.2.2.2.3.2">𝜋</ci><ci id="S4.E2.m1.2.2.2.3.3a.cmml" xref="S4.E2.m1.2.2.2.3.3"><mtext mathsize="70%" id="S4.E2.m1.2.2.2.3.3.cmml" xref="S4.E2.m1.2.2.2.3.3">ref</mtext></ci></apply><apply id="S4.E2.m1.2.2.2.1.1.1.cmml" xref="S4.E2.m1.2.2.2.1.1"><csymbol cd="latexml" id="S4.E2.m1.2.2.2.1.1.1.1.cmml" xref="S4.E2.m1.2.2.2.1.1.1.1">conditional</csymbol><ci id="S4.E2.m1.2.2.2.1.1.1.2.cmml" xref="S4.E2.m1.2.2.2.1.1.1.2">𝑦</ci><ci id="S4.E2.m1.2.2.2.1.1.1.3.cmml" xref="S4.E2.m1.2.2.2.1.1.1.3">𝑥</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E2.m1.7c">\displaystyle R(x,y)=r_{\phi}(x,y)-\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S4.SS3.SSS3.p2.7" class="ltx_p">whose expectation is equivalent to <a href="#S4.E1" title="1 ‣ 4.3.3 Online RLHF algorithm: MDLOO ‣ 4.3 Reinforcement learning from human feedback (RLHF) ‣ 4 Post-Training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Equation 1</span></a>. We consider the bandit setting where the generation of the entire response is considered as one action, and we do not use the value network (a.k.a. the critic) to obtain the per-token reward or advantage.</p>
</div>
<div id="S4.SS3.SSS3.p3" class="ltx_para">
<p id="S4.SS3.SSS3.p3.1" class="ltx_p">Similar to commonly used RLHF algorithms such as PPO <cite class="ltx_cite ltx_citemacro_citep">[Schulman et al., <a href="#bib.bib45" title="" class="ltx_ref">2017</a>]</cite>, we use a trust-region based policy iteration algorithm.
We made two main design choices in our online RL algorithm:</p>
</div>
<div id="S4.SS3.SSS3.p4" class="ltx_para">
<ul id="S4.I4" class="ltx_itemize">
<li id="S4.I4.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I4.i1.p1" class="ltx_para">
<p id="S4.I4.i1.p1.1" class="ltx_p">We use the Leave-One-Out (LOO) estimator to estimate the advantage of a prompt-response pair, similar to a recent work <cite class="ltx_cite ltx_citemacro_citep">[Ahmadian et al., <a href="#bib.bib3" title="" class="ltx_ref">2024</a>]</cite>.</p>
</div>
</li>
<li id="S4.I4.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I4.i2.p1" class="ltx_para">
<p id="S4.I4.i2.p1.1" class="ltx_p">We use Mirror Descent Policy Optimization (MDPO) <cite class="ltx_cite ltx_citemacro_citep">[Tomar et al., <a href="#bib.bib50" title="" class="ltx_ref">2020</a>]</cite> to optimize the policy, differently from the more commonly used clipping-based PPO method.</p>
</div>
</li>
</ul>
</div>
<div id="S4.SS3.SSS3.p5" class="ltx_para">
<p id="S4.SS3.SSS3.p5.1" class="ltx_p">Thus, we name our online RL algorithm <em id="S4.SS3.SSS3.p5.1.1" class="ltx_emph ltx_font_italic">Mirror Descent with Leave-One-Out estimation</em> (MDLOO). More specifically, during the decoding stage of the algorithm, we decode multiple responses for each prompt, and assign the <em id="S4.SS3.SSS3.p5.1.2" class="ltx_emph ltx_font_italic">advantage</em> of each response to be the difference of the reward of the (prompt, response) pair and the mean reward of the other responses generated by the same prompt. Intuitively, this estimator aims to measure how much better a response is compared to a typical response. Empirically, we find this advantage estimator crucial for stabilizing the RL algorithm and achieving strong results.
Moreover, we use a KL-regularization-based trust region method, i.e. MDPO, to control the policy change in each iteration. We find that this algorithm is more effective than PPO in our setting. More details of our online RLHF algorithm can be found in <a href="#A5.SS2" title="E.2 Online RL algorithm ‣ Appendix E Technical details for RLHF ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section E.2</span></a>.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S5.2.1.1" class="ltx_text ltx_font_sansserif">5</span> </span><span id="S5.3.2" class="ltx_text ltx_font_sansserif">Powering Apple Intelligence features</span>
</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">Our foundation models are designed for Apple Intelligence, the personal intelligence system integrated into supported models of iPhone, iPad, and Mac.
We have built these models to be fast and efficient. And while we have achieved impressive levels of broad capability in our base model, the actual relevant measure of its quality is how it performs on specific tasks across our operating systems.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">Here we have found that we can elevate the performance of even small models to best-in-class performance through task-specific fine-tuning and have developed an architecture, based on runtime-swappable adapters, to enable the single foundation model to be specialized for dozens of such tasks. A high-level overview is presented in <a href="#S5.F2" title="Figure 2 ‣ 5 Powering Apple Intelligence features ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 2</span></a>.</p>
</div>
<figure id="S5.F2" class="ltx_figure"><img src="/html/2407.21075/assets/x2.png" id="S5.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="553" height="325" alt="Architecture Diagram of Apple Intelligence.">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Architecture of Apple Intelligence with adapters for the language on-device and server models and the image models. In this report we are only describing the text models.</figcaption>
</figure>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S5.SS1.2.1.1" class="ltx_text ltx_font_sansserif">5.1</span> </span><span id="S5.SS1.3.2" class="ltx_text ltx_font_sansserif">Adapter architecture</span>
</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">Our foundation models are fine-tuned for users’ everyday activities, and can dynamically specialize themselves on-the-fly for the task at hand. We use LoRA <cite class="ltx_cite ltx_citemacro_citep">[Hu et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>]</cite> adapters, small neural network modules that can be plugged into various layers of the base model, to fine-tune our models for specific tasks.
For each task, we adapt all of AFM’s linear projection matrices in the self-attention layers and the fully connected layers in the pointwise feedforward networks.
By fine-tuning only the adapters, the original parameters of the base pre-trained model remain unchanged, preserving the general knowledge of the model while tailoring the adapters to support specific tasks.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">We represent the values of the adapter parameters using 16 bits, and for the <math id="S5.SS1.p2.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S5.SS1.p2.1.m1.1a"><mo id="S5.SS1.p2.1.m1.1.1" xref="S5.SS1.p2.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.1.m1.1b"><csymbol cd="latexml" id="S5.SS1.p2.1.m1.1.1.cmml" xref="S5.SS1.p2.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.1.m1.1c">\sim</annotation></semantics></math>3 billion parameter on-device model, the parameters for a rank 16 adapter typically require 10s of megabytes. The adapter models can be dynamically loaded, temporarily cached in memory, and swapped—giving our foundation model the ability to specialize itself on the fly for the task at hand while efficiently managing memory and guaranteeing the operating system’s responsiveness.</p>
</div>
<div id="S5.SS1.p3" class="ltx_para">
<p id="S5.SS1.p3.1" class="ltx_p">To facilitate the training of the adapters, we created an efficient infrastructure that allows us to rapidly add, retrain, test, and deploy adapters when the base model or the training data gets updated or new capabilities are required. It is worth noting that the adapter parameters are initialized using the accuracy-recovery adapter introduced in <a href="#S5.SS2" title="5.2 Optimizations ‣ 5 Powering Apple Intelligence features ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 5.2</span></a>.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S5.SS2.2.1.1" class="ltx_text ltx_font_sansserif">5.2</span> </span><span id="S5.SS2.3.2" class="ltx_text ltx_font_sansserif">Optimizations</span>
</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">The AFM models are designed to support our users throughout their daily activities, and both inference latency and power efficiency are important for the overall user experience.
We apply various optimization techniques to allow AFM to be efficiently deployed on-device and in Private Cloud Compute. These techniques significantly reduce memory, latency, and power usage while maintaining the overall model quality.</p>
</div>
<section id="S5.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="S5.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px1.p1.1" class="ltx_p">In order to fit AFM into a constrained memory budget of edge devices and reduce inference cost, it is critical to apply model quantization techniques to reduce the effective bits per weight while maintaining the model quality. Previous works have found that 4-bit quantized models only have marginal loss of quality (typically measured in pre-training metrics) compared to the original 32/16-bit float-point versions. Since AFM is expected to support a diverse set of product features, it is essential that the quantized model retains capabilities in specific domains critical to these use cases. To achieve an optimal trade-off between model capacity and inference performance, we have developed state-of-the-art quantization methods and a framework that utilizes accuracy-recovery adapters. This allows us to achieve near-lossless quantization that is on average less than 4 bit-per-weight, and provides flexible quantization scheme choices.</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Methods</h5>

<div id="S5.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px2.p1.1" class="ltx_p">The model is compressed and quantized, on average under 4-bit-per-weight, after the post-training stages (details of the quantization scheme will be discussed later). The quantized model often shows a moderate level of quality loss. Therefore, instead of directly passing the quantized model to application teams for feature development, we attach a set of parameter-efficient LoRA adapters for quality recovery. We make sure that these LoRA adapters training recipes are consistent with pre-training and post-training processes. Then, products will fine-tune their own feature-specific LoRA adapters by initializing the adapter weights from the accuracy-recovery adapters, while keeping the quantized base model frozen.</p>
</div>
<div id="S5.SS2.SSS0.Px2.p2" class="ltx_para">
<p id="S5.SS2.SSS0.Px2.p2.2" class="ltx_p">It is noteworthy that training accuracy-recovery adapters is sample-efficient and can be considered as a mini-version of training the base model. During the pre-training stage of the adapters, we only require approximately 10 billion tokens (<math id="S5.SS2.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="\sim 0.15\%" display="inline"><semantics id="S5.SS2.SSS0.Px2.p2.1.m1.1a"><mrow id="S5.SS2.SSS0.Px2.p2.1.m1.1.1" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.cmml"><mi id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.2" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.2.cmml"></mi><mo id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.1" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.1.cmml">∼</mo><mrow id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.cmml"><mn id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.2" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.2.cmml">0.15</mn><mo id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.1" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p2.1.m1.1b"><apply id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1"><csymbol cd="latexml" id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.1.cmml" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.1">similar-to</csymbol><csymbol cd="latexml" id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.2.cmml" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.2">absent</csymbol><apply id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.cmml" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3"><csymbol cd="latexml" id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.1.cmml" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.1">percent</csymbol><cn type="float" id="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.2.cmml" xref="S5.SS2.SSS0.Px2.p2.1.m1.1.1.3.2">0.15</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p2.1.m1.1c">\sim 0.15\%</annotation></semantics></math> of base model training) to fully recover the capacity for the quantized model. Since application adapters will fine tune from these accuracy-recovery adapters, they do not incur any additional memory usage or inference costs. Regarding adapter size, we found that adapter rank of 16 offers the optimal tradeoff between model capacity and inference performance. However, to provide flexibility for various use cases, we provide a suite of accuracy-recovery adapters in different ranks <math id="S5.SS2.SSS0.Px2.p2.2.m2.3" class="ltx_Math" alttext="\{8,16,32\}" display="inline"><semantics id="S5.SS2.SSS0.Px2.p2.2.m2.3a"><mrow id="S5.SS2.SSS0.Px2.p2.2.m2.3.4.2" xref="S5.SS2.SSS0.Px2.p2.2.m2.3.4.1.cmml"><mo stretchy="false" id="S5.SS2.SSS0.Px2.p2.2.m2.3.4.2.1" xref="S5.SS2.SSS0.Px2.p2.2.m2.3.4.1.cmml">{</mo><mn id="S5.SS2.SSS0.Px2.p2.2.m2.1.1" xref="S5.SS2.SSS0.Px2.p2.2.m2.1.1.cmml">8</mn><mo id="S5.SS2.SSS0.Px2.p2.2.m2.3.4.2.2" xref="S5.SS2.SSS0.Px2.p2.2.m2.3.4.1.cmml">,</mo><mn id="S5.SS2.SSS0.Px2.p2.2.m2.2.2" xref="S5.SS2.SSS0.Px2.p2.2.m2.2.2.cmml">16</mn><mo id="S5.SS2.SSS0.Px2.p2.2.m2.3.4.2.3" xref="S5.SS2.SSS0.Px2.p2.2.m2.3.4.1.cmml">,</mo><mn id="S5.SS2.SSS0.Px2.p2.2.m2.3.3" xref="S5.SS2.SSS0.Px2.p2.2.m2.3.3.cmml">32</mn><mo stretchy="false" id="S5.SS2.SSS0.Px2.p2.2.m2.3.4.2.4" xref="S5.SS2.SSS0.Px2.p2.2.m2.3.4.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p2.2.m2.3b"><set id="S5.SS2.SSS0.Px2.p2.2.m2.3.4.1.cmml" xref="S5.SS2.SSS0.Px2.p2.2.m2.3.4.2"><cn type="integer" id="S5.SS2.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S5.SS2.SSS0.Px2.p2.2.m2.1.1">8</cn><cn type="integer" id="S5.SS2.SSS0.Px2.p2.2.m2.2.2.cmml" xref="S5.SS2.SSS0.Px2.p2.2.m2.2.2">16</cn><cn type="integer" id="S5.SS2.SSS0.Px2.p2.2.m2.3.3.cmml" xref="S5.SS2.SSS0.Px2.p2.2.m2.3.3">32</cn></set></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p2.2.m2.3c">\{8,16,32\}</annotation></semantics></math> for application teams to select from. In <a href="#A6" title="Appendix F Accuracy-recovery adapters ablation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Appendix F</span></a>, we provide detailed evaluation results across unquantized, quantized, and accuracy-recovered models and show that the recovered models perform much closer to the unquantized version.</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Quantization schemes</h5>

<div id="S5.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px3.p1.1" class="ltx_p">Another benefit brought by accuracy-recovery adapters is that they allow more flexible choices of quantization schemes. Previously when quantizing LLMs, people typically group the weights into small blocks, normalize each block by the corresponding maximal absolute values to filter out outliers, then apply quantization algorithms in a block basis. While a larger block size yields lower effective bits per weight and a higher throughput, the quantization loss would increase. In order to balance this tradeoff, it is common to set block size to a small value, like 64 or 32. In our experiments, we found that accuracy-recovery adapters can greatly improve the pareto frontier in the tradeoff. More errors will be recovered for more aggressive quantization schemes. As a result, we are able to use a highly-efficient quantization scheme for AFM without worrying about losing model capacity. Specifically, our AFM-on-device model running on Apple Neural Engine (ANE) uses palettization: for projection weights, every 16 columns/rows share the same quantization constants (i.e., lookup tables) and are quantized using K-means with 16 unique values (4-bit). The quantization block size can be up to 100k. Besides, since AFM’s embedding layer is shared between the input and output, it is implemented differently from projection layers on ANE. Hence, we quantize the embedding using per-channel quantization with 8-bit integers for better efficiency.</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Mixed-precision quantization</h5>

<div id="S5.SS2.SSS0.Px4.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px4.p1.1" class="ltx_p">Residual connections exist in every transformer block and every layer in AFM. So it is unlikely that all layers have the equal importance. Following this intuition, we further reduce the memory usage by pushing some layers to use 2-bit quantization (default is 4-bit). On average, AFM-on-device can be compressed to only about 3.5 bits per weight (bpw) without significant quality loss. We choose to use 3.7 bpw in production as it already meets the memory requirements.</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px5" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Interactive model analysis</h5>

<div id="S5.SS2.SSS0.Px5.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px5.p1.1" class="ltx_p">We use an interactive model latency and power analysis tool, Talaria <cite class="ltx_cite ltx_citemacro_citep">[Hohman et al., <a href="#bib.bib21" title="" class="ltx_ref">2024</a>]</cite>, to better guide the bit rate selection for each operation.</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px6" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">More discussions</h5>

<div id="S5.SS2.SSS0.Px6.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px6.p1.1" class="ltx_p">The usage of quantized model and LoRA adapters look conceptually similar to QLoRA <cite class="ltx_cite ltx_citemacro_citep">[Dettmers et al., <a href="#bib.bib14" title="" class="ltx_ref">2024</a>]</cite>.
While QLoRA was designed to save computational resources during fine-tuning, our focus is on the ability to switch between different LoRA adapters to efficiently support high performance across various specific use cases.
Before feature-specific finetuning, we first train accuracy-recovery adapters on the same pretraining and post-training data, which is critical to preserve the model quality.
The accuracy-recovery framework can be combined with different quantization techniques, like GPTQ <cite class="ltx_cite ltx_citemacro_citep">[Frantar et al., <a href="#bib.bib16" title="" class="ltx_ref">2022</a>]</cite> and AWQ <cite class="ltx_cite ltx_citemacro_citep">[Lin et al., <a href="#bib.bib35" title="" class="ltx_ref">2024</a>]</cite>, since it does not depend directly on the quantization method itself.
The feature adapters described in <a href="#S5" title="5 Powering Apple Intelligence features ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 5</span></a> are initialized from these accuracy-recovery adapters.</p>
</div>
</section>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S5.SS3.2.1.1" class="ltx_text ltx_font_sansserif">5.3</span> </span><span id="S5.SS3.3.2" class="ltx_text ltx_font_sansserif">Case study: summarization</span>
</h3>

<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">We use the AFM-on-device model to power summarization features. We worked with our design teams to create specifications for summaries of Emails, Messages, and Notifications.</p>
</div>
<section id="S5.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="S5.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS3.SSS0.Px1.p1.1" class="ltx_p">While AFM-on-device is good at general summarization, we find it difficult to elicit summaries that strictly conform to the specification. Therefore, we fine tune a LoRA adapter on top of the quantized AFM-on-device for summarization. The adapter is initialized from the accuracy-recovery adapter as described in <a href="#S5.SS2" title="5.2 Optimizations ‣ 5 Powering Apple Intelligence features ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 5.2</span></a>.
We use a data mixture consisting of input payloads covering Emails, Messages, and Notifications. These payloads include public dataset, vendor data, and internally generated and submitted examples. All the data have been approved to use for production. Vendor data and internally generated data have been anonymized to remove the user information.
Given these payloads, we generated synthetic summaries using AFM-server according to product’s requirements. These payloads and summaries are used for training.</p>
</div>
</section>
<section id="S5.SS3.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Synthetic summaries</h5>

<div id="S5.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS3.SSS0.Px2.p1.1" class="ltx_p">We use AFM-server to generate synthetic summaries. We apply a series of rule-based filters followed by model based filters. Rule-based filters are based on heuristics such as length constraints, formatting constraints, points of view, voice, etc. Model-based filters are used to screen more challenging problems such as entailment.
Our synthetic data pipeline allows us to efficiently generate a large amount of training data and filter it out by an order of magnitude to retain high-quality examples for fine tuning.</p>
</div>
</section>
<section id="S5.SS3.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Prompt injection</h5>

<div id="S5.SS3.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS3.SSS0.Px3.p1.1" class="ltx_p">We find that AFM-on-device is prone to following instructions or answering questions that are present in the input content instead of summarizing it. To mitigate this issue, we identify a large set of examples with such content using heuristics, use AFM-server to generate summaries, as it does not exhibit similar behavior, and add this synthetic dataset to the fine tuning data mixture.</p>
</div>
</section>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S6.2.1.1" class="ltx_text ltx_font_sansserif">6</span> </span><span id="S6.3.2" class="ltx_text ltx_font_sansserif">Evaluation</span>
</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">We evaluate the AFM models on pre-training (<a href="#S6.SS1" title="6.1 Pre-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 6.1</span></a>), post-training (<a href="#S6.SS2" title="6.2 Post-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 6.2</span></a>), and most importantly, feature-specific (<a href="#S6.SS3" title="6.3 Summarization feature evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 6.3</span></a>) benchmarks.</p>
</div>
<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S6.SS1.2.1.1" class="ltx_text ltx_font_sansserif">6.1</span> </span><span id="S6.SS1.3.2" class="ltx_text ltx_font_sansserif">Pre-training evaluation</span>
</h3>

<div id="S6.SS1.p1" class="ltx_para">
<p id="S6.SS1.p1.1" class="ltx_p">In this section we present common few-shot pre-training evaluation metrics.
While these benchmarks are useful for tracking our progress on pre-training, we found that human evaluations on the post-trained models (<a href="#S6.SS2" title="6.2 Post-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 6.2</span></a>) and feature adapters (<a href="#S6.SS3" title="6.3 Summarization feature evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 6.3</span></a>) are more closely correlated to end-to-end user experience.</p>
</div>
<section id="S6.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph"></h5>

<div id="S6.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S6.SS1.SSS0.Px1.p1.1" class="ltx_p">We evaluate AFM pre-trained models with common open-sourced evaluation harnesses and benchmarks.
<a href="#S6.T2" title="Table 2 ‣ 6.1 Pre-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 2</span></a> presents the results of AFM-on-device and AFM-server on HELM MMLU v1.5.0 <cite class="ltx_cite ltx_citemacro_citep">[Liang et al., <a href="#bib.bib34" title="" class="ltx_ref">2023</a>]</cite>, which tests 5-shot multiple-choice question answering across 57 subjects.
Also see <a href="#S6.T3" title="Table 3 ‣ 6.1 Pre-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 3</span></a> and <a href="#S6.T4" title="Table 4 ‣ 6.1 Pre-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 4</span></a> for the results of AFM-server on a subset of the HuggingFace OpenLLM leaderboard V1 <cite class="ltx_cite ltx_citemacro_citep">[Huggingface, <a href="#bib.bib24" title="" class="ltx_ref">2024</a>]</cite> and the HELM-Lite v1.5.0 benchmark suite <cite class="ltx_cite ltx_citemacro_citep">[Stanford, <a href="#bib.bib47" title="" class="ltx_ref">2024</a>]</cite>, respectively.
These benchmarks show that the AFM pretrained models possess strong language and reasoning capabilities and provide a solid foundation for post-training and feature fine-tuning.</p>
</div>
<figure id="S6.T2" class="ltx_table">
<table id="S6.T2.1" class="ltx_tabular ltx_align_middle">
<tr id="S6.T2.1.1" class="ltx_tr">
<td id="S6.T2.1.1.1" class="ltx_td ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"></td>
<td id="S6.T2.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T2.1.1.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">AFM-on-device</span></td>
<td id="S6.T2.1.1.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T2.1.1.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">AFM-server</span></td>
</tr>
<tr id="S6.T2.1.2" class="ltx_tr">
<td id="S6.T2.1.2.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">MMLU (5 shot)</span></td>
<td id="S6.T2.1.2.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T2.1.2.2.1" class="ltx_text" style="font-size:90%;">61.4</span></td>
<td id="S6.T2.1.2.3" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T2.1.2.3.1" class="ltx_text" style="font-size:90%;">75.4</span></td>
</tr>
</table>
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 2: </span>HELM MMLU-5s <cite class="ltx_cite ltx_citemacro_citep">[Liang et al., <a href="#bib.bib34" title="" class="ltx_ref">2023</a>]</cite> v1.5.0 evaluation results.</figcaption>
</figure>
<figure id="S6.T3" class="ltx_table">
<table id="S6.T3.1" class="ltx_tabular ltx_align_middle">
<tr id="S6.T3.1.1" class="ltx_tr">
<td id="S6.T3.1.1.1" class="ltx_td ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"></td>
<td id="S6.T3.1.1.2" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.1.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">AFM-server</span></td>
</tr>
<tr id="S6.T3.1.2" class="ltx_tr">
<td id="S6.T3.1.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">MMLU (5-shot)</span></td>
<td id="S6.T3.1.2.2" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.2.2.1" class="ltx_text" style="font-size:90%;">75.3</span></td>
</tr>
<tr id="S6.T3.1.3" class="ltx_tr">
<td id="S6.T3.1.3.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.3.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">GSM8K (5-shot)</span></td>
<td id="S6.T3.1.3.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.3.2.1" class="ltx_text" style="font-size:90%;">72.4</span></td>
</tr>
<tr id="S6.T3.1.4" class="ltx_tr">
<td id="S6.T3.1.4.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.4.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">ARC-c (25-shot)</span></td>
<td id="S6.T3.1.4.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.4.2.1" class="ltx_text" style="font-size:90%;">69.7</span></td>
</tr>
<tr id="S6.T3.1.5" class="ltx_tr">
<td id="S6.T3.1.5.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.5.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">HellaSwag (10-shot)</span></td>
<td id="S6.T3.1.5.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.5.2.1" class="ltx_text" style="font-size:90%;">86.9</span></td>
</tr>
<tr id="S6.T3.1.6" class="ltx_tr">
<td id="S6.T3.1.6.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.6.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Winogrande (5-shot)</span></td>
<td id="S6.T3.1.6.2" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T3.1.6.2.1" class="ltx_text" style="font-size:90%;">79.2</span></td>
</tr>
</table>
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 3: </span>A subset of Open LLM Leaderboard <cite class="ltx_cite ltx_citemacro_citep">[Huggingface, <a href="#bib.bib24" title="" class="ltx_ref">2024</a>]</cite> V1 evaluation results.</figcaption>
</figure>
<figure id="S6.T4" class="ltx_table">
<table id="S6.T4.1" class="ltx_tabular ltx_align_middle">
<tr id="S6.T4.1.1" class="ltx_tr">
<td id="S6.T4.1.1.1" class="ltx_td ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"></td>
<td id="S6.T4.1.1.2" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.1.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">AFM-server</span></td>
</tr>
<tr id="S6.T4.1.2" class="ltx_tr">
<td id="S6.T4.1.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Narrative QA</span></td>
<td id="S6.T4.1.2.2" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.2.2.1" class="ltx_text" style="font-size:90%;">77.5</span></td>
</tr>
<tr id="S6.T4.1.3" class="ltx_tr">
<td id="S6.T4.1.3.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.3.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Natural Questions (open)</span></td>
<td id="S6.T4.1.3.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.3.2.1" class="ltx_text" style="font-size:90%;">73.8</span></td>
</tr>
<tr id="S6.T4.1.4" class="ltx_tr">
<td id="S6.T4.1.4.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.4.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Natural Questions (closed)</span></td>
<td id="S6.T4.1.4.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.4.2.1" class="ltx_text" style="font-size:90%;">43.1</span></td>
</tr>
<tr id="S6.T4.1.5" class="ltx_tr">
<td id="S6.T4.1.5.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.5.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Openbook QA</span></td>
<td id="S6.T4.1.5.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.5.2.1" class="ltx_text" style="font-size:90%;">89.6</span></td>
</tr>
<tr id="S6.T4.1.6" class="ltx_tr">
<td id="S6.T4.1.6.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.6.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">MMLU</span></td>
<td id="S6.T4.1.6.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.6.2.1" class="ltx_text" style="font-size:90%;">67.2</span></td>
</tr>
<tr id="S6.T4.1.7" class="ltx_tr">
<td id="S6.T4.1.7.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.7.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">MATH-CoT</span></td>
<td id="S6.T4.1.7.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.7.2.1" class="ltx_text" style="font-size:90%;">55.4</span></td>
</tr>
<tr id="S6.T4.1.8" class="ltx_tr">
<td id="S6.T4.1.8.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.8.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">GSM8K</span></td>
<td id="S6.T4.1.8.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.8.2.1" class="ltx_text" style="font-size:90%;">72.3</span></td>
</tr>
<tr id="S6.T4.1.9" class="ltx_tr">
<td id="S6.T4.1.9.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.9.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">LegalBench</span></td>
<td id="S6.T4.1.9.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.9.2.1" class="ltx_text" style="font-size:90%;">67.9</span></td>
</tr>
<tr id="S6.T4.1.10" class="ltx_tr">
<td id="S6.T4.1.10.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.10.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">MedQA</span></td>
<td id="S6.T4.1.10.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.10.2.1" class="ltx_text" style="font-size:90%;">64.4</span></td>
</tr>
<tr id="S6.T4.1.11" class="ltx_tr">
<td id="S6.T4.1.11.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.11.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">WMT 2014</span></td>
<td id="S6.T4.1.11.2" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="S6.T4.1.11.2.1" class="ltx_text" style="font-size:90%;">18.6</span></td>
</tr>
</table>
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 4: </span>HELM-Lite v1.5.0 <cite class="ltx_cite ltx_citemacro_citep">[Stanford, <a href="#bib.bib47" title="" class="ltx_ref">2024</a>]</cite> pre-training evaluation results. N.B. Many benchmarks (e.g. MMLU) differ significantly from commonly used settings.</figcaption>
</figure>
</section>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S6.SS2.2.1.1" class="ltx_text ltx_font_sansserif">6.2</span> </span><span id="S6.SS2.3.2" class="ltx_text ltx_font_sansserif">Post-training evaluation</span>
</h3>

<div id="S6.SS2.p1" class="ltx_para">
<p id="S6.SS2.p1.1" class="ltx_p">We evaluate post-training models on comprehensive benchmarks and
compare AFM models with various open-source models, as well as GPT-3.5 and GPT-4.<span id="footnotex1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>
We compared against the following model versions: gpt-3.5-turbo-0125, gpt-4-0125-preview, Gemini<span id="footnotex1.1" class="ltx_text">-</span>1.5<span id="footnotex1.2" class="ltx_text">-</span>Pro<span id="footnotex1.3" class="ltx_text">-</span>0514, DBRX Instruct, Phi-3-mini-4k-instruct, LLaMA 3 8B Instruct, LLaMA 3 70B Instruct, Mistral-7B-Instruct-v0.2, Mixtral-8x22B-Instruct-v0.1, Gemma-1.1-2B, and Gemma-1.1-7B.</span></span></span>
All results reported in this section are obtained using AFM-on-device and AFM-server base models without any adapter, in <span id="S6.SS2.p1.1.1" class="ltx_text ltx_font_typewriter">bfloat16</span> precision. In this section, we first present human evaluation results that measure the AFMs general capabilities, and then present results for several specific capabilities and domains.</p>
</div>
<section id="S6.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S6.SS2.SSS1.2.1.1" class="ltx_text ltx_font_sansserif">6.2.1</span> </span><span id="S6.SS2.SSS1.3.2" class="ltx_text ltx_font_sansserif">Human evaluation</span>
</h4>

<div id="S6.SS2.SSS1.p1" class="ltx_para">
<p id="S6.SS2.SSS1.p1.1" class="ltx_p">Human evaluation simulates practical use cases and user feedback, and so often serves as the gold standard for language model evaluation.
Consequently, we conduct extensive human evaluations both while developing the model and to evaluate its final form.
We collect sets of evaluation prompts to test the model on different aspects, including both general capabilities and safety.
For each prompt, two model responses are presented to human raters anonymously for side-by-side comparisons.
Depending on the nature of the evaluation,
a detailed guideline containing grading principles and examples of single-response ratings and side-by-side preference ratings is provided to human raters to ensure consistent grading standards and evaluation quality.
Each pair of model responses is graded by multiple graders and their ratings are aggregated for final results.
Overall, we find human evaluation to align better with user experience and provide a better evaluation signal than some academic benchmarks that use LLMs as graders. In this section, we present the results for human evaluation on general capabilities, and the safety evaluation results are provided in <a href="#S7.SS6" title="7.6 Evaluation ‣ 7 Responsible AI ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 7.6</span></a>.</p>
</div>
<div id="S6.SS2.SSS1.p2" class="ltx_para">
<p id="S6.SS2.SSS1.p2.1" class="ltx_p">We collect a comprehensive set of <math id="S6.SS2.SSS1.p2.1.m1.1" class="ltx_Math" alttext="1393" display="inline"><semantics id="S6.SS2.SSS1.p2.1.m1.1a"><mn id="S6.SS2.SSS1.p2.1.m1.1.1" xref="S6.SS2.SSS1.p2.1.m1.1.1.cmml">1393</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.SSS1.p2.1.m1.1b"><cn type="integer" id="S6.SS2.SSS1.p2.1.m1.1.1.cmml" xref="S6.SS2.SSS1.p2.1.m1.1.1">1393</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.SSS1.p2.1.m1.1c">1393</annotation></semantics></math> prompts to evaluate the general model capabilities.
These prompts are diverse across different difficulty levels and cover major categories including:
analytical reasoning,
brainstorming,
chatbot,
classification,
closed question answering,
coding,
extraction,
mathematical reasoning,
open question answering,
rewriting,
safety,
summarization,
and writing.
To prevent overfitting, when preparing training data, we conduct decontamination against our evaluation prompts.</p>
</div>
<figure id="S6.F3" class="ltx_figure"><img src="/html/2407.21075/assets/x3.png" id="S6.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="354" alt="Chart showing which models win, tie, or loose in a direct comparison. On-device comparisons are at the top and server comparisons are at the bottom.">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Side-by-side evaluation of AFM-on-device and AFM-server against comparable models. We find that our models are often preferred over competitor models by human graders.</figcaption>
</figure>
<div id="S6.SS2.SSS1.p3" class="ltx_para">
<p id="S6.SS2.SSS1.p3.1" class="ltx_p">In <a href="#S6.F3" title="Figure 3 ‣ 6.2.1 Human evaluation ‣ 6.2 Post-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 3</span></a>, we compare AFM with both open-source models (Phi-3, Gemma-1.1, Llama-3, Mistral, DBRX-Instruct) and commercial models (GPT-3.5, and GPT-4).
AFM models are preferred by human graders over competitor models.
In particular, AFM-on-device obtains a win rate of 47.7% when compared to Phi-3-mini despite being 25% smaller in model sizes,
and even outperforms open-source strong baselines Gemma-7B and Mistral-7B that are more than twice larger in the number of parameters.
When compared to closed-source models, AFM-server achieves competitive performance, scoring a win rate of more than <math id="S6.SS2.SSS1.p3.1.m1.1" class="ltx_Math" alttext="50\%" display="inline"><semantics id="S6.SS2.SSS1.p3.1.m1.1a"><mrow id="S6.SS2.SSS1.p3.1.m1.1.1" xref="S6.SS2.SSS1.p3.1.m1.1.1.cmml"><mn id="S6.SS2.SSS1.p3.1.m1.1.1.2" xref="S6.SS2.SSS1.p3.1.m1.1.1.2.cmml">50</mn><mo id="S6.SS2.SSS1.p3.1.m1.1.1.1" xref="S6.SS2.SSS1.p3.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.SSS1.p3.1.m1.1b"><apply id="S6.SS2.SSS1.p3.1.m1.1.1.cmml" xref="S6.SS2.SSS1.p3.1.m1.1.1"><csymbol cd="latexml" id="S6.SS2.SSS1.p3.1.m1.1.1.1.cmml" xref="S6.SS2.SSS1.p3.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S6.SS2.SSS1.p3.1.m1.1.1.2.cmml" xref="S6.SS2.SSS1.p3.1.m1.1.1.2">50</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.SSS1.p3.1.m1.1c">50\%</annotation></semantics></math> and a tie rate of 27.4% against GPT-3.5.</p>
</div>
</section>
<section id="S6.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S6.SS2.SSS2.2.1.1" class="ltx_text ltx_font_sansserif">6.2.2</span> </span><span id="S6.SS2.SSS2.3.2" class="ltx_text ltx_font_sansserif">Instruction following</span>
</h4>

<div id="S6.SS2.SSS2.p1" class="ltx_para">
<p id="S6.SS2.SSS2.p1.1" class="ltx_p">Instruction following (IF) is the core capability we desire of language models,
as real-world prompts are often sophisticated and contain complex instructions.
We emphasize the importance of instruction following in both our RLHF data collection and human evaluation.
In this subsection, we evaluate our models’ IF skills using automated benchmarks.</p>
</div>
<div id="S6.SS2.SSS2.p2" class="ltx_para">
<p id="S6.SS2.SSS2.p2.1" class="ltx_p">In <a href="#S6.F4" title="Figure 4 ‣ 6.2.2 Instruction following ‣ 6.2 Post-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 4</span></a>
we evaluate AFM-on-device and AFM-server on the public IFEval benchmark <cite class="ltx_cite ltx_citemacro_citep">[Zhou et al., <a href="#bib.bib65" title="" class="ltx_ref">2023</a>]</cite>, respectively.
This benchmark measures a language model’s capability to generate responses that precisely follow instructions in the prompt. The instructions in this benchmark typically include requirements on the response length, format, content, etc.
We find AFM-on-device and AFM-server to achieve superior performance on both instruction-level and prompt-level accuracy.
In addition,
we also benchmark AFM models on the AlpacaEval 2.0 LC benchmark <cite class="ltx_cite ltx_citemacro_citep">[Dubois et al., <a href="#bib.bib15" title="" class="ltx_ref">2024</a>]</cite> to measure general instruction-following capability, and results suggest that our models are highly competitive.</p>
</div>
<figure id="S6.F4" class="ltx_figure"><img src="/html/2407.21075/assets/x4.png" id="S6.F4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="507" height="569" alt="Chart with bars for the porformance of various models on various benchmarks. On-device models are on the left and server models are on the right.">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Instruction-following capability (measured with IFEval) for AFM models and relevant comparison models (higher is better). The AlpacaEval 2.0 LC results for Mistral 7B, Llama3 8B, Llama3 70B, DBRX-Instruct, and Mixtral 8x22B are obtained from <a target="_blank" href="https://tatsu-lab.github.io/alpaca_eval/" title="" class="ltx_ref ltx_href">the AlpacaEval leaderboard</a> <cite class="ltx_cite ltx_citemacro_citep">[Taori et al., <a href="#bib.bib49" title="" class="ltx_ref">2023</a>]</cite>. The Arena Hard results for comparison models are from <a target="_blank" href="https://github.com/lm-sys/arena-hard-auto" title="" class="ltx_ref ltx_href">the Arena-Hard-Auto leaderboard</a> <cite class="ltx_cite ltx_citemacro_citep">[Li et al., <a href="#bib.bib33" title="" class="ltx_ref">2024b</a>]</cite>. All other results are from our own evaluations.</figcaption>
</figure>
</section>
<section id="S6.SS2.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S6.SS2.SSS3.2.1.1" class="ltx_text ltx_font_sansserif">6.2.3</span> </span><span id="S6.SS2.SSS3.3.2" class="ltx_text ltx_font_sansserif">Tool use</span>
</h4>

<div id="S6.SS2.SSS3.p1" class="ltx_para">
<p id="S6.SS2.SSS3.p1.1" class="ltx_p">In tool use applications,
given a user request and a list of potential tools with descriptions,
the model can choose to issue tool calls by providing a structured output specifying the name and parameter values of the tools to call. We expect the tool descriptions to follow the OpenAPI specification.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>https://github.com/OAI/OpenAPI-Specification</span></span></span></p>
</div>
<div id="S6.SS2.SSS3.p2" class="ltx_para">
<p id="S6.SS2.SSS3.p2.1" class="ltx_p">We evaluate on the public Berkeley Function Calling Leaderboard benchmarks <cite class="ltx_cite ltx_citemacro_citep">[Patil et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>]</cite>
via native support of function calling, using the AST metrics.</p>
</div>
<div id="S6.SS2.SSS3.p3" class="ltx_para">
<p id="S6.SS2.SSS3.p3.1" class="ltx_p">As shown in <a href="#S6.F5" title="Figure 5 ‣ 6.2.3 Tool use ‣ 6.2 Post-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 5</span></a>, AFM-server achieves the best overall accuracy, outperforming Gemini-1.5-Pro-Preview-0514 and GPT-4.</p>
</div>
<figure id="S6.F5" class="ltx_figure"><img src="/html/2407.21075/assets/x5.png" id="S6.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="507" height="325" alt="Chart with bars comparing the score of various models on five function calling benchmarks.">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>
Berkeley Function Calling Leaderboard Benchmark
evaluation results on Function Calling API, along-side relevant sampled comparisons.
Numbers were collected from <a target="_blank" href="https://huggingface.co/spaces/gorilla-llm/berkeley-function-calling-leaderboard/blob/e8e0a803d621a067f515ea4a0b970240a9fc19ed/data.csv" title="" class="ltx_ref ltx_href">the Gorilla leaderboard</a> <cite class="ltx_cite ltx_citemacro_citep">[Patil et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>]</cite>.
</figcaption>
</figure>
</section>
<section id="S6.SS2.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S6.SS2.SSS4.2.1.1" class="ltx_text ltx_font_sansserif">6.2.4</span> </span><span id="S6.SS2.SSS4.3.2" class="ltx_text ltx_font_sansserif">Writing</span>
</h4>

<div id="S6.SS2.SSS4.p1" class="ltx_para">
<p id="S6.SS2.SSS4.p1.1" class="ltx_p">Writing is one of the most critical abilities for large language models to have, as it empowers various downstream use cases such as changing-of-tone, rewriting, and summarization.
However, assessing writing quality is a non-trivial task, and not well-covered in the above public benchmarks.</p>
</div>
<div id="S6.SS2.SSS4.p2" class="ltx_para">
<p id="S6.SS2.SSS4.p2.1" class="ltx_p">We evaluate AFM’s writing ability on our internal summarization and composition benchmarks,
consisting of a variety of writing instructions.
Following LLM-as-a-judge <cite class="ltx_cite ltx_citemacro_citep">[Zheng et al., <a href="#bib.bib63" title="" class="ltx_ref">2024</a>]</cite>, we design a grading instruction
for each summarization and composition task,
and prompt GPT-4 Turbo to assign a score from 1 to 10 for model responses.<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>
Due to the choice of using GPT-4 as judge, the score of GPT-4 Turbo can be overestimated.
</span></span></span>
We note that there are certain limitations and biases associated with using an LLM as a grader, such as length bias.</p>
</div>
<div id="S6.SS2.SSS4.p3" class="ltx_para">
<p id="S6.SS2.SSS4.p3.1" class="ltx_p">We compare AFM with a few of the most outstanding models,
along with smaller-scale open-source models.
As shown in <a href="#S6.F6" title="Figure 6 ‣ 6.2.4 Writing ‣ 6.2 Post-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 6</span></a>,
AFM-on-device can achieve comparable or superior performance when compared to Gemma-7B and Mistral-7B.
AFM-server significantly outperforms DBRX-Instruct and GPT3.5 and is comparable to GPT4.</p>
</div>
<figure id="S6.F6" class="ltx_figure"><img src="/html/2407.21075/assets/x6.png" id="S6.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="507" height="301" alt="Chart with bars showing the performance of various models for summarization and generation benchmarks. On-device models are on the left and server models are on the right.">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Writing ability on internal summarization and composition benchmarks (higher is better) for AFM-on-device and AFM-server alongside
relevant sampled comparisons. We find that our models perform better or similar to related models.</figcaption>
</figure>
</section>
<section id="S6.SS2.SSS5" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection"><span id="S6.SS2.SSS5.2.1.1" class="ltx_text ltx_font_sansserif">6.2.5</span> </span><span id="S6.SS2.SSS5.3.2" class="ltx_text ltx_font_sansserif">Math</span>
</h4>

<div id="S6.SS2.SSS5.p1" class="ltx_para">
<p id="S6.SS2.SSS5.p1.2" class="ltx_p">In <a href="#S6.F7" title="Figure 7 ‣ 6.2.5 Math ‣ 6.2 Post-training evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 7</span></a>, we compare post-training AFM’s performance on math benchmarks including GSM8K <cite class="ltx_cite ltx_citemacro_citep">[Cobbe et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>]</cite> and MATH <cite class="ltx_cite ltx_citemacro_citep">[Hendrycks et al., <a href="#bib.bib18" title="" class="ltx_ref">2021</a>]</cite>. We use <math id="S6.SS2.SSS5.p1.1.m1.1" class="ltx_Math" alttext="8" display="inline"><semantics id="S6.SS2.SSS5.p1.1.m1.1a"><mn id="S6.SS2.SSS5.p1.1.m1.1.1" xref="S6.SS2.SSS5.p1.1.m1.1.1.cmml">8</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.SSS5.p1.1.m1.1b"><cn type="integer" id="S6.SS2.SSS5.p1.1.m1.1.1.cmml" xref="S6.SS2.SSS5.p1.1.m1.1.1">8</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.SSS5.p1.1.m1.1c">8</annotation></semantics></math>-shot chain-of-thought (CoT) <cite class="ltx_cite ltx_citemacro_citep">[Wei et al., <a href="#bib.bib54" title="" class="ltx_ref">2022</a>]</cite> prompt for GSM8K and <math id="S6.SS2.SSS5.p1.2.m2.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S6.SS2.SSS5.p1.2.m2.1a"><mn id="S6.SS2.SSS5.p1.2.m2.1.1" xref="S6.SS2.SSS5.p1.2.m2.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.SSS5.p1.2.m2.1b"><cn type="integer" id="S6.SS2.SSS5.p1.2.m2.1.1.cmml" xref="S6.SS2.SSS5.p1.2.m2.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.SSS5.p1.2.m2.1c">4</annotation></semantics></math>-shot CoT prompt <cite class="ltx_cite ltx_citemacro_citep">[Lewkowycz et al., <a href="#bib.bib31" title="" class="ltx_ref">2022</a>]</cite> for MATH. We conduct all evaluations using an internal automated evaluation pipeline. We see that the AFM-on-device significantly outperforms Mistral-7B and Gemma-7B, even at less than half of their sizes.</p>
</div>
<figure id="S6.F7" class="ltx_figure"><img src="/html/2407.21075/assets/x7.png" id="S6.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="507" height="300" alt="Chart with bars showing the performance of various models for GSM8K and MATH benchmarks. On-device models are on the left and server models are on the right.">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Math benchmarks for AFM-on-device and AFM-server alongside relevant sampled comparisons. GSM8K is 8-shot and MATH is 4-shot.
All results are collected with an internal automated evaluation pipeline.
</figcaption>
</figure>
</section>
</section>
<section id="S6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S6.SS3.2.1.1" class="ltx_text ltx_font_sansserif">6.3</span> </span><span id="S6.SS3.3.2" class="ltx_text ltx_font_sansserif">Summarization feature evaluation</span>
</h3>

<div id="S6.SS3.p1" class="ltx_para">
<p id="S6.SS3.p1.1" class="ltx_p">The product team specifications for summarizing Emails, Messages, and Notifications necessitated a tailor-made set of guidelines, metrics, and specialized graders to evaluate summarization quality against various open-source, licensed, and proprietary datasets.</p>
</div>
<section id="S6.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Datasets.</h5>

<div id="S6.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S6.SS3.SSS0.Px1.p1.1" class="ltx_p">We sampled abundant payloads carefully for each use case.
These evaluation datasets emphasize a diverse set of inputs which our product features are likely to face in production, and include a stratified mixture of single and stacked documents of varying content types and lengths. We developed a pipeline to build evaluation datasets that simulate real user inputs.</p>
</div>
</section>
<section id="S6.SS3.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Graders.</h5>

<div id="S6.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S6.SS3.SSS0.Px2.p1.1" class="ltx_p">We enlisted a pool of highly-trained, full-time, Apple-employed human graders with specialized writing and comprehension skills to evaluate summarization quality. To qualify for grading projects, each grader must pass a series of eligibility and training steps, which include a required bachelor’s degree in a writing-related discipline, customized training sessions, and consistently high performance against internal grading quality benchmarks.</p>
</div>
</section>
<section id="S6.SS3.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Grading guidelines.</h5>

<div id="S6.SS3.SSS0.Px3.p1" class="ltx_para">
<p id="S6.SS3.SSS0.Px3.p1.1" class="ltx_p">During the evaluation task, graders are presented with a specification for the summary, the original input content, and the output summary. Graders assess the summary on each the following sub-dimensions of quality using 3 point scales (“good”, “neutral”, or “poor” ):</p>
<ul id="S6.I1" class="ltx_itemize">
<li id="S6.I1.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"></span>
<div id="S6.I1.ix1.p1" class="ltx_para">
<p id="S6.I1.ix1.p1.1" class="ltx_p"><span id="S6.I1.ix1.p1.1.1" class="ltx_text ltx_font_bold">Composition</span>: Evaluates the overall readability of the summary considering grammar, punctuation, spelling, and brevity.</p>
</div>
</li>
<li id="S6.I1.ix2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"></span>
<div id="S6.I1.ix2.p1" class="ltx_para">
<p id="S6.I1.ix2.p1.1" class="ltx_p"><span id="S6.I1.ix2.p1.1.1" class="ltx_text ltx_font_bold">Comprehensiveness</span>: Evaluates how comprehensive the summary is in capturing the essential points or calling out any actions/conclusions for the user.</p>
</div>
</li>
<li id="S6.I1.ix3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"></span>
<div id="S6.I1.ix3.p1" class="ltx_para">
<p id="S6.I1.ix3.p1.1" class="ltx_p"><span id="S6.I1.ix3.p1.1.1" class="ltx_text ltx_font_bold">Groundedness</span>: Evaluates how grounded the summary is with respect to the original payload. Summaries that are not completely grounded may contain details that are exaggerated, inferred, inaccurate, or hallucinated.</p>
</div>
</li>
<li id="S6.I1.ix4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"></span>
<div id="S6.I1.ix4.p1" class="ltx_para">
<p id="S6.I1.ix4.p1.1" class="ltx_p"><span id="S6.I1.ix4.p1.1.1" class="ltx_text ltx_font_bold">Following instructions</span>: Evaluates whether the summary meets specific style and formatting requirements. Requirements are tailored to each feature and reflect specific product and design expectations.</p>
</div>
</li>
<li id="S6.I1.ix5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item"></span>
<div id="S6.I1.ix5.p1" class="ltx_para">
<p id="S6.I1.ix5.p1.1" class="ltx_p"><span id="S6.I1.ix5.p1.1.1" class="ltx_text ltx_font_bold">Harmfulness</span>: Evaluates whether the summary contains content that is harmful or unsafe according to Apple’s safety taxonomy.</p>
</div>
</li>
</ul>
</div>
<div id="S6.SS3.SSS0.Px3.p2" class="ltx_para">
<p id="S6.SS3.SSS0.Px3.p2.1" class="ltx_p">A summary is classified as “poor” if <span id="S6.SS3.SSS0.Px3.p2.1.1" class="ltx_text ltx_font_italic">any</span> of the sub-dimensions are “poor” according to predefined product specifications. Likewise a summary is “good” only if <span id="S6.SS3.SSS0.Px3.p2.1.2" class="ltx_text ltx_font_italic">all</span> sub-dimensions are good. These classifications are used to compute “Good/Poor Result Ratio” metrics defined as the percentage of good/poor summaries out of all summaries.</p>
</div>
</section>
<section id="S6.SS3.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Results.</h5>

<div id="S6.SS3.SSS0.Px4.p1" class="ltx_para">
<p id="S6.SS3.SSS0.Px4.p1.1" class="ltx_p">We ask human graders to evaluate the summarization quality of the AFM-on-device adapter, Phi-3-mini, Llama-3-8B, and Gemma-7B.
<a href="#S6.F8" title="Figure 8 ‣ Results. ‣ 6.3 Summarization feature evaluation ‣ 6 Evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 8</span></a> shows that AFM-on-device-adapter overall outperforms the other models.</p>
</div>
<figure id="S6.F8" class="ltx_figure"><img src="/html/2407.21075/assets/x8.png" id="S6.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="507" height="362" alt="Chart with bars showing the good result ratio and poor result ratio for email, messages, and notification use cases.">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>Ratio of “good” and “poor” responses for three summarization use cases relative to all responses. Summaries are classified as “good”, “neutral”, or “poor” along five dimensions. A result is classified as “good” if <span id="S6.F8.3.1" class="ltx_text ltx_font_italic">all</span> of the dimensions are good (higher is better). A result is classified as “poor” if <span id="S6.F8.4.2" class="ltx_text ltx_font_italic">any</span> of the dimensions are poor (lower is better). Overall, our AFM-on-device adapter generates better summaries than comparable models.</figcaption>
</figure>
</section>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S7.2.1.1" class="ltx_text ltx_font_sansserif">7</span> </span><span id="S7.3.2" class="ltx_text ltx_font_sansserif">Responsible AI</span>
</h2>

<section id="S7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S7.SS1.2.1.1" class="ltx_text ltx_font_sansserif">7.1</span> </span><span id="S7.SS1.3.2" class="ltx_text ltx_font_sansserif">Overview</span>
</h3>

<div id="S7.SS1.p1" class="ltx_para">
<p id="S7.SS1.p1.1" class="ltx_p">Apple Intelligence is developed responsibly and designed with care to empower our users, represent them authentically, and protect their privacy. Of primary importance to our Responsible AI approach is that we are ultimately delivering intelligent, well-defined tools that address specific user needs. Having a clear definition of what a feature is intended to do allows us to better identify any potential safety gaps.</p>
</div>
<div id="S7.SS1.p2" class="ltx_para">
<p id="S7.SS1.p2.1" class="ltx_p">We have developed a safety taxonomy in order to be comprehensive and consistent in the design and evaluation of our generative AI-powered features. This taxonomy builds and extends Apple’s extensive experience in using artificial intelligence and machine learning to deliver helpful features to users around the world, and is updated regularly as we develop and test features. Currently, it consists of 12 primary categories comprised of 51 subcategories, including “Hate Speech, Stereotypes, and Slurs”, “Discrimination,
Marginalization, and Exclusion”, “Illegal Activities”, “Adult Sexual Material”, and “Graphic Violence.”</p>
</div>
<div id="S7.SS1.p3" class="ltx_para">
<p id="S7.SS1.p3.1" class="ltx_p">The taxonomy serves as a structured way to consider potential issues and risks relative to each specific feature. As new or additional risks are identified, we develop and revise the associated policies that are contextualized to each individual feature, taking into account the specific needs that it serves, the content it produces, and the appropriate mitigations.
They are developed with extensive internal and external input from academics, AI ethicists, trust and safety, and legal experts to better identify and understand the relevant risks, the potential severity of such risks, and the potential disparate impact these risks may have on certain groups.
These policies guide our work in data collection, human annotation, model training, guardrails development, evaluation, and red teaming.</p>
</div>
<div id="S7.SS1.p4" class="ltx_para">
<p id="S7.SS1.p4.1" class="ltx_p">Particularly, the taxonomy is not itself the sole determinant of our policy. For example, content that may fall within the safety taxonomy is not necessarily always blocked, as doing so unilaterally may be in conflict with other aspects of Apple’s Responsible AI development principles, such as “respecting how our users choose to use these tools to accomplish their goals.” Thus, features that operate as tools may be more permissive in the kinds of content they operate over and produce in order to effectively address the user’s intent. On the other hand, features that may generate content beyond a user’s specified intent may need to be more constrained. Regardless, we strive for some categories of harm to always be treated with special care (such as any content that relates to self harm) while other categories will always be blocked (such as illegal content).</p>
</div>
<div id="S7.SS1.p5" class="ltx_para">
<p id="S7.SS1.p5.1" class="ltx_p">In addition, our Responsible AI principles are built into every stage of Apple Foundation Models and Apple Intelligence as well as the safety taxonomy, which helps us evaluate risks and formulate policies feature by feature. We include safety-oriented data as part of our fine-tuning of specific adapters tailored by use case. Furthermore, at the time of inference, we also run guardrail models <cite class="ltx_cite ltx_citemacro_citep">[Inan et al., <a href="#bib.bib25" title="" class="ltx_ref">2023</a>]</cite> as pre- and post-processing steps to evaluate potential harm at both the input and output level. Finally, we have mechanisms in place to continuously and proactively improve our AI tools with the help of ongoing user feedback.</p>
</div>
</section>
<section id="S7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S7.SS2.2.1.1" class="ltx_text ltx_font_sansserif">7.2</span> </span><span id="S7.SS2.3.2" class="ltx_text ltx_font_sansserif">Pre-Training</span>
</h3>

<div id="S7.SS2.p1" class="ltx_para">
<p id="S7.SS2.p1.1" class="ltx_p">At the pre-training stage, we take several steps to ensure that the values as outlined above are upheld. We follow a strict data policy ensuring that no Apple user data is included, as well as conduct rigorous legal review for each component in the training corpus. Further, we perform safety filtering to reduce potentially harmful content, including NSFW content, profanity, spam, and PII or financial data.</p>
</div>
<div id="S7.SS2.p2" class="ltx_para">
<p id="S7.SS2.p2.1" class="ltx_p">Because pre-training is a step which is shared among various downstream features, our safety mitigations aim to retain general capabilities that allow us to iterate on the taxonomy and policy at a per-feature level, without hurting the helpfulness of these downstream models. We take learnings from prior work to avoid overly aggressive filtering at the pre-training stage, which has potential benefits in safety alignment <cite class="ltx_cite ltx_citemacro_citep">[Touvron et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>]</cite>. Intuitively, the pre-trained model should be aware of content that downstream features and policies may require it to handle – in some cases with care, or in other cases operating over such content directly.</p>
</div>
</section>
<section id="S7.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S7.SS3.2.1.1" class="ltx_text ltx_font_sansserif">7.3</span> </span><span id="S7.SS3.3.2" class="ltx_text ltx_font_sansserif">Post-Training</span>
</h3>

<div id="S7.SS3.p1" class="ltx_para">
<p id="S7.SS3.p1.1" class="ltx_p">In the post-training phase, we aim to instill a baseline level of alignment with our Responsible AI principles to avoid necessitating the full complexities of post-training (such as RLHF) in each downstream model that builds on top of the foundation model. In doing so, there are two key considerations:</p>
</div>
<div id="S7.SS3.p2" class="ltx_para">
<ol id="S7.I1" class="ltx_enumerate">
<li id="S7.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S7.I1.i1.p1" class="ltx_para">
<p id="S7.I1.i1.p1.1" class="ltx_p">We must ensure our models produce output that is helpful to users, while minimizing potential harm.</p>
</div>
</li>
<li id="S7.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S7.I1.i2.p1" class="ltx_para">
<p id="S7.I1.i2.p1.1" class="ltx_p">We must contextualize our safety taxonomy and policies on a feature by feature basis to deliver the best possible user experience.</p>
</div>
</li>
</ol>
</div>
<div id="S7.SS3.p3" class="ltx_para">
<p id="S7.SS3.p3.1" class="ltx_p">To balance helpfulness and harmlessness trade-off, our solution is to treat safety alignment as one of the many core post-training tasks that are evaluated and iterated on in tandem, instead of as a separate stage of training. Specifically, we include adversarial data into our SFT and RLHF training corpora that is curated according to our policy and values by partnering closely with trusted vendors. We also incorporate safety tasks and benchmarks into the automatic and human evaluations used during model development.</p>
</div>
<div id="S7.SS3.p4" class="ltx_para">
<p id="S7.SS3.p4.1" class="ltx_p">In total, over 10% of the training data are adversarial or related to safety or sensitive topics, including single and multi-turn safety category annotations, pairwise and overall preference ratings, and annotator rewrites. This data is either used directly or as seed data for synthetic data generation, as described in <a href="#S4.SS1.SSS2" title="4.1.2 Synthetic data ‣ 4.1 Data ‣ 4 Post-Training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 4.1.2</span></a>.</p>
</div>
<div id="S7.SS3.p5" class="ltx_para">
<p id="S7.SS3.p5.1" class="ltx_p">We do additional work to achieve appropriate safety behavior for each feature beyond baseline alignment. A primary way that we do this is by collecting safety-specific training data and including it when fine-tuning adapters. For instance,
in fine-tuning our summarization adapter we sought to improve aspects such as, improving robustness against malicious questions included within the content to be summarized, and reducing the likelihood that summaries would inadvertently amplify harmful or sensitive content to be summarized.</p>
</div>
</section>
<section id="S7.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S7.SS4.2.1.1" class="ltx_text ltx_font_sansserif">7.4</span> </span><span id="S7.SS4.3.2" class="ltx_text ltx_font_sansserif">Guarding against malicious code</span>
</h3>

<div id="S7.SS4.p1" class="ltx_para">
<p id="S7.SS4.p1.1" class="ltx_p">Code generation requires special care. Our code benchmarks involve actually executing the generated code to determine both syntactic and semantic correctness. Thus, responsible training of code models involves treating all generated code as unsafe by default – all code is always executed in a fully locked down environment with no access to the internet or any internal or external services. Specifically, the locked down environment is managed with FireCracker <cite class="ltx_cite ltx_citemacro_citep">[Agache et al., <a href="#bib.bib2" title="" class="ltx_ref">2020</a>]</cite>, with a FireCracker jailer at the cluster level.</p>
</div>
</section>
<section id="S7.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S7.SS5.2.1.1" class="ltx_text ltx_font_sansserif">7.5</span> </span><span id="S7.SS5.3.2" class="ltx_text ltx_font_sansserif">Red teaming</span>
</h3>

<div id="S7.SS5.p1" class="ltx_para">
<p id="S7.SS5.p1.1" class="ltx_p">Red teaming attempts to elicit safety policy violating responses from models, or harmful responses for which no policy yet exists. These results inform both policy development as well as the focus and content of safety evaluation datasets. These in turn can influence design, engineering, and shipping readiness decisions.</p>
</div>
<div id="S7.SS5.p2" class="ltx_para">
<p id="S7.SS5.p2.1" class="ltx_p">Red teaming is a fundamentally creative endeavor that requires red teamers to employ combinations of attack vectors to probe known model vulnerabilities, and try to discover new ones. Attack vectors used when engaging with language models include jailbreaks/prompt injections, persuasive techniques <cite class="ltx_cite ltx_citemacro_citep">[Zeng et al., <a href="#bib.bib61" title="" class="ltx_ref">2024</a>]</cite>, and linguistic features known to cause model misbehavior (e.g. slang, code-switching, emojis, typos).</p>
</div>
<div id="S7.SS5.p3" class="ltx_para">
<p id="S7.SS5.p3.1" class="ltx_p">We employ both manual and automatic red-teaming <cite class="ltx_cite ltx_citemacro_citep">[Ganguli et al., <a href="#bib.bib17" title="" class="ltx_ref">2022</a>]</cite> to elicit potentially unknown failure modes of the aligned models. More recent works <cite class="ltx_cite ltx_citemacro_citep">[Touvron et al., <a href="#bib.bib51" title="" class="ltx_ref">2023</a>]</cite> suggest that automated processes can potentially generate even more diverse prompts than humans, previously seen as the “gold” standard for data collection. These automated processes can include using the language models themselves to identify gaps, some of which may be unintuitive or even surprising. Such examples can be used directly as synthetic training or evaluation data and to inform future data collection efforts.</p>
</div>
<div id="S7.SS5.p4" class="ltx_para">
<p id="S7.SS5.p4.1" class="ltx_p">A basic human red teaming task schema is as follows: a red teamer is assigned a safety taxonomy category and attack vector(s). They author an input to the model, using that attack vector, that is intended to elicit a response containing content from that category. If the response does not contain the target content, the red teamer can engage in a fixed number of conversational turns, after which they provide a final harmfulness rating of the model output and list the taxonomy categor(ies) in it, if any. To ensure annotation quality, red teamers also provide an overall confidence score for their ratings.</p>
</div>
<div id="S7.SS5.p5" class="ltx_para">
<p id="S7.SS5.p5.1" class="ltx_p">In addition to red teaming at the base model level, we also red team specific features. Red teaming projects at the feature level use feature-specific guidelines with attack vectors informed by the feature’s safety policy and engineering concerns. These projects can provide in-depth probing of known risks for that particular feature and also adversarially probe for unknown vulnerabilities.</p>
</div>
<div id="S7.SS5.p6" class="ltx_para">
<p id="S7.SS5.p6.1" class="ltx_p">Our red teaming projects are run using internal and external crowds. To ensure responsible data collection, due to the sensitive nature of red teaming we: 1) make red teaming completely voluntary; 2) impose a strict time limit on how much each red teamer spends on the tasks per week; 3) provide health and well-being resources available around the clock; and 4) maintain an open line of communication with internal red teamers via weekly office hours and a Slack channel for them to communicate any concerns that arise.</p>
</div>
</section>
<section id="S7.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="S7.SS6.2.1.1" class="ltx_text ltx_font_sansserif">7.6</span> </span><span id="S7.SS6.3.2" class="ltx_text ltx_font_sansserif">Evaluation</span>
</h3>

<div id="S7.SS6.p1" class="ltx_para">
<p id="S7.SS6.p1.1" class="ltx_p">As mentioned in previous sections, safety is one of the many axes iterated on during foundation model development, and therefore undergoes the same automatic and human evaluation cycles during post-training.</p>
</div>
<section id="S7.SS6.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Safety evaluation set</h5>

<div id="S7.SS6.SSS0.Px1.p1" class="ltx_para">
<p id="S7.SS6.SSS0.Px1.p1.1" class="ltx_p">To reduce noise, cost, and turn-around time during human evaluations, we must ensure that our safety evaluation sets are clean, yet challenging and comprehensive. To that end, we filter out “easy" prompts which consistently yield low harmfulness responses across different versions of the model, and employ an embedding-based analysis to improve our evaluation prompt set coverage. Overall, we curate a set of over a thousand adversarial prompts to test AFM’s performance on harmful content, sensitive topics, and factuality according to our safety policy.</p>
</div>
<figure id="S7.F9" class="ltx_figure"><img src="/html/2407.21075/assets/x9.png" id="S7.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="507" height="190" alt="Chart with bars showing the performance of various models for output harmfulness. On-device models are on the left and server models are on the right.">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Fraction of violating responses for harmful content, sensitive topics, and factuality (lower is better). Our models are robust when faced with adversarial prompts.
</figcaption>
</figure>
<figure id="S7.F10" class="ltx_figure"><img src="/html/2407.21075/assets/x10.png" id="S7.F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="321" alt="Chart showing which models win, tie, or loose in a direct comparison for safety prompts. On-device comparisons are at the top and server comparisons are at the bottom.">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>Fraction of preferred responses in side-by-side evaluation of Apple’s foundation model against comparable models on safety prompts. Human graders found our responses safer and more helpful.
</figcaption>
</figure>
</section>
<section id="S7.SS6.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Safety evaluation results</h5>

<div id="S7.SS6.SSS0.Px2.p1" class="ltx_para">
<p id="S7.SS6.SSS0.Px2.p1.1" class="ltx_p"><a href="#S7.F9" title="Figure 9 ‣ Safety evaluation set ‣ 7.6 Evaluation ‣ 7 Responsible AI ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 9</span></a> summarizes the violation rates of different models evaluated by human graders on this safety evaluation set. Lower is better.
Both AFM-on-device and AFM-server are robust to adversarial prompts,
achieving violation rates significantly lower than open-source and commercial models.
In addition, we report side-by-side human preference on our safety evaluation prompts in
<a href="#S7.F10" title="Figure 10 ‣ Safety evaluation set ‣ 7.6 Evaluation ‣ 7 Responsible AI ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Figure 10</span></a>. AFM models are preferred by human graders as safe and helpful responses over competitor models.</p>
</div>
</section>
</section>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section"><span id="S8.2.1.1" class="ltx_text ltx_font_sansserif">8</span> </span><span id="S8.3.2" class="ltx_text ltx_font_sansserif">Conclusion</span>
</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">In this report we introduced the foundation language models that power Apple Intelligence features, AFM-on-device and AFM-server.
The models are designed to be fast and run efficiently on iPhone, iPad, and Mac as well as on Apple silicon servers via Private Cloud Compute.
They are trained to be highly capable in tasks like language understanding, instruction following, reasoning, writing, and tool use.
We have developed an innovative model architecture to specialize these models for our users’ most common tasks.
On top of the foundation models, feature-specific adapters are fine-tuned to provide high-quality user experiences such as summarization of emails, messages, and notifications.
Our models have been created with the purpose of helping users do everyday activities across their Apple products, grounded in Apple’s core values, and rooted in our Responsible AI principles at every stage.
These foundation models are at the heart of Apple Intelligence, the personal intelligence system built by Apple to continue empowering our users and enriching their lives.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Abbasi-Yadkori et al. [2019]</span>
<span class="ltx_bibblock">
Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellért Weisz.

</span>
<span class="ltx_bibblock">Politex: Regret bounds for policy iteration using expert prediction.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pages 3692–3702. PMLR, 2019.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Agache et al. [2020]</span>
<span class="ltx_bibblock">
Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa.

</span>
<span class="ltx_bibblock">Firecracker: Lightweight virtualization for serverless applications.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">17th USENIX symposium on networked systems design and implementation (NSDI 20)</em>, pages 419–434, 2020.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ahmadian et al. [2024]</span>
<span class="ltx_bibblock">
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker.

</span>
<span class="ltx_bibblock">Back to basics: Revisiting reinforce style optimization for learning from human feedback in LLMs.

</span>
<span class="ltx_bibblock">2024, <a target="_blank" href="http://arxiv.org/abs/2402.14740" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2402.14740</a>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ainslie et al. [2023]</span>
<span class="ltx_bibblock">
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai.

</span>
<span class="ltx_bibblock">GQA: Training generalized multi-query transformer models from multi-head checkpoints.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</em>, pages 4895–4901, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.emnlp-main.298</span>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Apple [2023]</span>
<span class="ltx_bibblock">
Apple.

</span>
<span class="ltx_bibblock">The AXLearn library for deep learning.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/apple/axlearn" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/apple/axlearn</a>, 2023.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Apple [2024a]</span>
<span class="ltx_bibblock">
Apple.

</span>
<span class="ltx_bibblock">Applebot description.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://support.apple.com/en-us/HT204683" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://support.apple.com/en-us/HT204683</a>, 2024a.

</span>
<span class="ltx_bibblock">Accessed: 2024-05-04.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Apple [2024b]</span>
<span class="ltx_bibblock">
Apple.

</span>
<span class="ltx_bibblock">Private cloud compute: A new frontier for ai privacy in the cloud.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://security.apple.com/blog/private-cloud-compute/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://security.apple.com/blog/private-cloud-compute/</a>, 2024b.

</span>
<span class="ltx_bibblock">Accessed: 2024-07-11.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Azar et al. [2024]</span>
<span class="ltx_bibblock">
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello.

</span>
<span class="ltx_bibblock">A general theoretical paradigm to understand learning from human preferences.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">International Conference on Artificial Intelligence and Statistics</em>, pages 4447–4455. PMLR, 2024, <a target="_blank" href="http://arxiv.org/abs/2310.12036" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2310.12036</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bradbury et al. [2018]</span>
<span class="ltx_bibblock">
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.

</span>
<span class="ltx_bibblock">JAX: composable transformations of Python+NumPy programs, 2018.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://github.com/google/jax" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://github.com/google/jax</a>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bradley and Terry [1952]</span>
<span class="ltx_bibblock">
Ralph Allan Bradley and Milton E Terry.

</span>
<span class="ltx_bibblock">Rank analysis of incomplete block designs: I. the method of paired comparisons.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Biometrika</em>, 39(3/4):324–345, 1952.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. [2022]</span>
<span class="ltx_bibblock">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock">2022, <a target="_blank" href="http://arxiv.org/abs/2204.02311" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2204.02311</a>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chung et al. [2024]</span>
<span class="ltx_bibblock">
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.

</span>
<span class="ltx_bibblock">Scaling instruction-finetuned language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 25(70):1–53, 2024, <a target="_blank" href="http://arxiv.org/abs/2210.11416" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2210.11416</a>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. [2021]</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems.

</span>
<span class="ltx_bibblock">2021, <a target="_blank" href="http://arxiv.org/abs/2110.14168" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2110.14168</a>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. [2024]</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized LLMs.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024, <a target="_blank" href="http://arxiv.org/abs/2305.14314" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2305.14314</a>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dubois et al. [2024]</span>
<span class="ltx_bibblock">
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto.

</span>
<span class="ltx_bibblock">Length-controlled alpacaeval: A simple way to debias automatic evaluators.

</span>
<span class="ltx_bibblock">2024, <a target="_blank" href="http://arxiv.org/abs/2404.04475" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2404.04475</a>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Frantar et al. [2022]</span>
<span class="ltx_bibblock">
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.

</span>
<span class="ltx_bibblock">Gptq: Accurate post-training quantization for generative pre-trained transformers.

</span>
<span class="ltx_bibblock">2022, <a target="_blank" href="http://arxiv.org/abs/2210.17323" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2210.17323</a>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ganguli et al. [2022]</span>
<span class="ltx_bibblock">
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al.

</span>
<span class="ltx_bibblock">Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.

</span>
<span class="ltx_bibblock">2022, <a target="_blank" href="http://arxiv.org/abs/2209.07858" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2209.07858</a>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. [2021]</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring mathematical problem solving with the math dataset.

</span>
<span class="ltx_bibblock">2021, <a target="_blank" href="http://arxiv.org/abs/2103.03874" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2103.03874</a>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton [2012]</span>
<span class="ltx_bibblock">
Geoffrey Hinton.

</span>
<span class="ltx_bibblock">Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">COURSERA: Neural networks for machine learning</em>, 4(2):26, 2012.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton et al. [2015]</span>
<span class="ltx_bibblock">
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.

</span>
<span class="ltx_bibblock">Distilling the knowledge in a neural network.

</span>
<span class="ltx_bibblock">2015, <a target="_blank" href="http://arxiv.org/abs/1503.02531" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:1503.02531</a>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hohman et al. [2024]</span>
<span class="ltx_bibblock">
Fred Hohman, Chaoqun Wang, Jinmook Lee, Jochen Görtler, Dominik Moritz, Jeffrey P Bigham, Zhile Ren, Cecile Foret, Qi Shan, and Xiaoyi Zhang.

</span>
<span class="ltx_bibblock">Talaria: Interactively optimizing machine learning models for efficient inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Proceedings of the CHI Conference on Human Factors in Computing Systems</em>, pages 1–19, 2024, <a target="_blank" href="http://arxiv.org/abs/2404.03085" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2404.03085</a>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hsieh et al. [2024]</span>
<span class="ltx_bibblock">
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg.

</span>
<span class="ltx_bibblock">Ruler: What’s the real context size of your long-context language models?

</span>
<span class="ltx_bibblock">2024, <a target="_blank" href="http://arxiv.org/abs/2404.06654" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2404.06654</a>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. [2021]</span>
<span class="ltx_bibblock">
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.

</span>
<span class="ltx_bibblock">Lora: Low-rank adaptation of large language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2021, <a target="_blank" href="http://arxiv.org/abs/2106.09685" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2106.09685</a>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huggingface [2024]</span>
<span class="ltx_bibblock">
Huggingface.

</span>
<span class="ltx_bibblock">Open LLM evaluation.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard</a>, 2024.

</span>
<span class="ltx_bibblock">Accessed: 2024-07-09.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Inan et al. [2023]</span>
<span class="ltx_bibblock">
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al.

</span>
<span class="ltx_bibblock">Llama guard: LLM-based input-output safeguard for human-ai conversations.

</span>
<span class="ltx_bibblock">2023, <a target="_blank" href="http://arxiv.org/abs/2312.06674" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2312.06674</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kohlschütter et al. [2010]</span>
<span class="ltx_bibblock">
Christian Kohlschütter, Peter Fankhauser, and Wolfgang Nejdl.

</span>
<span class="ltx_bibblock">Boilerplate detection using shallow text features.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Third ACM International Conference on Web Search and Data Mining</em>, WSDM ’10, page 441–450, New York, NY, USA, 2010. Association for Computing Machinery.

</span>
<span class="ltx_bibblock">ISBN 9781605588896.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1145/1718487.1718542</span>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kong et al. [2024]</span>
<span class="ltx_bibblock">
Xiang Kong, Tom Gunter, and Ruoming Pang.

</span>
<span class="ltx_bibblock">Large language model-guided document selection.

</span>
<span class="ltx_bibblock">2024, <a target="_blank" href="http://arxiv.org/abs/2406.04638" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2406.04638</a>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kool et al. [2019]</span>
<span class="ltx_bibblock">
Wouter Kool, Herke van Hoof, and Max Welling.

</span>
<span class="ltx_bibblock">Buy 4 REINFORCE samples, get a baseline for free!

</span>
<span class="ltx_bibblock">2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=r1lgTGL5DE" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=r1lgTGL5DE</a>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lazic et al. [2021]</span>
<span class="ltx_bibblock">
Nevena Lazic, Dong Yin, Yasin Abbasi-Yadkori, and Csaba Szepesvari.

</span>
<span class="ltx_bibblock">Improved regret bound and experience replay in regularized policy iteration.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pages 6032–6042. PMLR, 2021, <a target="_blank" href="http://arxiv.org/abs/2102.12611" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2102.12611</a>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lei et al. [2023]</span>
<span class="ltx_bibblock">
Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, et al.

</span>
<span class="ltx_bibblock">Conditional adapters: Parameter-efficient transfer learning with fast inference.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36:8152–8172, 2023, <a target="_blank" href="http://arxiv.org/abs/2304.04947" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2304.04947</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lewkowycz et al. [2022]</span>
<span class="ltx_bibblock">
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al.

</span>
<span class="ltx_bibblock">Solving quantitative reasoning problems with language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:3843–3857, 2022.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2024a]</span>
<span class="ltx_bibblock">
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar.

</span>
<span class="ltx_bibblock">Datacomp-lm: In search of the next generation of training sets for language models.

</span>
<span class="ltx_bibblock">2024a, <a target="_blank" href="http://arxiv.org/abs/2406.11794" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2406.11794</a>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2024b]</span>
<span class="ltx_bibblock">
Tianle* Li, Wei-Lin* Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica.

</span>
<span class="ltx_bibblock">From live data to high-quality benchmarks: The arena-hard pipeline.

</span>
<span class="ltx_bibblock">April 2024b.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://lmsys.org/blog/2024-04-19-arena-hard/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://lmsys.org/blog/2024-04-19-arena-hard/</a>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liang et al. [2023]</span>
<span class="ltx_bibblock">
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.

</span>
<span class="ltx_bibblock">Holistic evaluation of language models.

</span>
<span class="ltx_bibblock">2023, <a target="_blank" href="http://arxiv.org/abs/2211.09110" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2211.09110</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. [2024]</span>
<span class="ltx_bibblock">
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han.

</span>
<span class="ltx_bibblock">Awq: Activation-aware weight quantization for on-device llm compression and acceleration.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning and Systems</em>, 6:87–100, 2024, <a target="_blank" href="http://arxiv.org/abs/2306.00978" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2306.00978</a>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. [2024]</span>
<span class="ltx_bibblock">
Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin.

</span>
<span class="ltx_bibblock">Scaling laws of rope-based extrapolation.

</span>
<span class="ltx_bibblock">2024, <a target="_blank" href="http://arxiv.org/abs/2310.05209" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2310.05209</a>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Loshchilov and Hutter [2019]</span>
<span class="ltx_bibblock">
Ilya Loshchilov and Frank Hutter.

</span>
<span class="ltx_bibblock">Decoupled weight decay regularization.

</span>
<span class="ltx_bibblock">2019, <a target="_blank" href="http://arxiv.org/abs/1711.05101" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:1711.05101</a>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Louizos et al. [2018]</span>
<span class="ltx_bibblock">
Christos Louizos, Max Welling, and Diederik P. Kingma.

</span>
<span class="ltx_bibblock">Learning sparse neural networks through <math id="bib.bib38.1.m1.1" class="ltx_Math" alttext="l_{0}" display="inline"><semantics id="bib.bib38.1.m1.1a"><msub id="bib.bib38.1.m1.1.1" xref="bib.bib38.1.m1.1.1.cmml"><mi id="bib.bib38.1.m1.1.1.2" xref="bib.bib38.1.m1.1.1.2.cmml">l</mi><mn id="bib.bib38.1.m1.1.1.3" xref="bib.bib38.1.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="bib.bib38.1.m1.1b"><apply id="bib.bib38.1.m1.1.1.cmml" xref="bib.bib38.1.m1.1.1"><csymbol cd="ambiguous" id="bib.bib38.1.m1.1.1.1.cmml" xref="bib.bib38.1.m1.1.1">subscript</csymbol><ci id="bib.bib38.1.m1.1.1.2.cmml" xref="bib.bib38.1.m1.1.1.2">𝑙</ci><cn type="integer" id="bib.bib38.1.m1.1.1.3.cmml" xref="bib.bib38.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="bib.bib38.1.m1.1c">l_{0}</annotation></semantics></math> regularization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.2.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2018, <a target="_blank" href="http://arxiv.org/abs/1712.01312" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:1712.01312</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=H1Y8hhg0b" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=H1Y8hhg0b</a>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nguyen and Salazar [2019]</span>
<span class="ltx_bibblock">
Toan Q. Nguyen and Julian Salazar.

</span>
<span class="ltx_bibblock">Transformers without tears: Improving the normalization of self-attention.

</span>
<span class="ltx_bibblock">In Jan Niehues, Rolando Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico, editors, <em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 16th International Conference on Spoken Language Translation</em>, Hong Kong, November 2-3 2019. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2019.iwslt-1.17" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2019.iwslt-1.17</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ouyang et al. [2022]</span>
<span class="ltx_bibblock">
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.

</span>
<span class="ltx_bibblock">Training language models to follow instructions with human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:27730–27744, 2022, <a target="_blank" href="http://arxiv.org/abs/2203.02155" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2203.02155</a>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Patil et al. [2023]</span>
<span class="ltx_bibblock">
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez.

</span>
<span class="ltx_bibblock">Gorilla: Large language model connected with massive APIs.

</span>
<span class="ltx_bibblock">2023, <a target="_blank" href="http://arxiv.org/abs/2305.15334" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2305.15334</a>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Press and Wolf [2016]</span>
<span class="ltx_bibblock">
Ofir Press and Lior Wolf.

</span>
<span class="ltx_bibblock">Using the output embedding to improve language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Conference of the European Chapter of the Association for Computational Linguistics</em>, 2016, <a target="_blank" href="http://arxiv.org/abs/1608.05859" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:1608.05859</a>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rafailov et al. [2024]</span>
<span class="ltx_bibblock">
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.

</span>
<span class="ltx_bibblock">Direct preference optimization: Your language model is secretly a reward model.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024, <a target="_blank" href="http://arxiv.org/abs/2305.18290" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2305.18290</a>.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schulman et al. [2015]</span>
<span class="ltx_bibblock">
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.

</span>
<span class="ltx_bibblock">Trust region policy optimization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pages 1889–1897. PMLR, 2015, <a target="_blank" href="http://arxiv.org/abs/1502.05477" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:1502.05477</a>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schulman et al. [2017]</span>
<span class="ltx_bibblock">
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.

</span>
<span class="ltx_bibblock">Proximal policy optimization algorithms.

</span>
<span class="ltx_bibblock">2017, <a target="_blank" href="http://arxiv.org/abs/1707.06347" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:1707.06347</a>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer [2020]</span>
<span class="ltx_bibblock">
Noam Shazeer.

</span>
<span class="ltx_bibblock">Glu variants improve transformer, 2020, <a target="_blank" href="http://arxiv.org/abs/2002.05202" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2002.05202</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2002.05202" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2002.05202</a>.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Stanford [2024]</span>
<span class="ltx_bibblock">
Stanford.

</span>
<span class="ltx_bibblock">HELM Lite: Lightweight and broad capabilities evaluation.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://crfm.stanford.edu/helm/lite/v1.5.0/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://crfm.stanford.edu/helm/lite/v1.5.0/</a>, 2024.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Su et al. [2024]</span>
<span class="ltx_bibblock">
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.

</span>
<span class="ltx_bibblock">Roformer: Enhanced transformer with rotary position embedding.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Neurocomputing</em>, 568:127063, 2024.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Taori et al. [2023]</span>
<span class="ltx_bibblock">
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.

</span>
<span class="ltx_bibblock">Stanford alpaca: An instruction-following llama model.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/stanford_alpaca" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/stanford_alpaca</a>, 2023.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tomar et al. [2020]</span>
<span class="ltx_bibblock">
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh.

</span>
<span class="ltx_bibblock">Mirror descent policy optimization.

</span>
<span class="ltx_bibblock">2020, <a target="_blank" href="http://arxiv.org/abs/2005.09814" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2005.09814</a>.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. [2023]</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock">2023, <a target="_blank" href="http://arxiv.org/abs/2307.09288" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2307.09288</a>.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. [2017]</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock">In <em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 2017, <a target="_blank" href="http://arxiv.org/abs/1706.03762" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:1706.03762</a>.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2020]</span>
<span class="ltx_bibblock">
Ziheng Wang, Jeremy Wohlwend, and Tao Lei.

</span>
<span class="ltx_bibblock">Structured pruning of large language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)</em>, pages 6151–6162, 2020, <a target="_blank" href="http://arxiv.org/abs/1910.04732" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:1910.04732</a>.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2020.emnlp-main.496</span>.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. [2022]</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.

</span>
<span class="ltx_bibblock">Chain-of-thought prompting elicits reasoning in large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:24824–24837, 2022.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Williams [1992]</span>
<span class="ltx_bibblock">
Ronald J Williams.

</span>
<span class="ltx_bibblock">Simple statistical gradient-following algorithms for connectionist reinforcement learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">Machine Learning</em>, 8:229–256, 1992.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wortsman et al. [2023]</span>
<span class="ltx_bibblock">
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith.

</span>
<span class="ltx_bibblock">Small-scale proxies for large-scale transformer training instabilities.

</span>
<span class="ltx_bibblock">2023, <a target="_blank" href="http://arxiv.org/abs/2309.14322" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2309.14322</a>.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xia et al. [2023]</span>
<span class="ltx_bibblock">
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen.

</span>
<span class="ltx_bibblock">Sheared llama: Accelerating language model pre-training via structured pruning.

</span>
<span class="ltx_bibblock">2023, <a target="_blank" href="http://arxiv.org/abs/2310.06694" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2310.06694</a>.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. [2023]</span>
<span class="ltx_bibblock">
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.

</span>
<span class="ltx_bibblock">WizardLM: Empowering large language models to follow complex instructions.

</span>
<span class="ltx_bibblock">2023, <a target="_blank" href="http://arxiv.org/abs/2304.12244" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2304.12244</a>.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. [2022]</span>
<span class="ltx_bibblock">
Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao.

</span>
<span class="ltx_bibblock">Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.

</span>
<span class="ltx_bibblock">2022, <a target="_blank" href="http://arxiv.org/abs/2203.03466" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2203.03466</a>.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. [2023]</span>
<span class="ltx_bibblock">
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.

</span>
<span class="ltx_bibblock">Metamath: Bootstrap your own mathematical questions for large language models.

</span>
<span class="ltx_bibblock">2023, <a target="_blank" href="http://arxiv.org/abs/2309.12284" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2309.12284</a>.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zeng et al. [2024]</span>
<span class="ltx_bibblock">
Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi.

</span>
<span class="ltx_bibblock">How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms, 2024, <a target="_blank" href="http://arxiv.org/abs/2401.06373" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2401.06373</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2401.06373" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2401.06373</a>.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang and Sennrich [2019]</span>
<span class="ltx_bibblock">
Biao Zhang and Rico Sennrich.

</span>
<span class="ltx_bibblock">Root mean square layer normalization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib62.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf</a>.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zheng et al. [2024]</span>
<span class="ltx_bibblock">
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.

</span>
<span class="ltx_bibblock">Judging LLM-as-a-judge with MT-bench and chatbot arena.

</span>
<span class="ltx_bibblock"><em id="bib.bib63.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. [2024]</span>
<span class="ltx_bibblock">
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al.

</span>
<span class="ltx_bibblock">LIMA: Less is more for alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib64.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. [2023]</span>
<span class="ltx_bibblock">
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou.

</span>
<span class="ltx_bibblock">Instruction-following evaluation for large language models.

</span>
<span class="ltx_bibblock">2023, <a target="_blank" href="http://arxiv.org/abs/2311.07911" title="" class="ltx_ref ltx_href ltx_font_typewriter">arXiv:2311.07911</a>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_font_sansserif ltx_title_section">Contributors</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">Within each section, contributors are listed in alphabetical order by first name.</p>
</div>
<section id="Sx1.SSx1" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_title_subsection">Foundation Models</h3>

<div id="Sx1.SSx1.p1" class="ltx_para">
<p id="Sx1.SSx1.p1.1" class="ltx_p">Andy Narayanan,
Aonan Zhang,
Bowen Zhang,
Chen Chen,
Chong Wang (inference efficiency lead),
Chung-Cheng Chiu,
David Qiu,
Deepak Gopinath,
Dian Ang Yap,
Dong Yin,
Feng Nan,
Floris Weers,
Guoli Yin,
Haoshuo Huang,
Jianyu Wang,
Jiarui Lu,
John Peebles,
Ke Ye,
Mark Lee,
Nan Du,
Qibin Chen,
Quentin Keunebroek,
Ruoming Pang (overall lead),
Sam Wiseman,
Syd Evans,
Tao Lei,
Tom Gunter (pre-train lead),
Vivek Rathod,
Xiang Kong,
Xianzhi Du,
Yanghao Li,
Yongqiang Wang,
Yuan Gao,
Zaid Ahmed,
Zhaoyang Xu,
Zhiyun Lu,
Zirui Wang (post-train lead)</p>
</div>
</section>
<section id="Sx1.SSx2" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_title_subsection">Data, Evaluation, and Responsible AI</h3>

<div id="Sx1.SSx2.p1" class="ltx_para">
<p id="Sx1.SSx2.p1.1" class="ltx_p">Al Rashid,
Albin Madappally Jose,
Alec Doane,
Alfredo Bencomo,
Allison Vanderby,
Andrew Hansen,
Ankur Jain,
Anupama Mann Anupama,
Areeba Kamal,
Bugu Wu,
Carolina Brum,
Charlie Maalouf,
Chinguun Erdenebileg,
Chris Dulhanty,
Dominik Moritz,
Doug Kang,
Eduardo Jimenez,
Evan Ladd,
Fangping Shi,
Felix Bai,
Frank Chu,
Fred Hohman,
Hadas Kotek,
Hannah Gillis Coleman,
Jane Li,
Jeffrey Bigham,
Jeffery Cao,
Jeff Lai,
Jessica Cheung,
Jiulong Shan,
Joe Zhou,
John Li,
Jun Qin,
Karanjeet Singh,
Karla Vega,
Ke Ye,
Kelvin Zou,
Laura Heckman,
Lauren Gardiner,
Margit Bowler,
Mark Lee,
Maria Cordell,
Meng Cao,
Nicole Hay,
Nilesh Shahdadpuri,
Otto Godwin,
Pranay Dighe,
Pushyami Rachapudi,
Ramsey Tantawi,
Roman Frigg,
Sam Davarnia,
Sanskruti Shah,
Saptarshi Guha,
Sasha Sirovica,
Shen Ma,
Shuang Ma,
Simon Wang,
Sulgi Kim,
Suma Jayaram,
Vaishaal Shankar,
Varsha Paidi,
Vivek Kumar,
Xiang Kong,
Xin Wang,
Xin Zheng,
Walker Cheng,
Yael Shrager,
Yang Ye,
Yasu Tanaka,
Yihao Guo,
Yunsong Meng,
Zhao Tang Luo,
Zhi Ouyang,
Zhiyun Lu
</p>
</div>
</section>
<section id="Sx1.SSx3" class="ltx_subsection">
<h3 class="ltx_title ltx_font_sansserif ltx_title_subsection">Adapters, Optimizations, and Summarization</h3>

<div id="Sx1.SSx3.p1" class="ltx_para">
<p id="Sx1.SSx3.p1.1" class="ltx_p">Alp Aygar,
Alvin Wan,
Andrew Walkingshaw,
Andy Narayanan,
Antonie Lin,
Arsalan Farooq,
Brent Ramerth,
Chong Wang,
Colorado Reed,
Chris Bartels,
Chris Chaney,
David Riazati,
Eric Liang Yang,
Erin Feldman,
Gabriel Hochstrasser,
Guillaume Seguin,
Guoli Yin,
Irina Belousova,
Jianyu Wang,
Joris Pelemans,
Karen Yang,
Keivan Alizadeh Vahid,
Liangliang Cao,
Mahyar Najibi ,
Marco Zuliani,
Max Horton,
Minsik Cho,
Nikhil Bhendawade,
Patrick Dong,
Piotr Maj,
Pulkit Agrawal,
Qi Shan,
Qibin Chen,
Qichen Fu,
Regan Poston,
Sam Xu,
Shuangning Liu,
Sushma Rao,
Tashweena Heeramun,
Thomas Merth,
Uday Rayala,
Victor Cui,
Vivek Rangarajan Sridhar,
Vivek Rathod,
Wencong Zhang,
Wenqi Zhang,
Wentao Wu,
Xiang Kong,
Xingyu Zhou,
Xinwen Liu,
Yang Zhao,
Yin Xia,
Zhile Ren,
Zhongzheng Ren
</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="Ax1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">Appendix</h2>

</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Core pre-training recipe ablation</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.5" class="ltx_p">We compare our chosen settings for ‘core’ pre-training from <a href="#S3.SS2.SSS1" title="3.2.1 Core pre-training ‣ 3.2 Recipe ‣ 3 Pre-training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Section 3.2.1</span></a> (optimizer, scaling-law-predicted batch-size, weight-decay, etc.) to a baseline based on <cite class="ltx_cite ltx_citemacro_citep">[Wortsman et al., <a href="#bib.bib56" title="" class="ltx_ref">2023</a>]</cite>. In particular, the baseline uses AdamW with a standard hyperparameter configuration of <math id="A1.p1.1.m1.1" class="ltx_Math" alttext="\beta_{1}=0.9" display="inline"><semantics id="A1.p1.1.m1.1a"><mrow id="A1.p1.1.m1.1.1" xref="A1.p1.1.m1.1.1.cmml"><msub id="A1.p1.1.m1.1.1.2" xref="A1.p1.1.m1.1.1.2.cmml"><mi id="A1.p1.1.m1.1.1.2.2" xref="A1.p1.1.m1.1.1.2.2.cmml">β</mi><mn id="A1.p1.1.m1.1.1.2.3" xref="A1.p1.1.m1.1.1.2.3.cmml">1</mn></msub><mo id="A1.p1.1.m1.1.1.1" xref="A1.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.p1.1.m1.1.1.3" xref="A1.p1.1.m1.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.1.m1.1b"><apply id="A1.p1.1.m1.1.1.cmml" xref="A1.p1.1.m1.1.1"><eq id="A1.p1.1.m1.1.1.1.cmml" xref="A1.p1.1.m1.1.1.1"></eq><apply id="A1.p1.1.m1.1.1.2.cmml" xref="A1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.p1.1.m1.1.1.2.1.cmml" xref="A1.p1.1.m1.1.1.2">subscript</csymbol><ci id="A1.p1.1.m1.1.1.2.2.cmml" xref="A1.p1.1.m1.1.1.2.2">𝛽</ci><cn type="integer" id="A1.p1.1.m1.1.1.2.3.cmml" xref="A1.p1.1.m1.1.1.2.3">1</cn></apply><cn type="float" id="A1.p1.1.m1.1.1.3.cmml" xref="A1.p1.1.m1.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.1.m1.1c">\beta_{1}=0.9</annotation></semantics></math>, <math id="A1.p1.2.m2.1" class="ltx_Math" alttext="\beta_{2}=0.95" display="inline"><semantics id="A1.p1.2.m2.1a"><mrow id="A1.p1.2.m2.1.1" xref="A1.p1.2.m2.1.1.cmml"><msub id="A1.p1.2.m2.1.1.2" xref="A1.p1.2.m2.1.1.2.cmml"><mi id="A1.p1.2.m2.1.1.2.2" xref="A1.p1.2.m2.1.1.2.2.cmml">β</mi><mn id="A1.p1.2.m2.1.1.2.3" xref="A1.p1.2.m2.1.1.2.3.cmml">2</mn></msub><mo id="A1.p1.2.m2.1.1.1" xref="A1.p1.2.m2.1.1.1.cmml">=</mo><mn id="A1.p1.2.m2.1.1.3" xref="A1.p1.2.m2.1.1.3.cmml">0.95</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.2.m2.1b"><apply id="A1.p1.2.m2.1.1.cmml" xref="A1.p1.2.m2.1.1"><eq id="A1.p1.2.m2.1.1.1.cmml" xref="A1.p1.2.m2.1.1.1"></eq><apply id="A1.p1.2.m2.1.1.2.cmml" xref="A1.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A1.p1.2.m2.1.1.2.1.cmml" xref="A1.p1.2.m2.1.1.2">subscript</csymbol><ci id="A1.p1.2.m2.1.1.2.2.cmml" xref="A1.p1.2.m2.1.1.2.2">𝛽</ci><cn type="integer" id="A1.p1.2.m2.1.1.2.3.cmml" xref="A1.p1.2.m2.1.1.2.3">2</cn></apply><cn type="float" id="A1.p1.2.m2.1.1.3.cmml" xref="A1.p1.2.m2.1.1.3">0.95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.2.m2.1c">\beta_{2}=0.95</annotation></semantics></math>, <math id="A1.p1.3.m3.1" class="ltx_Math" alttext="\epsilon=1\mathrm{e}{-15}" display="inline"><semantics id="A1.p1.3.m3.1a"><mrow id="A1.p1.3.m3.1.1" xref="A1.p1.3.m3.1.1.cmml"><mi id="A1.p1.3.m3.1.1.2" xref="A1.p1.3.m3.1.1.2.cmml">ϵ</mi><mo id="A1.p1.3.m3.1.1.1" xref="A1.p1.3.m3.1.1.1.cmml">=</mo><mrow id="A1.p1.3.m3.1.1.3" xref="A1.p1.3.m3.1.1.3.cmml"><mrow id="A1.p1.3.m3.1.1.3.2" xref="A1.p1.3.m3.1.1.3.2.cmml"><mn id="A1.p1.3.m3.1.1.3.2.2" xref="A1.p1.3.m3.1.1.3.2.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="A1.p1.3.m3.1.1.3.2.1" xref="A1.p1.3.m3.1.1.3.2.1.cmml">​</mo><mi mathvariant="normal" id="A1.p1.3.m3.1.1.3.2.3" xref="A1.p1.3.m3.1.1.3.2.3.cmml">e</mi></mrow><mo id="A1.p1.3.m3.1.1.3.1" xref="A1.p1.3.m3.1.1.3.1.cmml">−</mo><mn id="A1.p1.3.m3.1.1.3.3" xref="A1.p1.3.m3.1.1.3.3.cmml">15</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.3.m3.1b"><apply id="A1.p1.3.m3.1.1.cmml" xref="A1.p1.3.m3.1.1"><eq id="A1.p1.3.m3.1.1.1.cmml" xref="A1.p1.3.m3.1.1.1"></eq><ci id="A1.p1.3.m3.1.1.2.cmml" xref="A1.p1.3.m3.1.1.2">italic-ϵ</ci><apply id="A1.p1.3.m3.1.1.3.cmml" xref="A1.p1.3.m3.1.1.3"><minus id="A1.p1.3.m3.1.1.3.1.cmml" xref="A1.p1.3.m3.1.1.3.1"></minus><apply id="A1.p1.3.m3.1.1.3.2.cmml" xref="A1.p1.3.m3.1.1.3.2"><times id="A1.p1.3.m3.1.1.3.2.1.cmml" xref="A1.p1.3.m3.1.1.3.2.1"></times><cn type="integer" id="A1.p1.3.m3.1.1.3.2.2.cmml" xref="A1.p1.3.m3.1.1.3.2.2">1</cn><ci id="A1.p1.3.m3.1.1.3.2.3.cmml" xref="A1.p1.3.m3.1.1.3.2.3">e</ci></apply><cn type="integer" id="A1.p1.3.m3.1.1.3.3.cmml" xref="A1.p1.3.m3.1.1.3.3">15</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.3.m3.1c">\epsilon=1\mathrm{e}{-15}</annotation></semantics></math>, and a decoupled weight decay of <math id="A1.p1.4.m4.1" class="ltx_Math" alttext="1\mathrm{e}{-4}" display="inline"><semantics id="A1.p1.4.m4.1a"><mrow id="A1.p1.4.m4.1.1" xref="A1.p1.4.m4.1.1.cmml"><mrow id="A1.p1.4.m4.1.1.2" xref="A1.p1.4.m4.1.1.2.cmml"><mn id="A1.p1.4.m4.1.1.2.2" xref="A1.p1.4.m4.1.1.2.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="A1.p1.4.m4.1.1.2.1" xref="A1.p1.4.m4.1.1.2.1.cmml">​</mo><mi mathvariant="normal" id="A1.p1.4.m4.1.1.2.3" xref="A1.p1.4.m4.1.1.2.3.cmml">e</mi></mrow><mo id="A1.p1.4.m4.1.1.1" xref="A1.p1.4.m4.1.1.1.cmml">−</mo><mn id="A1.p1.4.m4.1.1.3" xref="A1.p1.4.m4.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.4.m4.1b"><apply id="A1.p1.4.m4.1.1.cmml" xref="A1.p1.4.m4.1.1"><minus id="A1.p1.4.m4.1.1.1.cmml" xref="A1.p1.4.m4.1.1.1"></minus><apply id="A1.p1.4.m4.1.1.2.cmml" xref="A1.p1.4.m4.1.1.2"><times id="A1.p1.4.m4.1.1.2.1.cmml" xref="A1.p1.4.m4.1.1.2.1"></times><cn type="integer" id="A1.p1.4.m4.1.1.2.2.cmml" xref="A1.p1.4.m4.1.1.2.2">1</cn><ci id="A1.p1.4.m4.1.1.2.3.cmml" xref="A1.p1.4.m4.1.1.2.3">e</ci></apply><cn type="integer" id="A1.p1.4.m4.1.1.3.cmml" xref="A1.p1.4.m4.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.4.m4.1c">1\mathrm{e}{-4}</annotation></semantics></math>, decaying the learning rate to <math id="A1.p1.5.m5.1" class="ltx_Math" alttext="0.0001" display="inline"><semantics id="A1.p1.5.m5.1a"><mn id="A1.p1.5.m5.1.1" xref="A1.p1.5.m5.1.1.cmml">0.0001</mn><annotation-xml encoding="MathML-Content" id="A1.p1.5.m5.1b"><cn type="float" id="A1.p1.5.m5.1.1.cmml" xref="A1.p1.5.m5.1.1">0.0001</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.5.m5.1c">0.0001</annotation></semantics></math> of peak, with a batch size of 1024 sequences. Otherwise both recipes are identical. Training covers 3.1T tokens using the AFM-on-device architecture but with a different data mixture to that used by the official AFM training runs.</p>
</div>
<figure id="A1.T5" class="ltx_table">
<table id="A1.T5.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A1.T5.1.1" class="ltx_tr">
<td id="A1.T5.1.1.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.1.1.1" class="ltx_text ltx_font_bold">Task</span></td>
<td id="A1.T5.1.1.2" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.1.2.1" class="ltx_text ltx_font_bold">Baseline (acc)</span></td>
<td id="A1.T5.1.1.3" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.1.3.1" class="ltx_text ltx_font_bold">AFM (acc)</span></td>
</tr>
<tr id="A1.T5.1.2" class="ltx_tr">
<td id="A1.T5.1.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.2.1.1" class="ltx_text ltx_font_bold">arc_challenge</span></td>
<td id="A1.T5.1.2.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">41.9</td>
<td id="A1.T5.1.2.3" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">44.6</td>
</tr>
<tr id="A1.T5.1.3" class="ltx_tr">
<td id="A1.T5.1.3.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.3.1.1" class="ltx_text ltx_font_bold">arc_easy</span></td>
<td id="A1.T5.1.3.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">75.6</td>
<td id="A1.T5.1.3.3" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">76.1</td>
</tr>
<tr id="A1.T5.1.4" class="ltx_tr">
<td id="A1.T5.1.4.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.4.1.1" class="ltx_text ltx_font_bold">hellaswag</span></td>
<td id="A1.T5.1.4.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">54.3</td>
<td id="A1.T5.1.4.3" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">55.0</td>
</tr>
<tr id="A1.T5.1.5" class="ltx_tr">
<td id="A1.T5.1.5.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.5.1.1" class="ltx_text ltx_font_bold">lambada</span></td>
<td id="A1.T5.1.5.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">69.3</td>
<td id="A1.T5.1.5.3" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">68.9</td>
</tr>
<tr id="A1.T5.1.6" class="ltx_tr">
<td id="A1.T5.1.6.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.6.1.1" class="ltx_text ltx_font_bold">piqa</span></td>
<td id="A1.T5.1.6.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">78.3</td>
<td id="A1.T5.1.6.3" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">78.4</td>
</tr>
<tr id="A1.T5.1.7" class="ltx_tr">
<td id="A1.T5.1.7.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.7.1.1" class="ltx_text ltx_font_bold">sciq</span></td>
<td id="A1.T5.1.7.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">94.5</td>
<td id="A1.T5.1.7.3" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">94.7</td>
</tr>
<tr id="A1.T5.1.8" class="ltx_tr">
<td id="A1.T5.1.8.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.8.1.1" class="ltx_text ltx_font_bold">winogrande</span></td>
<td id="A1.T5.1.8.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">67.3</td>
<td id="A1.T5.1.8.3" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">66.9</td>
</tr>
<tr id="A1.T5.1.9" class="ltx_tr">
<td id="A1.T5.1.9.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.9.1.1" class="ltx_text ltx_font_bold">triviaqa (1-shot)</span></td>
<td id="A1.T5.1.9.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">40.5</td>
<td id="A1.T5.1.9.3" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">41.0</td>
</tr>
<tr id="A1.T5.1.10" class="ltx_tr">
<td id="A1.T5.1.10.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.10.1.1" class="ltx_text ltx_font_bold">webqs (1-shot)</span></td>
<td id="A1.T5.1.10.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">20.6</td>
<td id="A1.T5.1.10.3" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">20.6</td>
</tr>
<tr id="A1.T5.1.11" class="ltx_tr">
<td id="A1.T5.1.11.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.11.1.1" class="ltx_text ltx_font_bold">CoreEN average</span></td>
<td id="A1.T5.1.11.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">60.2</td>
<td id="A1.T5.1.11.3" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">60.7</td>
</tr>
<tr id="A1.T5.1.12" class="ltx_tr">
<td id="A1.T5.1.12.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.12.1.1" class="ltx_text ltx_font_bold">GSM8K (8-shot CoT)</span></td>
<td id="A1.T5.1.12.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">16.6</td>
<td id="A1.T5.1.12.3" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">18.9</td>
</tr>
<tr id="A1.T5.1.13" class="ltx_tr">
<td id="A1.T5.1.13.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="A1.T5.1.13.1.1" class="ltx_text ltx_font_bold">MMLU (5-shot)</span></td>
<td id="A1.T5.1.13.2" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">45.4</td>
<td id="A1.T5.1.13.3" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">45.5</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Core pre-training recipe ablation few-shot results. Unless otherwise noted, we use 0-shot prompts. We note that AFM’s recipe allows for slight improvements across the majority of tasks, although the difference is typically very small. The data mixture differs from the official AFM runs.</figcaption>
</figure>
<div id="A1.p2" class="ltx_para">
<p id="A1.p2.1" class="ltx_p">In <a href="#A1.T5" title="Table 5 ‣ Appendix A Core pre-training recipe ablation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 5</span></a>, AFM’s recipe demonstrates a slight improvement over the baseline. This likely indicates that the most important recipe settings are already well-enough configured by the baseline for this model size and training budget.</p>
</div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Ablations on pruning and distillation</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">Here we detail the evaluation results of using structural pruning and distillation separately and show they can be combined together to get the best performance.</p>
</div>
<div id="A2.p2" class="ltx_para">
<p id="A2.p2.1" class="ltx_p"><a href="#A2.T6" title="Table 6 ‣ Appendix B Ablations on pruning and distillation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 6</span></a> shows the ablation results of training 3B models using an early version of our pre-training data mixture.
As shown in the table, both pruning and distillation methods can outperform a baseline model trained from scratch. For example, pruning and distillation achieve a MMLU score of 42.9% and 44.9% respectively, whereas a baseline using 50% more steps gets 34.6%.
It is also interesting that pruning achieves a higher score on the CoreEn benchmark, while distillation is better on MMLU.
Finally, when combining these two methods together, we observe further improvements on MMLU and GSM8k by a large margin, getting better or on par results compared to the baseline trained using 5<math id="A2.p2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A2.p2.1.m1.1a"><mo id="A2.p2.1.m1.1.1" xref="A2.p2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A2.p2.1.m1.1b"><times id="A2.p2.1.m1.1.1.cmml" xref="A2.p2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A2.p2.1.m1.1c">\times</annotation></semantics></math> more computation.</p>
</div>
<figure id="A2.T6" class="ltx_table">
<table id="A2.T6.5" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T6.5.6" class="ltx_tr">
<td id="A2.T6.5.6.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.6.1.1" class="ltx_text ltx_font_bold">Metric/Method</span></td>
<td id="A2.T6.5.6.2" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.6.2.1" class="ltx_text ltx_font_bold">Baseline</span></td>
<td id="A2.T6.5.6.3" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.6.3.1" class="ltx_text ltx_font_bold">Prune</span></td>
<td id="A2.T6.5.6.4" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.6.4.1" class="ltx_text ltx_font_bold">Distill</span></td>
<td id="A2.T6.5.6.5" class="ltx_td ltx_align_right ltx_border_r ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.6.5.1" class="ltx_text ltx_font_bold">Both</span></td>
<td id="A2.T6.5.6.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.6.6.1" class="ltx_text ltx_font_bold">Baseline</span></td>
</tr>
<tr id="A2.T6.5.5" class="ltx_tr">
<td id="A2.T6.5.5.6" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.5.6.1" class="ltx_text ltx_font_bold">Training cost</span></td>
<td id="A2.T6.1.1.1" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">1.5<math id="A2.T6.1.1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A2.T6.1.1.1.m1.1a"><mo id="A2.T6.1.1.1.m1.1.1" xref="A2.T6.1.1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A2.T6.1.1.1.m1.1b"><times id="A2.T6.1.1.1.m1.1.1.cmml" xref="A2.T6.1.1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A2.T6.1.1.1.m1.1c">\times</annotation></semantics></math>
</td>
<td id="A2.T6.2.2.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">1<math id="A2.T6.2.2.2.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A2.T6.2.2.2.m1.1a"><mo id="A2.T6.2.2.2.m1.1.1" xref="A2.T6.2.2.2.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A2.T6.2.2.2.m1.1b"><times id="A2.T6.2.2.2.m1.1.1.cmml" xref="A2.T6.2.2.2.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A2.T6.2.2.2.m1.1c">\times</annotation></semantics></math>
</td>
<td id="A2.T6.3.3.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">1<math id="A2.T6.3.3.3.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A2.T6.3.3.3.m1.1a"><mo id="A2.T6.3.3.3.m1.1.1" xref="A2.T6.3.3.3.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A2.T6.3.3.3.m1.1b"><times id="A2.T6.3.3.3.m1.1.1.cmml" xref="A2.T6.3.3.3.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A2.T6.3.3.3.m1.1c">\times</annotation></semantics></math>
</td>
<td id="A2.T6.4.4.4" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">1<math id="A2.T6.4.4.4.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A2.T6.4.4.4.m1.1a"><mo id="A2.T6.4.4.4.m1.1.1" xref="A2.T6.4.4.4.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A2.T6.4.4.4.m1.1b"><times id="A2.T6.4.4.4.m1.1.1.cmml" xref="A2.T6.4.4.4.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A2.T6.4.4.4.m1.1c">\times</annotation></semantics></math>
</td>
<td id="A2.T6.5.5.5" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">5<math id="A2.T6.5.5.5.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="A2.T6.5.5.5.m1.1a"><mo id="A2.T6.5.5.5.m1.1.1" xref="A2.T6.5.5.5.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="A2.T6.5.5.5.m1.1b"><times id="A2.T6.5.5.5.m1.1.1.cmml" xref="A2.T6.5.5.5.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="A2.T6.5.5.5.m1.1c">\times</annotation></semantics></math>
</td>
</tr>
<tr id="A2.T6.5.7" class="ltx_tr">
<td id="A2.T6.5.7.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.7.1.1" class="ltx_text ltx_font_bold">MMLU (5-shot)</span></td>
<td id="A2.T6.5.7.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">34.6</td>
<td id="A2.T6.5.7.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">42.9</td>
<td id="A2.T6.5.7.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">44.9</td>
<td id="A2.T6.5.7.5" class="ltx_td ltx_align_right ltx_border_r ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.7.5.1" class="ltx_text ltx_font_bold">49.3</span></td>
<td id="A2.T6.5.7.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">45.4</td>
</tr>
<tr id="A2.T6.5.8" class="ltx_tr">
<td id="A2.T6.5.8.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.8.1.1" class="ltx_text ltx_font_bold">GSM8K (8-shot CoT)</span></td>
<td id="A2.T6.5.8.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">12.7</td>
<td id="A2.T6.5.8.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">13.5</td>
<td id="A2.T6.5.8.4" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">11.0</td>
<td id="A2.T6.5.8.5" class="ltx_td ltx_align_right ltx_border_r" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.8.5.1" class="ltx_text ltx_font_bold">16.8</span></td>
<td id="A2.T6.5.8.6" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.8.6.1" class="ltx_text ltx_font_bold">16.9</span></td>
</tr>
<tr id="A2.T6.5.9" class="ltx_tr">
<td id="A2.T6.5.9.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.9.1.1" class="ltx_text ltx_font_bold">CoreEN Average</span></td>
<td id="A2.T6.5.9.2" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">59.8</td>
<td id="A2.T6.5.9.3" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="A2.T6.5.9.3.1" class="ltx_text ltx_font_bold">61.0</span></td>
<td id="A2.T6.5.9.4" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">58.1</td>
<td id="A2.T6.5.9.5" class="ltx_td ltx_align_right ltx_border_bb ltx_border_r" style="padding-top:1pt;padding-bottom:1pt;">59.7</td>
<td id="A2.T6.5.9.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">60.3</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Ablation results of pruning and distillation methods. The training data is an early version that differs from the official AFM runs.</figcaption>
</figure>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Pre-training stage-breakdown evaluations</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">We present few-shot evaluation results after core, continued, and long-context pre-training stages, for a subset of evaluation metrics that we find to be low-variance, diverse, and correlated with downstream evaluation after post-training. These metrics are derived using an internal harness and set of benchmark formulations, which are not optimized for absolute performance (e.g. we do not apply length normalization, and use more difficult test splits where available—for TriviaQA as one example). They are therefore not suitable for comparison with other published results.</p>
</div>
<figure id="A3.T7" class="ltx_table">
<table id="A3.T7.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A3.T7.1.1" class="ltx_tr">
<td id="A3.T7.1.1.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.1.1.1" class="ltx_text ltx_font_bold">AFM-on-device</span></td>
<td id="A3.T7.1.1.2" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.1.2.1" class="ltx_text ltx_font_bold">Core</span></td>
<td id="A3.T7.1.1.3" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.1.3.1" class="ltx_text ltx_font_bold">Continued</span></td>
<td id="A3.T7.1.1.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.1.4.1" class="ltx_text ltx_font_bold">Context lengthened</span></td>
</tr>
<tr id="A3.T7.1.2" class="ltx_tr">
<td id="A3.T7.1.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.2.1.1" class="ltx_text ltx_font_bold">ARC_C</span></td>
<td id="A3.T7.1.2.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">43.17</td>
<td id="A3.T7.1.2.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">47.53</td>
<td id="A3.T7.1.2.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">45.39</td>
</tr>
<tr id="A3.T7.1.3" class="ltx_tr">
<td id="A3.T7.1.3.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.3.1.1" class="ltx_text ltx_font_bold">ARC_E</span></td>
<td id="A3.T7.1.3.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">74.87</td>
<td id="A3.T7.1.3.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">78.62</td>
<td id="A3.T7.1.3.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">78.37</td>
</tr>
<tr id="A3.T7.1.4" class="ltx_tr">
<td id="A3.T7.1.4.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.4.1.1" class="ltx_text ltx_font_bold">HellaSwag</span></td>
<td id="A3.T7.1.4.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">54.70</td>
<td id="A3.T7.1.4.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">55.50</td>
<td id="A3.T7.1.4.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">55.24</td>
</tr>
<tr id="A3.T7.1.5" class="ltx_tr">
<td id="A3.T7.1.5.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.5.1.1" class="ltx_text ltx_font_bold">LAMBADA</span></td>
<td id="A3.T7.1.5.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">73.51</td>
<td id="A3.T7.1.5.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">70.13</td>
<td id="A3.T7.1.5.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">69.90</td>
</tr>
<tr id="A3.T7.1.6" class="ltx_tr">
<td id="A3.T7.1.6.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.6.1.1" class="ltx_text ltx_font_bold">PIQA</span></td>
<td id="A3.T7.1.6.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">77.37</td>
<td id="A3.T7.1.6.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">78.67</td>
<td id="A3.T7.1.6.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">78.40</td>
</tr>
<tr id="A3.T7.1.7" class="ltx_tr">
<td id="A3.T7.1.7.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.7.1.1" class="ltx_text ltx_font_bold">SciQ</span></td>
<td id="A3.T7.1.7.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">94.90</td>
<td id="A3.T7.1.7.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">95.80</td>
<td id="A3.T7.1.7.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">95.70</td>
</tr>
<tr id="A3.T7.1.8" class="ltx_tr">
<td id="A3.T7.1.8.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.8.1.1" class="ltx_text ltx_font_bold">WinoGrande</span></td>
<td id="A3.T7.1.8.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">65.82</td>
<td id="A3.T7.1.8.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">67.32</td>
<td id="A3.T7.1.8.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">67.01</td>
</tr>
<tr id="A3.T7.1.9" class="ltx_tr">
<td id="A3.T7.1.9.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.9.1.1" class="ltx_text ltx_font_bold">TriviaQA (1 shot)</span></td>
<td id="A3.T7.1.9.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">42.46</td>
<td id="A3.T7.1.9.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">39.13</td>
<td id="A3.T7.1.9.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">38.11</td>
</tr>
<tr id="A3.T7.1.10" class="ltx_tr">
<td id="A3.T7.1.10.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.10.1.1" class="ltx_text ltx_font_bold">WebQS (1 shot)</span></td>
<td id="A3.T7.1.10.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">19.24</td>
<td id="A3.T7.1.10.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">18.06</td>
<td id="A3.T7.1.10.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">17.22</td>
</tr>
<tr id="A3.T7.1.11" class="ltx_tr">
<td id="A3.T7.1.11.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.11.1.1" class="ltx_text ltx_font_bold">CoreEN average</span></td>
<td id="A3.T7.1.11.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">60.67</td>
<td id="A3.T7.1.11.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">61.20</td>
<td id="A3.T7.1.11.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">60.59</td>
</tr>
<tr id="A3.T7.1.12" class="ltx_tr">
<td id="A3.T7.1.12.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.12.1.1" class="ltx_text ltx_font_bold">MMLU (5 shot)</span></td>
<td id="A3.T7.1.12.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">57.00</td>
<td id="A3.T7.1.12.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">61.35</td>
<td id="A3.T7.1.12.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">60.64</td>
</tr>
<tr id="A3.T7.1.13" class="ltx_tr">
<td id="A3.T7.1.13.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.13.1.1" class="ltx_text ltx_font_bold">GSM8K (8 shot CoT)</span></td>
<td id="A3.T7.1.13.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">27.45</td>
<td id="A3.T7.1.13.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">42.53</td>
<td id="A3.T7.1.13.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">40.00</td>
</tr>
<tr id="A3.T7.1.14" class="ltx_tr">
<td id="A3.T7.1.14.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.14.1.1" class="ltx_text ltx_font_bold">MATH (4 shot CoT)</span></td>
<td id="A3.T7.1.14.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">8.31</td>
<td id="A3.T7.1.14.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">16.97</td>
<td id="A3.T7.1.14.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">15.48</td>
</tr>
<tr id="A3.T7.1.15" class="ltx_tr">
<td id="A3.T7.1.15.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.15.1.1" class="ltx_text ltx_font_bold">HumanEval-Py pass@1</span></td>
<td id="A3.T7.1.15.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">16.48</td>
<td id="A3.T7.1.15.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">27.38</td>
<td id="A3.T7.1.15.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">30.84</td>
</tr>
<tr id="A3.T7.1.16" class="ltx_tr">
<td id="A3.T7.1.16.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T7.1.16.1.1" class="ltx_text ltx_font_bold">MultiPLE-Swift pass@1</span></td>
<td id="A3.T7.1.16.2" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">8.88</td>
<td id="A3.T7.1.16.3" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">19.24</td>
<td id="A3.T7.1.16.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">18.06</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>Pre-training evaluation for AFM-on-device with an internal harness. Unless otherwise noted, we use 0-shot prompts. TriviaQA evaluation is on the larger and more challenging “Web” split.</figcaption>
</figure>
<figure id="A3.T8" class="ltx_table">
<table id="A3.T8.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A3.T8.1.1" class="ltx_tr">
<td id="A3.T8.1.1.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.1.1.1" class="ltx_text ltx_font_bold">AFM-server</span></td>
<td id="A3.T8.1.1.2" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.1.2.1" class="ltx_text ltx_font_bold">Core</span></td>
<td id="A3.T8.1.1.3" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.1.3.1" class="ltx_text ltx_font_bold">Continued</span></td>
<td id="A3.T8.1.1.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.1.4.1" class="ltx_text ltx_font_bold">Context lengthened</span></td>
</tr>
<tr id="A3.T8.1.2" class="ltx_tr">
<td id="A3.T8.1.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.2.1.1" class="ltx_text ltx_font_bold">ARC_C</span></td>
<td id="A3.T8.1.2.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">58.28</td>
<td id="A3.T8.1.2.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">58.87</td>
<td id="A3.T8.1.2.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">57.94</td>
</tr>
<tr id="A3.T8.1.3" class="ltx_tr">
<td id="A3.T8.1.3.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.3.1.1" class="ltx_text ltx_font_bold">ARC_E</span></td>
<td id="A3.T8.1.3.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">85.61</td>
<td id="A3.T8.1.3.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">85.44</td>
<td id="A3.T8.1.3.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">85.06</td>
</tr>
<tr id="A3.T8.1.4" class="ltx_tr">
<td id="A3.T8.1.4.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.4.1.1" class="ltx_text ltx_font_bold">HellaSwag</span></td>
<td id="A3.T8.1.4.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">64.17</td>
<td id="A3.T8.1.4.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">64.53</td>
<td id="A3.T8.1.4.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">64.37</td>
</tr>
<tr id="A3.T8.1.5" class="ltx_tr">
<td id="A3.T8.1.5.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.5.1.1" class="ltx_text ltx_font_bold">LAMBADA</span></td>
<td id="A3.T8.1.5.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">78.38</td>
<td id="A3.T8.1.5.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">77.59</td>
<td id="A3.T8.1.5.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">77.82</td>
</tr>
<tr id="A3.T8.1.6" class="ltx_tr">
<td id="A3.T8.1.6.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.6.1.1" class="ltx_text ltx_font_bold">PIQA</span></td>
<td id="A3.T8.1.6.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">82.37</td>
<td id="A3.T8.1.6.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">81.99</td>
<td id="A3.T8.1.6.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">81.88</td>
</tr>
<tr id="A3.T8.1.7" class="ltx_tr">
<td id="A3.T8.1.7.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.7.1.1" class="ltx_text ltx_font_bold">SciQ</span></td>
<td id="A3.T8.1.7.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">96.60</td>
<td id="A3.T8.1.7.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">97.10</td>
<td id="A3.T8.1.7.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">97.00</td>
</tr>
<tr id="A3.T8.1.8" class="ltx_tr">
<td id="A3.T8.1.8.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.8.1.1" class="ltx_text ltx_font_bold">WinoGrande</span></td>
<td id="A3.T8.1.8.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">80.51</td>
<td id="A3.T8.1.8.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">79.16</td>
<td id="A3.T8.1.8.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">79.08</td>
</tr>
<tr id="A3.T8.1.9" class="ltx_tr">
<td id="A3.T8.1.9.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.9.1.1" class="ltx_text ltx_font_bold">TriviaQA (1 shot)</span></td>
<td id="A3.T8.1.9.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">54.33</td>
<td id="A3.T8.1.9.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">53.57</td>
<td id="A3.T8.1.9.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">53.42</td>
</tr>
<tr id="A3.T8.1.10" class="ltx_tr">
<td id="A3.T8.1.10.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.10.1.1" class="ltx_text ltx_font_bold">WebQS (1 shot)</span></td>
<td id="A3.T8.1.10.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">29.97</td>
<td id="A3.T8.1.10.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">27.66</td>
<td id="A3.T8.1.10.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">27.41</td>
</tr>
<tr id="A3.T8.1.11" class="ltx_tr">
<td id="A3.T8.1.11.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.11.1.1" class="ltx_text ltx_font_bold">CoreEN average</span></td>
<td id="A3.T8.1.11.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">70.02</td>
<td id="A3.T8.1.11.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">69.55</td>
<td id="A3.T8.1.11.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">69.33</td>
</tr>
<tr id="A3.T8.1.12" class="ltx_tr">
<td id="A3.T8.1.12.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.12.1.1" class="ltx_text ltx_font_bold">MMLU (5 shot)</span></td>
<td id="A3.T8.1.12.2" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">74.00</td>
<td id="A3.T8.1.12.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">75.24</td>
<td id="A3.T8.1.12.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">74.80</td>
</tr>
<tr id="A3.T8.1.13" class="ltx_tr">
<td id="A3.T8.1.13.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.13.1.1" class="ltx_text ltx_font_bold">GSM8K (8 shot CoT)</span></td>
<td id="A3.T8.1.13.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">75.44</td>
<td id="A3.T8.1.13.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">74.83</td>
<td id="A3.T8.1.13.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">75.51</td>
</tr>
<tr id="A3.T8.1.14" class="ltx_tr">
<td id="A3.T8.1.14.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.14.1.1" class="ltx_text ltx_font_bold">MATH (4 shot CoT)</span></td>
<td id="A3.T8.1.14.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">32.24</td>
<td id="A3.T8.1.14.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">36.48</td>
<td id="A3.T8.1.14.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">35.77</td>
</tr>
<tr id="A3.T8.1.15" class="ltx_tr">
<td id="A3.T8.1.15.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.15.1.1" class="ltx_text ltx_font_bold">HumanEval-Py</span></td>
<td id="A3.T8.1.15.2" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">33.23</td>
<td id="A3.T8.1.15.3" class="ltx_td ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">40.77</td>
<td id="A3.T8.1.15.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">39.55</td>
</tr>
<tr id="A3.T8.1.16" class="ltx_tr">
<td id="A3.T8.1.16.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="A3.T8.1.16.1.1" class="ltx_text ltx_font_bold">MultiPLE-Swift</span></td>
<td id="A3.T8.1.16.2" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">30.15</td>
<td id="A3.T8.1.16.3" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">37.70</td>
<td id="A3.T8.1.16.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">38.11</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 8: </span>Pre-training evaluation for AFM-server with an internal harness. Unless otherwise noted, we use 0-shot prompts. TriviaQA evaluation is on the larger and more challenging “Web" split.</figcaption>
</figure>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.1" class="ltx_p">In <a href="#A3.T7" title="Table 7 ‣ Appendix C Pre-training stage-breakdown evaluations ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 7</span></a> and <a href="#A3.T8" title="Table 8 ‣ Appendix C Pre-training stage-breakdown evaluations ‣ Apple Intelligence Foundation Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> we present internal benchmarks after all three stages of pre-training. As expected, continued pre-training acts to improve math and particularly code model capabilities, whilst subtly improving a few other benchmarks. The context-lengthening stage leaves the majority of these benchmarks on-par, with changes (positive and negative) typically within the range of what we consider to be evaluation noise.</p>
</div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Long-context evaluation</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">Although the focus for this version of AFM was not to support context lengths longer than 8k, in <a href="#A4.T9" title="Table 9 ‣ Appendix D Long-context evaluation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 9</span></a> we use the RULER <cite class="ltx_cite ltx_citemacro_citep">[Hsieh et al., <a href="#bib.bib22" title="" class="ltx_ref">2024</a>]</cite> benchmark to evaluate AFM-server at 4k to 32k context lengths. We note that the model is capable of performing perfectly up to a sequence length of <math id="A4.p1.1.m1.1" class="ltx_Math" alttext="\geq\mathrm{32k}" display="inline"><semantics id="A4.p1.1.m1.1a"><mrow id="A4.p1.1.m1.1.1" xref="A4.p1.1.m1.1.1.cmml"><mi id="A4.p1.1.m1.1.1.2" xref="A4.p1.1.m1.1.1.2.cmml"></mi><mo id="A4.p1.1.m1.1.1.1" xref="A4.p1.1.m1.1.1.1.cmml">≥</mo><mrow id="A4.p1.1.m1.1.1.3" xref="A4.p1.1.m1.1.1.3.cmml"><mn id="A4.p1.1.m1.1.1.3.2" xref="A4.p1.1.m1.1.1.3.2.cmml">32</mn><mo lspace="0em" rspace="0em" id="A4.p1.1.m1.1.1.3.1" xref="A4.p1.1.m1.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="A4.p1.1.m1.1.1.3.3" xref="A4.p1.1.m1.1.1.3.3.cmml">k</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p1.1.m1.1b"><apply id="A4.p1.1.m1.1.1.cmml" xref="A4.p1.1.m1.1.1"><geq id="A4.p1.1.m1.1.1.1.cmml" xref="A4.p1.1.m1.1.1.1"></geq><csymbol cd="latexml" id="A4.p1.1.m1.1.1.2.cmml" xref="A4.p1.1.m1.1.1.2">absent</csymbol><apply id="A4.p1.1.m1.1.1.3.cmml" xref="A4.p1.1.m1.1.1.3"><times id="A4.p1.1.m1.1.1.3.1.cmml" xref="A4.p1.1.m1.1.1.3.1"></times><cn type="integer" id="A4.p1.1.m1.1.1.3.2.cmml" xref="A4.p1.1.m1.1.1.3.2">32</cn><ci id="A4.p1.1.m1.1.1.3.3.cmml" xref="A4.p1.1.m1.1.1.3.3">k</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.1.m1.1c">\geq\mathrm{32k}</annotation></semantics></math> when tested against straightforward retrieval-like tasks, e.g., needle-in-the-haystack (NIAH). It is clear, however, that the model performance gradually suffers with an increasing context length on RULER, a more complex evaluation benchmark than NIAH, suggesting that the real context length for AFM-server, for tasks beyond retrieval, is currently at most 24k.</p>
</div>
<figure id="A4.T9" class="ltx_table">
<table id="A4.T9.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A4.T9.1.1" class="ltx_tr">
<td id="A4.T9.1.1.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A4.T9.1.1.1.1" class="ltx_text ltx_font_bold">AFM-server</span></td>
<td id="A4.T9.1.1.2" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-top:1pt;padding-bottom:1pt;"><span id="A4.T9.1.1.2.1" class="ltx_text ltx_font_bold">Average acc</span></td>
</tr>
<tr id="A4.T9.1.2" class="ltx_tr">
<td id="A4.T9.1.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;"><span id="A4.T9.1.2.1.1" class="ltx_text ltx_font_bold">Ctx @ 4096</span></td>
<td id="A4.T9.1.2.2" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:1pt;padding-bottom:1pt;">91.7</td>
</tr>
<tr id="A4.T9.1.3" class="ltx_tr">
<td id="A4.T9.1.3.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A4.T9.1.3.1.1" class="ltx_text ltx_font_bold">Ctx @ 8192</span></td>
<td id="A4.T9.1.3.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">87.7</td>
</tr>
<tr id="A4.T9.1.4" class="ltx_tr">
<td id="A4.T9.1.4.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A4.T9.1.4.1.1" class="ltx_text ltx_font_bold">Ctx @ 16384</span></td>
<td id="A4.T9.1.4.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">84.1</td>
</tr>
<tr id="A4.T9.1.5" class="ltx_tr">
<td id="A4.T9.1.5.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A4.T9.1.5.1.1" class="ltx_text ltx_font_bold">Ctx @ 20480</span></td>
<td id="A4.T9.1.5.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">79.1</td>
</tr>
<tr id="A4.T9.1.6" class="ltx_tr">
<td id="A4.T9.1.6.1" class="ltx_td ltx_align_left" style="padding-top:1pt;padding-bottom:1pt;"><span id="A4.T9.1.6.1.1" class="ltx_text ltx_font_bold">Ctx @ 24576</span></td>
<td id="A4.T9.1.6.2" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:1pt;padding-bottom:1pt;">75.8</td>
</tr>
<tr id="A4.T9.1.7" class="ltx_tr">
<td id="A4.T9.1.7.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;"><span id="A4.T9.1.7.1.1" class="ltx_text ltx_font_bold">Ctx @ 32768</span></td>
<td id="A4.T9.1.7.2" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-top:1pt;padding-bottom:1pt;">43.3</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 9: </span>RULER <cite class="ltx_cite ltx_citemacro_citep">[Hsieh et al., <a href="#bib.bib22" title="" class="ltx_ref">2024</a>]</cite> average evaluation results, averaged over 13 synthetic long-context tasks using 500 examples per task.</figcaption>
</figure>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Technical details for RLHF</h2>

<section id="A5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="A5.SS1.2.1.1" class="ltx_text ltx_font_sansserif">E.1</span> </span><span id="A5.SS1.3.2" class="ltx_text ltx_font_sansserif">Reward modeling</span>
</h3>

<div id="A5.SS1.p1" class="ltx_para">
<p id="A5.SS1.p1.7" class="ltx_p">The human preference data that we use in reward model training has the following format:</p>
<ul id="A5.I1" class="ltx_itemize">
<li id="A5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i1.p1" class="ltx_para">
<p id="A5.I1.i1.p1.1" class="ltx_p"><math id="A5.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A5.I1.i1.p1.1.m1.1a"><mi id="A5.I1.i1.p1.1.m1.1.1" xref="A5.I1.i1.p1.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A5.I1.i1.p1.1.m1.1b"><ci id="A5.I1.i1.p1.1.m1.1.1.cmml" xref="A5.I1.i1.p1.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i1.p1.1.m1.1c">x</annotation></semantics></math>: the prompt;</p>
</div>
</li>
<li id="A5.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i2.p1" class="ltx_para">
<p id="A5.I1.i2.p1.1" class="ltx_p"><math id="A5.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="y_{c}" display="inline"><semantics id="A5.I1.i2.p1.1.m1.1a"><msub id="A5.I1.i2.p1.1.m1.1.1" xref="A5.I1.i2.p1.1.m1.1.1.cmml"><mi id="A5.I1.i2.p1.1.m1.1.1.2" xref="A5.I1.i2.p1.1.m1.1.1.2.cmml">y</mi><mi id="A5.I1.i2.p1.1.m1.1.1.3" xref="A5.I1.i2.p1.1.m1.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="A5.I1.i2.p1.1.m1.1b"><apply id="A5.I1.i2.p1.1.m1.1.1.cmml" xref="A5.I1.i2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i2.p1.1.m1.1.1.1.cmml" xref="A5.I1.i2.p1.1.m1.1.1">subscript</csymbol><ci id="A5.I1.i2.p1.1.m1.1.1.2.cmml" xref="A5.I1.i2.p1.1.m1.1.1.2">𝑦</ci><ci id="A5.I1.i2.p1.1.m1.1.1.3.cmml" xref="A5.I1.i2.p1.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i2.p1.1.m1.1c">y_{c}</annotation></semantics></math>: the chosen (preferred) response;</p>
</div>
</li>
<li id="A5.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i3.p1" class="ltx_para">
<p id="A5.I1.i3.p1.1" class="ltx_p"><math id="A5.I1.i3.p1.1.m1.1" class="ltx_Math" alttext="y_{r}" display="inline"><semantics id="A5.I1.i3.p1.1.m1.1a"><msub id="A5.I1.i3.p1.1.m1.1.1" xref="A5.I1.i3.p1.1.m1.1.1.cmml"><mi id="A5.I1.i3.p1.1.m1.1.1.2" xref="A5.I1.i3.p1.1.m1.1.1.2.cmml">y</mi><mi id="A5.I1.i3.p1.1.m1.1.1.3" xref="A5.I1.i3.p1.1.m1.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="A5.I1.i3.p1.1.m1.1b"><apply id="A5.I1.i3.p1.1.m1.1.1.cmml" xref="A5.I1.i3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i3.p1.1.m1.1.1.1.cmml" xref="A5.I1.i3.p1.1.m1.1.1">subscript</csymbol><ci id="A5.I1.i3.p1.1.m1.1.1.2.cmml" xref="A5.I1.i3.p1.1.m1.1.1.2">𝑦</ci><ci id="A5.I1.i3.p1.1.m1.1.1.3.cmml" xref="A5.I1.i3.p1.1.m1.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i3.p1.1.m1.1c">y_{r}</annotation></semantics></math>: the rejected response;</p>
</div>
</li>
<li id="A5.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i4.p1" class="ltx_para">
<p id="A5.I1.i4.p1.1" class="ltx_p"><math id="A5.I1.i4.p1.1.m1.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="A5.I1.i4.p1.1.m1.1a"><mi mathvariant="normal" id="A5.I1.i4.p1.1.m1.1.1" xref="A5.I1.i4.p1.1.m1.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="A5.I1.i4.p1.1.m1.1b"><ci id="A5.I1.i4.p1.1.m1.1.1.cmml" xref="A5.I1.i4.p1.1.m1.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i4.p1.1.m1.1c">\ell</annotation></semantics></math>: the level of the human preference;</p>
</div>
</li>
<li id="A5.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i5.p1" class="ltx_para">
<p id="A5.I1.i5.p1.2" class="ltx_p"><math id="A5.I1.i5.p1.1.m1.1" class="ltx_Math" alttext="z^{\text{if}}_{c}" display="inline"><semantics id="A5.I1.i5.p1.1.m1.1a"><msubsup id="A5.I1.i5.p1.1.m1.1.1" xref="A5.I1.i5.p1.1.m1.1.1.cmml"><mi id="A5.I1.i5.p1.1.m1.1.1.2.2" xref="A5.I1.i5.p1.1.m1.1.1.2.2.cmml">z</mi><mi id="A5.I1.i5.p1.1.m1.1.1.3" xref="A5.I1.i5.p1.1.m1.1.1.3.cmml">c</mi><mtext id="A5.I1.i5.p1.1.m1.1.1.2.3" xref="A5.I1.i5.p1.1.m1.1.1.2.3a.cmml">if</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.I1.i5.p1.1.m1.1b"><apply id="A5.I1.i5.p1.1.m1.1.1.cmml" xref="A5.I1.i5.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i5.p1.1.m1.1.1.1.cmml" xref="A5.I1.i5.p1.1.m1.1.1">subscript</csymbol><apply id="A5.I1.i5.p1.1.m1.1.1.2.cmml" xref="A5.I1.i5.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i5.p1.1.m1.1.1.2.1.cmml" xref="A5.I1.i5.p1.1.m1.1.1">superscript</csymbol><ci id="A5.I1.i5.p1.1.m1.1.1.2.2.cmml" xref="A5.I1.i5.p1.1.m1.1.1.2.2">𝑧</ci><ci id="A5.I1.i5.p1.1.m1.1.1.2.3a.cmml" xref="A5.I1.i5.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="A5.I1.i5.p1.1.m1.1.1.2.3.cmml" xref="A5.I1.i5.p1.1.m1.1.1.2.3">if</mtext></ci></apply><ci id="A5.I1.i5.p1.1.m1.1.1.3.cmml" xref="A5.I1.i5.p1.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i5.p1.1.m1.1c">z^{\text{if}}_{c}</annotation></semantics></math> and <math id="A5.I1.i5.p1.2.m2.1" class="ltx_Math" alttext="z^{\text{if}}_{r}" display="inline"><semantics id="A5.I1.i5.p1.2.m2.1a"><msubsup id="A5.I1.i5.p1.2.m2.1.1" xref="A5.I1.i5.p1.2.m2.1.1.cmml"><mi id="A5.I1.i5.p1.2.m2.1.1.2.2" xref="A5.I1.i5.p1.2.m2.1.1.2.2.cmml">z</mi><mi id="A5.I1.i5.p1.2.m2.1.1.3" xref="A5.I1.i5.p1.2.m2.1.1.3.cmml">r</mi><mtext id="A5.I1.i5.p1.2.m2.1.1.2.3" xref="A5.I1.i5.p1.2.m2.1.1.2.3a.cmml">if</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.I1.i5.p1.2.m2.1b"><apply id="A5.I1.i5.p1.2.m2.1.1.cmml" xref="A5.I1.i5.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.I1.i5.p1.2.m2.1.1.1.cmml" xref="A5.I1.i5.p1.2.m2.1.1">subscript</csymbol><apply id="A5.I1.i5.p1.2.m2.1.1.2.cmml" xref="A5.I1.i5.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.I1.i5.p1.2.m2.1.1.2.1.cmml" xref="A5.I1.i5.p1.2.m2.1.1">superscript</csymbol><ci id="A5.I1.i5.p1.2.m2.1.1.2.2.cmml" xref="A5.I1.i5.p1.2.m2.1.1.2.2">𝑧</ci><ci id="A5.I1.i5.p1.2.m2.1.1.2.3a.cmml" xref="A5.I1.i5.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="A5.I1.i5.p1.2.m2.1.1.2.3.cmml" xref="A5.I1.i5.p1.2.m2.1.1.2.3">if</mtext></ci></apply><ci id="A5.I1.i5.p1.2.m2.1.1.3.cmml" xref="A5.I1.i5.p1.2.m2.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i5.p1.2.m2.1c">z^{\text{if}}_{r}</annotation></semantics></math>: the instruction-following property of the two responses;</p>
</div>
</li>
<li id="A5.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i6.p1" class="ltx_para">
<p id="A5.I1.i6.p1.2" class="ltx_p"><math id="A5.I1.i6.p1.1.m1.1" class="ltx_Math" alttext="z^{\text{verb}}_{c}" display="inline"><semantics id="A5.I1.i6.p1.1.m1.1a"><msubsup id="A5.I1.i6.p1.1.m1.1.1" xref="A5.I1.i6.p1.1.m1.1.1.cmml"><mi id="A5.I1.i6.p1.1.m1.1.1.2.2" xref="A5.I1.i6.p1.1.m1.1.1.2.2.cmml">z</mi><mi id="A5.I1.i6.p1.1.m1.1.1.3" xref="A5.I1.i6.p1.1.m1.1.1.3.cmml">c</mi><mtext id="A5.I1.i6.p1.1.m1.1.1.2.3" xref="A5.I1.i6.p1.1.m1.1.1.2.3a.cmml">verb</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.I1.i6.p1.1.m1.1b"><apply id="A5.I1.i6.p1.1.m1.1.1.cmml" xref="A5.I1.i6.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i6.p1.1.m1.1.1.1.cmml" xref="A5.I1.i6.p1.1.m1.1.1">subscript</csymbol><apply id="A5.I1.i6.p1.1.m1.1.1.2.cmml" xref="A5.I1.i6.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i6.p1.1.m1.1.1.2.1.cmml" xref="A5.I1.i6.p1.1.m1.1.1">superscript</csymbol><ci id="A5.I1.i6.p1.1.m1.1.1.2.2.cmml" xref="A5.I1.i6.p1.1.m1.1.1.2.2">𝑧</ci><ci id="A5.I1.i6.p1.1.m1.1.1.2.3a.cmml" xref="A5.I1.i6.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="A5.I1.i6.p1.1.m1.1.1.2.3.cmml" xref="A5.I1.i6.p1.1.m1.1.1.2.3">verb</mtext></ci></apply><ci id="A5.I1.i6.p1.1.m1.1.1.3.cmml" xref="A5.I1.i6.p1.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i6.p1.1.m1.1c">z^{\text{verb}}_{c}</annotation></semantics></math> and <math id="A5.I1.i6.p1.2.m2.1" class="ltx_Math" alttext="z^{\text{verb}}_{r}" display="inline"><semantics id="A5.I1.i6.p1.2.m2.1a"><msubsup id="A5.I1.i6.p1.2.m2.1.1" xref="A5.I1.i6.p1.2.m2.1.1.cmml"><mi id="A5.I1.i6.p1.2.m2.1.1.2.2" xref="A5.I1.i6.p1.2.m2.1.1.2.2.cmml">z</mi><mi id="A5.I1.i6.p1.2.m2.1.1.3" xref="A5.I1.i6.p1.2.m2.1.1.3.cmml">r</mi><mtext id="A5.I1.i6.p1.2.m2.1.1.2.3" xref="A5.I1.i6.p1.2.m2.1.1.2.3a.cmml">verb</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.I1.i6.p1.2.m2.1b"><apply id="A5.I1.i6.p1.2.m2.1.1.cmml" xref="A5.I1.i6.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.I1.i6.p1.2.m2.1.1.1.cmml" xref="A5.I1.i6.p1.2.m2.1.1">subscript</csymbol><apply id="A5.I1.i6.p1.2.m2.1.1.2.cmml" xref="A5.I1.i6.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.I1.i6.p1.2.m2.1.1.2.1.cmml" xref="A5.I1.i6.p1.2.m2.1.1">superscript</csymbol><ci id="A5.I1.i6.p1.2.m2.1.1.2.2.cmml" xref="A5.I1.i6.p1.2.m2.1.1.2.2">𝑧</ci><ci id="A5.I1.i6.p1.2.m2.1.1.2.3a.cmml" xref="A5.I1.i6.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="A5.I1.i6.p1.2.m2.1.1.2.3.cmml" xref="A5.I1.i6.p1.2.m2.1.1.2.3">verb</mtext></ci></apply><ci id="A5.I1.i6.p1.2.m2.1.1.3.cmml" xref="A5.I1.i6.p1.2.m2.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i6.p1.2.m2.1c">z^{\text{verb}}_{r}</annotation></semantics></math>: the verbosity of the two responses;</p>
</div>
</li>
<li id="A5.I1.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i7.p1" class="ltx_para">
<p id="A5.I1.i7.p1.2" class="ltx_p"><math id="A5.I1.i7.p1.1.m1.1" class="ltx_Math" alttext="z^{\text{truth}}_{c}" display="inline"><semantics id="A5.I1.i7.p1.1.m1.1a"><msubsup id="A5.I1.i7.p1.1.m1.1.1" xref="A5.I1.i7.p1.1.m1.1.1.cmml"><mi id="A5.I1.i7.p1.1.m1.1.1.2.2" xref="A5.I1.i7.p1.1.m1.1.1.2.2.cmml">z</mi><mi id="A5.I1.i7.p1.1.m1.1.1.3" xref="A5.I1.i7.p1.1.m1.1.1.3.cmml">c</mi><mtext id="A5.I1.i7.p1.1.m1.1.1.2.3" xref="A5.I1.i7.p1.1.m1.1.1.2.3a.cmml">truth</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.I1.i7.p1.1.m1.1b"><apply id="A5.I1.i7.p1.1.m1.1.1.cmml" xref="A5.I1.i7.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i7.p1.1.m1.1.1.1.cmml" xref="A5.I1.i7.p1.1.m1.1.1">subscript</csymbol><apply id="A5.I1.i7.p1.1.m1.1.1.2.cmml" xref="A5.I1.i7.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i7.p1.1.m1.1.1.2.1.cmml" xref="A5.I1.i7.p1.1.m1.1.1">superscript</csymbol><ci id="A5.I1.i7.p1.1.m1.1.1.2.2.cmml" xref="A5.I1.i7.p1.1.m1.1.1.2.2">𝑧</ci><ci id="A5.I1.i7.p1.1.m1.1.1.2.3a.cmml" xref="A5.I1.i7.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="A5.I1.i7.p1.1.m1.1.1.2.3.cmml" xref="A5.I1.i7.p1.1.m1.1.1.2.3">truth</mtext></ci></apply><ci id="A5.I1.i7.p1.1.m1.1.1.3.cmml" xref="A5.I1.i7.p1.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i7.p1.1.m1.1c">z^{\text{truth}}_{c}</annotation></semantics></math> and <math id="A5.I1.i7.p1.2.m2.1" class="ltx_Math" alttext="z^{\text{truth}}_{r}" display="inline"><semantics id="A5.I1.i7.p1.2.m2.1a"><msubsup id="A5.I1.i7.p1.2.m2.1.1" xref="A5.I1.i7.p1.2.m2.1.1.cmml"><mi id="A5.I1.i7.p1.2.m2.1.1.2.2" xref="A5.I1.i7.p1.2.m2.1.1.2.2.cmml">z</mi><mi id="A5.I1.i7.p1.2.m2.1.1.3" xref="A5.I1.i7.p1.2.m2.1.1.3.cmml">r</mi><mtext id="A5.I1.i7.p1.2.m2.1.1.2.3" xref="A5.I1.i7.p1.2.m2.1.1.2.3a.cmml">truth</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.I1.i7.p1.2.m2.1b"><apply id="A5.I1.i7.p1.2.m2.1.1.cmml" xref="A5.I1.i7.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.I1.i7.p1.2.m2.1.1.1.cmml" xref="A5.I1.i7.p1.2.m2.1.1">subscript</csymbol><apply id="A5.I1.i7.p1.2.m2.1.1.2.cmml" xref="A5.I1.i7.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.I1.i7.p1.2.m2.1.1.2.1.cmml" xref="A5.I1.i7.p1.2.m2.1.1">superscript</csymbol><ci id="A5.I1.i7.p1.2.m2.1.1.2.2.cmml" xref="A5.I1.i7.p1.2.m2.1.1.2.2">𝑧</ci><ci id="A5.I1.i7.p1.2.m2.1.1.2.3a.cmml" xref="A5.I1.i7.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="A5.I1.i7.p1.2.m2.1.1.2.3.cmml" xref="A5.I1.i7.p1.2.m2.1.1.2.3">truth</mtext></ci></apply><ci id="A5.I1.i7.p1.2.m2.1.1.3.cmml" xref="A5.I1.i7.p1.2.m2.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i7.p1.2.m2.1c">z^{\text{truth}}_{r}</annotation></semantics></math>: the truthfulness of the two responses;</p>
</div>
</li>
<li id="A5.I1.i8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A5.I1.i8.p1" class="ltx_para">
<p id="A5.I1.i8.p1.2" class="ltx_p"><math id="A5.I1.i8.p1.1.m1.1" class="ltx_Math" alttext="z^{\text{harm}}_{c}" display="inline"><semantics id="A5.I1.i8.p1.1.m1.1a"><msubsup id="A5.I1.i8.p1.1.m1.1.1" xref="A5.I1.i8.p1.1.m1.1.1.cmml"><mi id="A5.I1.i8.p1.1.m1.1.1.2.2" xref="A5.I1.i8.p1.1.m1.1.1.2.2.cmml">z</mi><mi id="A5.I1.i8.p1.1.m1.1.1.3" xref="A5.I1.i8.p1.1.m1.1.1.3.cmml">c</mi><mtext id="A5.I1.i8.p1.1.m1.1.1.2.3" xref="A5.I1.i8.p1.1.m1.1.1.2.3a.cmml">harm</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.I1.i8.p1.1.m1.1b"><apply id="A5.I1.i8.p1.1.m1.1.1.cmml" xref="A5.I1.i8.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i8.p1.1.m1.1.1.1.cmml" xref="A5.I1.i8.p1.1.m1.1.1">subscript</csymbol><apply id="A5.I1.i8.p1.1.m1.1.1.2.cmml" xref="A5.I1.i8.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.I1.i8.p1.1.m1.1.1.2.1.cmml" xref="A5.I1.i8.p1.1.m1.1.1">superscript</csymbol><ci id="A5.I1.i8.p1.1.m1.1.1.2.2.cmml" xref="A5.I1.i8.p1.1.m1.1.1.2.2">𝑧</ci><ci id="A5.I1.i8.p1.1.m1.1.1.2.3a.cmml" xref="A5.I1.i8.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="A5.I1.i8.p1.1.m1.1.1.2.3.cmml" xref="A5.I1.i8.p1.1.m1.1.1.2.3">harm</mtext></ci></apply><ci id="A5.I1.i8.p1.1.m1.1.1.3.cmml" xref="A5.I1.i8.p1.1.m1.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i8.p1.1.m1.1c">z^{\text{harm}}_{c}</annotation></semantics></math> and <math id="A5.I1.i8.p1.2.m2.1" class="ltx_Math" alttext="z^{\text{harm}}_{r}" display="inline"><semantics id="A5.I1.i8.p1.2.m2.1a"><msubsup id="A5.I1.i8.p1.2.m2.1.1" xref="A5.I1.i8.p1.2.m2.1.1.cmml"><mi id="A5.I1.i8.p1.2.m2.1.1.2.2" xref="A5.I1.i8.p1.2.m2.1.1.2.2.cmml">z</mi><mi id="A5.I1.i8.p1.2.m2.1.1.3" xref="A5.I1.i8.p1.2.m2.1.1.3.cmml">r</mi><mtext id="A5.I1.i8.p1.2.m2.1.1.2.3" xref="A5.I1.i8.p1.2.m2.1.1.2.3a.cmml">harm</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.I1.i8.p1.2.m2.1b"><apply id="A5.I1.i8.p1.2.m2.1.1.cmml" xref="A5.I1.i8.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.I1.i8.p1.2.m2.1.1.1.cmml" xref="A5.I1.i8.p1.2.m2.1.1">subscript</csymbol><apply id="A5.I1.i8.p1.2.m2.1.1.2.cmml" xref="A5.I1.i8.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.I1.i8.p1.2.m2.1.1.2.1.cmml" xref="A5.I1.i8.p1.2.m2.1.1">superscript</csymbol><ci id="A5.I1.i8.p1.2.m2.1.1.2.2.cmml" xref="A5.I1.i8.p1.2.m2.1.1.2.2">𝑧</ci><ci id="A5.I1.i8.p1.2.m2.1.1.2.3a.cmml" xref="A5.I1.i8.p1.2.m2.1.1.2.3"><mtext mathsize="70%" id="A5.I1.i8.p1.2.m2.1.1.2.3.cmml" xref="A5.I1.i8.p1.2.m2.1.1.2.3">harm</mtext></ci></apply><ci id="A5.I1.i8.p1.2.m2.1.1.3.cmml" xref="A5.I1.i8.p1.2.m2.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.I1.i8.p1.2.m2.1c">z^{\text{harm}}_{r}</annotation></semantics></math>: the harmlessness of the two responses.</p>
</div>
</li>
</ul>
<p id="A5.SS1.p1.6" class="ltx_p">In our reward modeling, the preference level <math id="A5.SS1.p1.1.m1.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="A5.SS1.p1.1.m1.1a"><mi mathvariant="normal" id="A5.SS1.p1.1.m1.1.1" xref="A5.SS1.p1.1.m1.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="A5.SS1.p1.1.m1.1b"><ci id="A5.SS1.p1.1.m1.1.1.cmml" xref="A5.SS1.p1.1.m1.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p1.1.m1.1c">\ell</annotation></semantics></math> takes <math id="A5.SS1.p1.2.m2.1" class="ltx_Math" alttext="4" display="inline"><semantics id="A5.SS1.p1.2.m2.1a"><mn id="A5.SS1.p1.2.m2.1.1" xref="A5.SS1.p1.2.m2.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="A5.SS1.p1.2.m2.1b"><cn type="integer" id="A5.SS1.p1.2.m2.1.1.cmml" xref="A5.SS1.p1.2.m2.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p1.2.m2.1c">4</annotation></semantics></math> possible values, indicating that the chosen response is negligibly better, slightly better, better, or significantly better than the rejected response. As for the single sided gradings, each label, e.g., <math id="A5.SS1.p1.3.m3.1" class="ltx_Math" alttext="z^{\text{if}}_{c}" display="inline"><semantics id="A5.SS1.p1.3.m3.1a"><msubsup id="A5.SS1.p1.3.m3.1.1" xref="A5.SS1.p1.3.m3.1.1.cmml"><mi id="A5.SS1.p1.3.m3.1.1.2.2" xref="A5.SS1.p1.3.m3.1.1.2.2.cmml">z</mi><mi id="A5.SS1.p1.3.m3.1.1.3" xref="A5.SS1.p1.3.m3.1.1.3.cmml">c</mi><mtext id="A5.SS1.p1.3.m3.1.1.2.3" xref="A5.SS1.p1.3.m3.1.1.2.3a.cmml">if</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.SS1.p1.3.m3.1b"><apply id="A5.SS1.p1.3.m3.1.1.cmml" xref="A5.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A5.SS1.p1.3.m3.1.1.1.cmml" xref="A5.SS1.p1.3.m3.1.1">subscript</csymbol><apply id="A5.SS1.p1.3.m3.1.1.2.cmml" xref="A5.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A5.SS1.p1.3.m3.1.1.2.1.cmml" xref="A5.SS1.p1.3.m3.1.1">superscript</csymbol><ci id="A5.SS1.p1.3.m3.1.1.2.2.cmml" xref="A5.SS1.p1.3.m3.1.1.2.2">𝑧</ci><ci id="A5.SS1.p1.3.m3.1.1.2.3a.cmml" xref="A5.SS1.p1.3.m3.1.1.2.3"><mtext mathsize="70%" id="A5.SS1.p1.3.m3.1.1.2.3.cmml" xref="A5.SS1.p1.3.m3.1.1.2.3">if</mtext></ci></apply><ci id="A5.SS1.p1.3.m3.1.1.3.cmml" xref="A5.SS1.p1.3.m3.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p1.3.m3.1c">z^{\text{if}}_{c}</annotation></semantics></math>, takes <math id="A5.SS1.p1.4.m4.1" class="ltx_Math" alttext="3" display="inline"><semantics id="A5.SS1.p1.4.m4.1a"><mn id="A5.SS1.p1.4.m4.1.1" xref="A5.SS1.p1.4.m4.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="A5.SS1.p1.4.m4.1b"><cn type="integer" id="A5.SS1.p1.4.m4.1.1.cmml" xref="A5.SS1.p1.4.m4.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p1.4.m4.1c">3</annotation></semantics></math> possible values. For instruction following, truthfulness, and harmlessness, the <math id="A5.SS1.p1.5.m5.1" class="ltx_Math" alttext="3" display="inline"><semantics id="A5.SS1.p1.5.m5.1a"><mn id="A5.SS1.p1.5.m5.1.1" xref="A5.SS1.p1.5.m5.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="A5.SS1.p1.5.m5.1b"><cn type="integer" id="A5.SS1.p1.5.m5.1.1.cmml" xref="A5.SS1.p1.5.m5.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p1.5.m5.1c">3</annotation></semantics></math> values correspond to the cases where the response has major issue, minor issue, or no issue. For verbosity, the <math id="A5.SS1.p1.6.m6.1" class="ltx_Math" alttext="3" display="inline"><semantics id="A5.SS1.p1.6.m6.1a"><mn id="A5.SS1.p1.6.m6.1.1" xref="A5.SS1.p1.6.m6.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="A5.SS1.p1.6.m6.1b"><cn type="integer" id="A5.SS1.p1.6.m6.1.1.cmml" xref="A5.SS1.p1.6.m6.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p1.6.m6.1c">3</annotation></semantics></math> values correspond to the cases where the response is too verbose, too short, or just right.</p>
</div>
<div id="A5.SS1.p2" class="ltx_para">
<p id="A5.SS1.p2.8" class="ltx_p">We use a multi-head architecture for the reward model. More specifically, we take a decoder-only transformer and obtain the last-layer embedding of the last non-padding token. We attach one linear and four MLP heads to the embedding. Denote the model parameters by <math id="A5.SS1.p2.1.m1.1" class="ltx_Math" alttext="\phi" display="inline"><semantics id="A5.SS1.p2.1.m1.1a"><mi id="A5.SS1.p2.1.m1.1.1" xref="A5.SS1.p2.1.m1.1.1.cmml">ϕ</mi><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.1.m1.1b"><ci id="A5.SS1.p2.1.m1.1.1.cmml" xref="A5.SS1.p2.1.m1.1.1">italic-ϕ</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.1.m1.1c">\phi</annotation></semantics></math> and the input prompt-response pair by <math id="A5.SS1.p2.2.m2.2" class="ltx_Math" alttext="(x,y)" display="inline"><semantics id="A5.SS1.p2.2.m2.2a"><mrow id="A5.SS1.p2.2.m2.2.3.2" xref="A5.SS1.p2.2.m2.2.3.1.cmml"><mo stretchy="false" id="A5.SS1.p2.2.m2.2.3.2.1" xref="A5.SS1.p2.2.m2.2.3.1.cmml">(</mo><mi id="A5.SS1.p2.2.m2.1.1" xref="A5.SS1.p2.2.m2.1.1.cmml">x</mi><mo id="A5.SS1.p2.2.m2.2.3.2.2" xref="A5.SS1.p2.2.m2.2.3.1.cmml">,</mo><mi id="A5.SS1.p2.2.m2.2.2" xref="A5.SS1.p2.2.m2.2.2.cmml">y</mi><mo stretchy="false" id="A5.SS1.p2.2.m2.2.3.2.3" xref="A5.SS1.p2.2.m2.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.2.m2.2b"><interval closure="open" id="A5.SS1.p2.2.m2.2.3.1.cmml" xref="A5.SS1.p2.2.m2.2.3.2"><ci id="A5.SS1.p2.2.m2.1.1.cmml" xref="A5.SS1.p2.2.m2.1.1">𝑥</ci><ci id="A5.SS1.p2.2.m2.2.2.cmml" xref="A5.SS1.p2.2.m2.2.2">𝑦</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.2.m2.2c">(x,y)</annotation></semantics></math>. The linear head outputs the preference reward <math id="A5.SS1.p2.3.m3.2" class="ltx_Math" alttext="r_{\phi}(x,y)\in\mathbb{R}" display="inline"><semantics id="A5.SS1.p2.3.m3.2a"><mrow id="A5.SS1.p2.3.m3.2.3" xref="A5.SS1.p2.3.m3.2.3.cmml"><mrow id="A5.SS1.p2.3.m3.2.3.2" xref="A5.SS1.p2.3.m3.2.3.2.cmml"><msub id="A5.SS1.p2.3.m3.2.3.2.2" xref="A5.SS1.p2.3.m3.2.3.2.2.cmml"><mi id="A5.SS1.p2.3.m3.2.3.2.2.2" xref="A5.SS1.p2.3.m3.2.3.2.2.2.cmml">r</mi><mi id="A5.SS1.p2.3.m3.2.3.2.2.3" xref="A5.SS1.p2.3.m3.2.3.2.2.3.cmml">ϕ</mi></msub><mo lspace="0em" rspace="0em" id="A5.SS1.p2.3.m3.2.3.2.1" xref="A5.SS1.p2.3.m3.2.3.2.1.cmml">​</mo><mrow id="A5.SS1.p2.3.m3.2.3.2.3.2" xref="A5.SS1.p2.3.m3.2.3.2.3.1.cmml"><mo stretchy="false" id="A5.SS1.p2.3.m3.2.3.2.3.2.1" xref="A5.SS1.p2.3.m3.2.3.2.3.1.cmml">(</mo><mi id="A5.SS1.p2.3.m3.1.1" xref="A5.SS1.p2.3.m3.1.1.cmml">x</mi><mo id="A5.SS1.p2.3.m3.2.3.2.3.2.2" xref="A5.SS1.p2.3.m3.2.3.2.3.1.cmml">,</mo><mi id="A5.SS1.p2.3.m3.2.2" xref="A5.SS1.p2.3.m3.2.2.cmml">y</mi><mo stretchy="false" id="A5.SS1.p2.3.m3.2.3.2.3.2.3" xref="A5.SS1.p2.3.m3.2.3.2.3.1.cmml">)</mo></mrow></mrow><mo id="A5.SS1.p2.3.m3.2.3.1" xref="A5.SS1.p2.3.m3.2.3.1.cmml">∈</mo><mi id="A5.SS1.p2.3.m3.2.3.3" xref="A5.SS1.p2.3.m3.2.3.3.cmml">ℝ</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.3.m3.2b"><apply id="A5.SS1.p2.3.m3.2.3.cmml" xref="A5.SS1.p2.3.m3.2.3"><in id="A5.SS1.p2.3.m3.2.3.1.cmml" xref="A5.SS1.p2.3.m3.2.3.1"></in><apply id="A5.SS1.p2.3.m3.2.3.2.cmml" xref="A5.SS1.p2.3.m3.2.3.2"><times id="A5.SS1.p2.3.m3.2.3.2.1.cmml" xref="A5.SS1.p2.3.m3.2.3.2.1"></times><apply id="A5.SS1.p2.3.m3.2.3.2.2.cmml" xref="A5.SS1.p2.3.m3.2.3.2.2"><csymbol cd="ambiguous" id="A5.SS1.p2.3.m3.2.3.2.2.1.cmml" xref="A5.SS1.p2.3.m3.2.3.2.2">subscript</csymbol><ci id="A5.SS1.p2.3.m3.2.3.2.2.2.cmml" xref="A5.SS1.p2.3.m3.2.3.2.2.2">𝑟</ci><ci id="A5.SS1.p2.3.m3.2.3.2.2.3.cmml" xref="A5.SS1.p2.3.m3.2.3.2.2.3">italic-ϕ</ci></apply><interval closure="open" id="A5.SS1.p2.3.m3.2.3.2.3.1.cmml" xref="A5.SS1.p2.3.m3.2.3.2.3.2"><ci id="A5.SS1.p2.3.m3.1.1.cmml" xref="A5.SS1.p2.3.m3.1.1">𝑥</ci><ci id="A5.SS1.p2.3.m3.2.2.cmml" xref="A5.SS1.p2.3.m3.2.2">𝑦</ci></interval></apply><ci id="A5.SS1.p2.3.m3.2.3.3.cmml" xref="A5.SS1.p2.3.m3.2.3.3">ℝ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.3.m3.2c">r_{\phi}(x,y)\in\mathbb{R}</annotation></semantics></math>. The four MLP heads are classification heads representing the instruction-following, verbosity, truthfulness, and harmlessness property of the response. We denote the output logits of the <math id="A5.SS1.p2.4.m4.1" class="ltx_Math" alttext="4" display="inline"><semantics id="A5.SS1.p2.4.m4.1a"><mn id="A5.SS1.p2.4.m4.1.1" xref="A5.SS1.p2.4.m4.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.4.m4.1b"><cn type="integer" id="A5.SS1.p2.4.m4.1.1.cmml" xref="A5.SS1.p2.4.m4.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.4.m4.1c">4</annotation></semantics></math> classification heads by <math id="A5.SS1.p2.5.m5.1" class="ltx_Math" alttext="u_{\phi}^{\text{if}}" display="inline"><semantics id="A5.SS1.p2.5.m5.1a"><msubsup id="A5.SS1.p2.5.m5.1.1" xref="A5.SS1.p2.5.m5.1.1.cmml"><mi id="A5.SS1.p2.5.m5.1.1.2.2" xref="A5.SS1.p2.5.m5.1.1.2.2.cmml">u</mi><mi id="A5.SS1.p2.5.m5.1.1.2.3" xref="A5.SS1.p2.5.m5.1.1.2.3.cmml">ϕ</mi><mtext id="A5.SS1.p2.5.m5.1.1.3" xref="A5.SS1.p2.5.m5.1.1.3a.cmml">if</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.5.m5.1b"><apply id="A5.SS1.p2.5.m5.1.1.cmml" xref="A5.SS1.p2.5.m5.1.1"><csymbol cd="ambiguous" id="A5.SS1.p2.5.m5.1.1.1.cmml" xref="A5.SS1.p2.5.m5.1.1">superscript</csymbol><apply id="A5.SS1.p2.5.m5.1.1.2.cmml" xref="A5.SS1.p2.5.m5.1.1"><csymbol cd="ambiguous" id="A5.SS1.p2.5.m5.1.1.2.1.cmml" xref="A5.SS1.p2.5.m5.1.1">subscript</csymbol><ci id="A5.SS1.p2.5.m5.1.1.2.2.cmml" xref="A5.SS1.p2.5.m5.1.1.2.2">𝑢</ci><ci id="A5.SS1.p2.5.m5.1.1.2.3.cmml" xref="A5.SS1.p2.5.m5.1.1.2.3">italic-ϕ</ci></apply><ci id="A5.SS1.p2.5.m5.1.1.3a.cmml" xref="A5.SS1.p2.5.m5.1.1.3"><mtext mathsize="70%" id="A5.SS1.p2.5.m5.1.1.3.cmml" xref="A5.SS1.p2.5.m5.1.1.3">if</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.5.m5.1c">u_{\phi}^{\text{if}}</annotation></semantics></math>, <math id="A5.SS1.p2.6.m6.1" class="ltx_Math" alttext="u_{\phi}^{\text{verb}}" display="inline"><semantics id="A5.SS1.p2.6.m6.1a"><msubsup id="A5.SS1.p2.6.m6.1.1" xref="A5.SS1.p2.6.m6.1.1.cmml"><mi id="A5.SS1.p2.6.m6.1.1.2.2" xref="A5.SS1.p2.6.m6.1.1.2.2.cmml">u</mi><mi id="A5.SS1.p2.6.m6.1.1.2.3" xref="A5.SS1.p2.6.m6.1.1.2.3.cmml">ϕ</mi><mtext id="A5.SS1.p2.6.m6.1.1.3" xref="A5.SS1.p2.6.m6.1.1.3a.cmml">verb</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.6.m6.1b"><apply id="A5.SS1.p2.6.m6.1.1.cmml" xref="A5.SS1.p2.6.m6.1.1"><csymbol cd="ambiguous" id="A5.SS1.p2.6.m6.1.1.1.cmml" xref="A5.SS1.p2.6.m6.1.1">superscript</csymbol><apply id="A5.SS1.p2.6.m6.1.1.2.cmml" xref="A5.SS1.p2.6.m6.1.1"><csymbol cd="ambiguous" id="A5.SS1.p2.6.m6.1.1.2.1.cmml" xref="A5.SS1.p2.6.m6.1.1">subscript</csymbol><ci id="A5.SS1.p2.6.m6.1.1.2.2.cmml" xref="A5.SS1.p2.6.m6.1.1.2.2">𝑢</ci><ci id="A5.SS1.p2.6.m6.1.1.2.3.cmml" xref="A5.SS1.p2.6.m6.1.1.2.3">italic-ϕ</ci></apply><ci id="A5.SS1.p2.6.m6.1.1.3a.cmml" xref="A5.SS1.p2.6.m6.1.1.3"><mtext mathsize="70%" id="A5.SS1.p2.6.m6.1.1.3.cmml" xref="A5.SS1.p2.6.m6.1.1.3">verb</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.6.m6.1c">u_{\phi}^{\text{verb}}</annotation></semantics></math>, <math id="A5.SS1.p2.7.m7.1" class="ltx_Math" alttext="u_{\phi}^{\text{truth}}" display="inline"><semantics id="A5.SS1.p2.7.m7.1a"><msubsup id="A5.SS1.p2.7.m7.1.1" xref="A5.SS1.p2.7.m7.1.1.cmml"><mi id="A5.SS1.p2.7.m7.1.1.2.2" xref="A5.SS1.p2.7.m7.1.1.2.2.cmml">u</mi><mi id="A5.SS1.p2.7.m7.1.1.2.3" xref="A5.SS1.p2.7.m7.1.1.2.3.cmml">ϕ</mi><mtext id="A5.SS1.p2.7.m7.1.1.3" xref="A5.SS1.p2.7.m7.1.1.3a.cmml">truth</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.7.m7.1b"><apply id="A5.SS1.p2.7.m7.1.1.cmml" xref="A5.SS1.p2.7.m7.1.1"><csymbol cd="ambiguous" id="A5.SS1.p2.7.m7.1.1.1.cmml" xref="A5.SS1.p2.7.m7.1.1">superscript</csymbol><apply id="A5.SS1.p2.7.m7.1.1.2.cmml" xref="A5.SS1.p2.7.m7.1.1"><csymbol cd="ambiguous" id="A5.SS1.p2.7.m7.1.1.2.1.cmml" xref="A5.SS1.p2.7.m7.1.1">subscript</csymbol><ci id="A5.SS1.p2.7.m7.1.1.2.2.cmml" xref="A5.SS1.p2.7.m7.1.1.2.2">𝑢</ci><ci id="A5.SS1.p2.7.m7.1.1.2.3.cmml" xref="A5.SS1.p2.7.m7.1.1.2.3">italic-ϕ</ci></apply><ci id="A5.SS1.p2.7.m7.1.1.3a.cmml" xref="A5.SS1.p2.7.m7.1.1.3"><mtext mathsize="70%" id="A5.SS1.p2.7.m7.1.1.3.cmml" xref="A5.SS1.p2.7.m7.1.1.3">truth</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.7.m7.1c">u_{\phi}^{\text{truth}}</annotation></semantics></math>, <math id="A5.SS1.p2.8.m8.1" class="ltx_Math" alttext="u_{\phi}^{\text{harm}}" display="inline"><semantics id="A5.SS1.p2.8.m8.1a"><msubsup id="A5.SS1.p2.8.m8.1.1" xref="A5.SS1.p2.8.m8.1.1.cmml"><mi id="A5.SS1.p2.8.m8.1.1.2.2" xref="A5.SS1.p2.8.m8.1.1.2.2.cmml">u</mi><mi id="A5.SS1.p2.8.m8.1.1.2.3" xref="A5.SS1.p2.8.m8.1.1.2.3.cmml">ϕ</mi><mtext id="A5.SS1.p2.8.m8.1.1.3" xref="A5.SS1.p2.8.m8.1.1.3a.cmml">harm</mtext></msubsup><annotation-xml encoding="MathML-Content" id="A5.SS1.p2.8.m8.1b"><apply id="A5.SS1.p2.8.m8.1.1.cmml" xref="A5.SS1.p2.8.m8.1.1"><csymbol cd="ambiguous" id="A5.SS1.p2.8.m8.1.1.1.cmml" xref="A5.SS1.p2.8.m8.1.1">superscript</csymbol><apply id="A5.SS1.p2.8.m8.1.1.2.cmml" xref="A5.SS1.p2.8.m8.1.1"><csymbol cd="ambiguous" id="A5.SS1.p2.8.m8.1.1.2.1.cmml" xref="A5.SS1.p2.8.m8.1.1">subscript</csymbol><ci id="A5.SS1.p2.8.m8.1.1.2.2.cmml" xref="A5.SS1.p2.8.m8.1.1.2.2">𝑢</ci><ci id="A5.SS1.p2.8.m8.1.1.2.3.cmml" xref="A5.SS1.p2.8.m8.1.1.2.3">italic-ϕ</ci></apply><ci id="A5.SS1.p2.8.m8.1.1.3a.cmml" xref="A5.SS1.p2.8.m8.1.1.3"><mtext mathsize="70%" id="A5.SS1.p2.8.m8.1.1.3.cmml" xref="A5.SS1.p2.8.m8.1.1.3">harm</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.p2.8.m8.1c">u_{\phi}^{\text{harm}}</annotation></semantics></math>, respectively.</p>
</div>
<section id="A5.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Soft label loss.</h5>

<div id="A5.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="A5.SS1.SSS0.Px1.p1.12" class="ltx_p">We train the preference reward <math id="A5.SS1.SSS0.Px1.p1.1.m1.2" class="ltx_Math" alttext="r_{\phi}(x,y)" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.1.m1.2a"><mrow id="A5.SS1.SSS0.Px1.p1.1.m1.2.3" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.cmml"><msub id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.cmml"><mi id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.2" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.2.cmml">r</mi><mi id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.3" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.3.cmml">ϕ</mi></msub><mo lspace="0em" rspace="0em" id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.1" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.1.cmml">​</mo><mrow id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.2" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.1.cmml"><mo stretchy="false" id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.2.1" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.1.cmml">(</mo><mi id="A5.SS1.SSS0.Px1.p1.1.m1.1.1" xref="A5.SS1.SSS0.Px1.p1.1.m1.1.1.cmml">x</mi><mo id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.2.2" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.1.cmml">,</mo><mi id="A5.SS1.SSS0.Px1.p1.1.m1.2.2" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.2.cmml">y</mi><mo stretchy="false" id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.2.3" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.1.m1.2b"><apply id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.cmml" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3"><times id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.1.cmml" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.1"></times><apply id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.cmml" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.1.cmml" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.2.cmml" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.2">𝑟</ci><ci id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.3.cmml" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.2.3">italic-ϕ</ci></apply><interval closure="open" id="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.1.cmml" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.3.3.2"><ci id="A5.SS1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.1.m1.1.1">𝑥</ci><ci id="A5.SS1.SSS0.Px1.p1.1.m1.2.2.cmml" xref="A5.SS1.SSS0.Px1.p1.1.m1.2.2">𝑦</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.1.m1.2c">r_{\phi}(x,y)</annotation></semantics></math> based on Bradley-Terry-Luce (BTL) model <cite class="ltx_cite ltx_citemacro_citep">[Bradley and Terry, <a href="#bib.bib10" title="" class="ltx_ref">1952</a>]</cite>. Recall that in BTL model, the probability that <math id="A5.SS1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="y_{c}" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.2.m2.1a"><msub id="A5.SS1.SSS0.Px1.p1.2.m2.1.1" xref="A5.SS1.SSS0.Px1.p1.2.m2.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.2.m2.1.1.2" xref="A5.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml">y</mi><mi id="A5.SS1.SSS0.Px1.p1.2.m2.1.1.3" xref="A5.SS1.SSS0.Px1.p1.2.m2.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.2.m2.1b"><apply id="A5.SS1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.2.m2.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.2.m2.1.1.2">𝑦</ci><ci id="A5.SS1.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.2.m2.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.2.m2.1c">y_{c}</annotation></semantics></math> is preferred over <math id="A5.SS1.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="y_{r}" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.3.m3.1a"><msub id="A5.SS1.SSS0.Px1.p1.3.m3.1.1" xref="A5.SS1.SSS0.Px1.p1.3.m3.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.3.m3.1.1.2" xref="A5.SS1.SSS0.Px1.p1.3.m3.1.1.2.cmml">y</mi><mi id="A5.SS1.SSS0.Px1.p1.3.m3.1.1.3" xref="A5.SS1.SSS0.Px1.p1.3.m3.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.3.m3.1b"><apply id="A5.SS1.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.3.m3.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.3.m3.1.1.2">𝑦</ci><ci id="A5.SS1.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.3.m3.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.3.m3.1c">y_{r}</annotation></semantics></math> is modeled as <math id="A5.SS1.SSS0.Px1.p1.4.m4.3" class="ltx_Math" alttext="\sigma(r_{\phi}(x,y_{c})-r_{\phi}(x,y_{r}))" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.4.m4.3a"><mrow id="A5.SS1.SSS0.Px1.p1.4.m4.3.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.cmml"><mi id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.2.cmml">​</mo><mrow id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.cmml"><mo stretchy="false" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.cmml">(</mo><mrow id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.cmml"><mrow id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.cmml"><msub id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.cmml"><mi id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.2.cmml">r</mi><mi id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.3.cmml">ϕ</mi></msub><mo lspace="0em" rspace="0em" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.2.cmml">​</mo><mrow id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.2.cmml">(</mo><mi id="A5.SS1.SSS0.Px1.p1.4.m4.1.1" xref="A5.SS1.SSS0.Px1.p1.4.m4.1.1.cmml">x</mi><mo id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.2.cmml">,</mo><msub id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.2.cmml">y</mi><mi id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.3.cmml">c</mi></msub><mo stretchy="false" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.4" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.3.cmml">−</mo><mrow id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.cmml"><msub id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.cmml"><mi id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.2.cmml">r</mi><mi id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.3.cmml">ϕ</mi></msub><mo lspace="0em" rspace="0em" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.2.cmml">​</mo><mrow id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.2.cmml"><mo stretchy="false" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.2.cmml">(</mo><mi id="A5.SS1.SSS0.Px1.p1.4.m4.2.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.2.2.cmml">x</mi><mo id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.2.cmml">,</mo><msub id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.2" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.2.cmml">y</mi><mi id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.3.cmml">r</mi></msub><mo stretchy="false" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.4" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.3" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.4.m4.3b"><apply id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3"><times id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.2"></times><ci id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.3.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.3">𝜎</ci><apply id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1"><minus id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.3"></minus><apply id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1"><times id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.2"></times><apply id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.1.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.2">𝑟</ci><ci id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.3.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.3.3">italic-ϕ</ci></apply><interval closure="open" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1"><ci id="A5.SS1.SSS0.Px1.p1.4.m4.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.1.1">𝑥</ci><apply id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.2">𝑦</ci><ci id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.1.1.1.1.3">𝑐</ci></apply></interval></apply><apply id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2"><times id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.2"></times><apply id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.1.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.2">𝑟</ci><ci id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.3.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.3.3">italic-ϕ</ci></apply><interval closure="open" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1"><ci id="A5.SS1.SSS0.Px1.p1.4.m4.2.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.2.2">𝑥</ci><apply id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.2">𝑦</ci><ci id="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.4.m4.3.3.1.1.1.2.1.1.1.3">𝑟</ci></apply></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.4.m4.3c">\sigma(r_{\phi}(x,y_{c})-r_{\phi}(x,y_{r}))</annotation></semantics></math>, where <math id="A5.SS1.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.5.m5.1a"><mi id="A5.SS1.SSS0.Px1.p1.5.m5.1.1" xref="A5.SS1.SSS0.Px1.p1.5.m5.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.5.m5.1b"><ci id="A5.SS1.SSS0.Px1.p1.5.m5.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.5.m5.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.5.m5.1c">\sigma</annotation></semantics></math> is the sigmoid function. Intuitively, this probability should be larger if the preferred response <math id="A5.SS1.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="y_{c}" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.6.m6.1a"><msub id="A5.SS1.SSS0.Px1.p1.6.m6.1.1" xref="A5.SS1.SSS0.Px1.p1.6.m6.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.6.m6.1.1.2" xref="A5.SS1.SSS0.Px1.p1.6.m6.1.1.2.cmml">y</mi><mi id="A5.SS1.SSS0.Px1.p1.6.m6.1.1.3" xref="A5.SS1.SSS0.Px1.p1.6.m6.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.6.m6.1b"><apply id="A5.SS1.SSS0.Px1.p1.6.m6.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.6.m6.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.6.m6.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.6.m6.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.6.m6.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.6.m6.1.1.2">𝑦</ci><ci id="A5.SS1.SSS0.Px1.p1.6.m6.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.6.m6.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.6.m6.1c">y_{c}</annotation></semantics></math> is annotated as significantly better than the rejected response <math id="A5.SS1.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="y_{r}" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.7.m7.1a"><msub id="A5.SS1.SSS0.Px1.p1.7.m7.1.1" xref="A5.SS1.SSS0.Px1.p1.7.m7.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.7.m7.1.1.2" xref="A5.SS1.SSS0.Px1.p1.7.m7.1.1.2.cmml">y</mi><mi id="A5.SS1.SSS0.Px1.p1.7.m7.1.1.3" xref="A5.SS1.SSS0.Px1.p1.7.m7.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.7.m7.1b"><apply id="A5.SS1.SSS0.Px1.p1.7.m7.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.7.m7.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.7.m7.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.7.m7.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.7.m7.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.7.m7.1.1.2">𝑦</ci><ci id="A5.SS1.SSS0.Px1.p1.7.m7.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.7.m7.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.7.m7.1c">y_{r}</annotation></semantics></math>, and smaller if <math id="A5.SS1.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="y_{c}" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.8.m8.1a"><msub id="A5.SS1.SSS0.Px1.p1.8.m8.1.1" xref="A5.SS1.SSS0.Px1.p1.8.m8.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.8.m8.1.1.2" xref="A5.SS1.SSS0.Px1.p1.8.m8.1.1.2.cmml">y</mi><mi id="A5.SS1.SSS0.Px1.p1.8.m8.1.1.3" xref="A5.SS1.SSS0.Px1.p1.8.m8.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.8.m8.1b"><apply id="A5.SS1.SSS0.Px1.p1.8.m8.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.8.m8.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.8.m8.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.8.m8.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.8.m8.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.8.m8.1.1.2">𝑦</ci><ci id="A5.SS1.SSS0.Px1.p1.8.m8.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.8.m8.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.8.m8.1c">y_{c}</annotation></semantics></math> is only negligibly better than <math id="A5.SS1.SSS0.Px1.p1.9.m9.1" class="ltx_Math" alttext="y_{r}" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.9.m9.1a"><msub id="A5.SS1.SSS0.Px1.p1.9.m9.1.1" xref="A5.SS1.SSS0.Px1.p1.9.m9.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.9.m9.1.1.2" xref="A5.SS1.SSS0.Px1.p1.9.m9.1.1.2.cmml">y</mi><mi id="A5.SS1.SSS0.Px1.p1.9.m9.1.1.3" xref="A5.SS1.SSS0.Px1.p1.9.m9.1.1.3.cmml">r</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.9.m9.1b"><apply id="A5.SS1.SSS0.Px1.p1.9.m9.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.9.m9.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.9.m9.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.9.m9.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.9.m9.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.9.m9.1.1.2">𝑦</ci><ci id="A5.SS1.SSS0.Px1.p1.9.m9.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.9.m9.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.9.m9.1c">y_{r}</annotation></semantics></math>. We incorporate this information using the preference level <math id="A5.SS1.SSS0.Px1.p1.10.m10.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.10.m10.1a"><mi mathvariant="normal" id="A5.SS1.SSS0.Px1.p1.10.m10.1.1" xref="A5.SS1.SSS0.Px1.p1.10.m10.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.10.m10.1b"><ci id="A5.SS1.SSS0.Px1.p1.10.m10.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.10.m10.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.10.m10.1c">\ell</annotation></semantics></math>. More specifically, for each preference level <math id="A5.SS1.SSS0.Px1.p1.11.m11.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.11.m11.1a"><mi mathvariant="normal" id="A5.SS1.SSS0.Px1.p1.11.m11.1.1" xref="A5.SS1.SSS0.Px1.p1.11.m11.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.11.m11.1b"><ci id="A5.SS1.SSS0.Px1.p1.11.m11.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.11.m11.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.11.m11.1c">\ell</annotation></semantics></math>, we design a <em id="A5.SS1.SSS0.Px1.p1.12.1" class="ltx_emph ltx_font_italic">target</em> preference probability <math id="A5.SS1.SSS0.Px1.p1.12.m12.1" class="ltx_Math" alttext="p_{\ell}" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.12.m12.1a"><msub id="A5.SS1.SSS0.Px1.p1.12.m12.1.1" xref="A5.SS1.SSS0.Px1.p1.12.m12.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.12.m12.1.1.2" xref="A5.SS1.SSS0.Px1.p1.12.m12.1.1.2.cmml">p</mi><mi mathvariant="normal" id="A5.SS1.SSS0.Px1.p1.12.m12.1.1.3" xref="A5.SS1.SSS0.Px1.p1.12.m12.1.1.3.cmml">ℓ</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.12.m12.1b"><apply id="A5.SS1.SSS0.Px1.p1.12.m12.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.12.m12.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.12.m12.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.12.m12.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.12.m12.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.12.m12.1.1.2">𝑝</ci><ci id="A5.SS1.SSS0.Px1.p1.12.m12.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.12.m12.1.1.3">ℓ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.12.m12.1c">p_{\ell}</annotation></semantics></math>. Then we use a <em id="A5.SS1.SSS0.Px1.p1.12.2" class="ltx_emph ltx_font_italic">soft label</em> loss as follows:</p>
<table id="A5.E3" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="A5.E3X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A5.E3X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle L_{\text{ranking}}(\phi)=" display="inline"><semantics id="A5.E3X.2.1.1.m1.1a"><mrow id="A5.E3X.2.1.1.m1.1.2" xref="A5.E3X.2.1.1.m1.1.2.cmml"><mrow id="A5.E3X.2.1.1.m1.1.2.2" xref="A5.E3X.2.1.1.m1.1.2.2.cmml"><msub id="A5.E3X.2.1.1.m1.1.2.2.2" xref="A5.E3X.2.1.1.m1.1.2.2.2.cmml"><mi id="A5.E3X.2.1.1.m1.1.2.2.2.2" xref="A5.E3X.2.1.1.m1.1.2.2.2.2.cmml">L</mi><mtext id="A5.E3X.2.1.1.m1.1.2.2.2.3" xref="A5.E3X.2.1.1.m1.1.2.2.2.3a.cmml">ranking</mtext></msub><mo lspace="0em" rspace="0em" id="A5.E3X.2.1.1.m1.1.2.2.1" xref="A5.E3X.2.1.1.m1.1.2.2.1.cmml">​</mo><mrow id="A5.E3X.2.1.1.m1.1.2.2.3.2" xref="A5.E3X.2.1.1.m1.1.2.2.cmml"><mo stretchy="false" id="A5.E3X.2.1.1.m1.1.2.2.3.2.1" xref="A5.E3X.2.1.1.m1.1.2.2.cmml">(</mo><mi id="A5.E3X.2.1.1.m1.1.1" xref="A5.E3X.2.1.1.m1.1.1.cmml">ϕ</mi><mo stretchy="false" id="A5.E3X.2.1.1.m1.1.2.2.3.2.2" xref="A5.E3X.2.1.1.m1.1.2.2.cmml">)</mo></mrow></mrow><mo id="A5.E3X.2.1.1.m1.1.2.1" xref="A5.E3X.2.1.1.m1.1.2.1.cmml">=</mo><mi id="A5.E3X.2.1.1.m1.1.2.3" xref="A5.E3X.2.1.1.m1.1.2.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A5.E3X.2.1.1.m1.1b"><apply id="A5.E3X.2.1.1.m1.1.2.cmml" xref="A5.E3X.2.1.1.m1.1.2"><eq id="A5.E3X.2.1.1.m1.1.2.1.cmml" xref="A5.E3X.2.1.1.m1.1.2.1"></eq><apply id="A5.E3X.2.1.1.m1.1.2.2.cmml" xref="A5.E3X.2.1.1.m1.1.2.2"><times id="A5.E3X.2.1.1.m1.1.2.2.1.cmml" xref="A5.E3X.2.1.1.m1.1.2.2.1"></times><apply id="A5.E3X.2.1.1.m1.1.2.2.2.cmml" xref="A5.E3X.2.1.1.m1.1.2.2.2"><csymbol cd="ambiguous" id="A5.E3X.2.1.1.m1.1.2.2.2.1.cmml" xref="A5.E3X.2.1.1.m1.1.2.2.2">subscript</csymbol><ci id="A5.E3X.2.1.1.m1.1.2.2.2.2.cmml" xref="A5.E3X.2.1.1.m1.1.2.2.2.2">𝐿</ci><ci id="A5.E3X.2.1.1.m1.1.2.2.2.3a.cmml" xref="A5.E3X.2.1.1.m1.1.2.2.2.3"><mtext mathsize="70%" id="A5.E3X.2.1.1.m1.1.2.2.2.3.cmml" xref="A5.E3X.2.1.1.m1.1.2.2.2.3">ranking</mtext></ci></apply><ci id="A5.E3X.2.1.1.m1.1.1.cmml" xref="A5.E3X.2.1.1.m1.1.1">italic-ϕ</ci></apply><csymbol cd="latexml" id="A5.E3X.2.1.1.m1.1.2.3.cmml" xref="A5.E3X.2.1.1.m1.1.2.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.E3X.2.1.1.m1.1c">\displaystyle L_{\text{ranking}}(\phi)=</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A5.E3X.3.2.2.m1.2" class="ltx_math_unparsed" alttext="\displaystyle-p_{\ell}\log(\sigma(r_{\phi}(x,y_{c})-r_{\phi}(x,y_{r}))" display="inline"><semantics id="A5.E3X.3.2.2.m1.2a"><mrow id="A5.E3X.3.2.2.m1.2b"><mo id="A5.E3X.3.2.2.m1.2.3">−</mo><msub id="A5.E3X.3.2.2.m1.2.4"><mi id="A5.E3X.3.2.2.m1.2.4.2">p</mi><mi mathvariant="normal" id="A5.E3X.3.2.2.m1.2.4.3">ℓ</mi></msub><mi id="A5.E3X.3.2.2.m1.2.5">log</mi><mrow id="A5.E3X.3.2.2.m1.2.6"><mo stretchy="false" id="A5.E3X.3.2.2.m1.2.6.1">(</mo><mi id="A5.E3X.3.2.2.m1.2.6.2">σ</mi><mrow id="A5.E3X.3.2.2.m1.2.6.3"><mo stretchy="false" id="A5.E3X.3.2.2.m1.2.6.3.1">(</mo><msub id="A5.E3X.3.2.2.m1.2.6.3.2"><mi id="A5.E3X.3.2.2.m1.2.6.3.2.2">r</mi><mi id="A5.E3X.3.2.2.m1.2.6.3.2.3">ϕ</mi></msub><mrow id="A5.E3X.3.2.2.m1.2.6.3.3"><mo stretchy="false" id="A5.E3X.3.2.2.m1.2.6.3.3.1">(</mo><mi id="A5.E3X.3.2.2.m1.1.1">x</mi><mo id="A5.E3X.3.2.2.m1.2.6.3.3.2">,</mo><msub id="A5.E3X.3.2.2.m1.2.6.3.3.3"><mi id="A5.E3X.3.2.2.m1.2.6.3.3.3.2">y</mi><mi id="A5.E3X.3.2.2.m1.2.6.3.3.3.3">c</mi></msub><mo stretchy="false" id="A5.E3X.3.2.2.m1.2.6.3.3.4">)</mo></mrow><mo id="A5.E3X.3.2.2.m1.2.6.3.4">−</mo><msub id="A5.E3X.3.2.2.m1.2.6.3.5"><mi id="A5.E3X.3.2.2.m1.2.6.3.5.2">r</mi><mi id="A5.E3X.3.2.2.m1.2.6.3.5.3">ϕ</mi></msub><mrow id="A5.E3X.3.2.2.m1.2.6.3.6"><mo stretchy="false" id="A5.E3X.3.2.2.m1.2.6.3.6.1">(</mo><mi id="A5.E3X.3.2.2.m1.2.2">x</mi><mo id="A5.E3X.3.2.2.m1.2.6.3.6.2">,</mo><msub id="A5.E3X.3.2.2.m1.2.6.3.6.3"><mi id="A5.E3X.3.2.2.m1.2.6.3.6.3.2">y</mi><mi id="A5.E3X.3.2.2.m1.2.6.3.6.3.3">r</mi></msub><mo stretchy="false" id="A5.E3X.3.2.2.m1.2.6.3.6.4">)</mo></mrow><mo stretchy="false" id="A5.E3X.3.2.2.m1.2.6.3.7">)</mo></mrow></mrow></mrow><annotation encoding="application/x-tex" id="A5.E3X.3.2.2.m1.2c">\displaystyle-p_{\ell}\log(\sigma(r_{\phi}(x,y_{c})-r_{\phi}(x,y_{r}))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(3)</span></td>
</tr>
<tr id="A5.E3Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A5.E3Xa.2.1.1.m1.2" class="ltx_math_unparsed" alttext="\displaystyle-(1-p_{\ell})\log(\sigma(r_{\phi}(x,y_{r})-r_{\phi}(x,y_{c}))." display="inline"><semantics id="A5.E3Xa.2.1.1.m1.2a"><mrow id="A5.E3Xa.2.1.1.m1.2b"><mo id="A5.E3Xa.2.1.1.m1.2.3">−</mo><mrow id="A5.E3Xa.2.1.1.m1.2.4"><mo stretchy="false" id="A5.E3Xa.2.1.1.m1.2.4.1">(</mo><mn id="A5.E3Xa.2.1.1.m1.2.4.2">1</mn><mo id="A5.E3Xa.2.1.1.m1.2.4.3">−</mo><msub id="A5.E3Xa.2.1.1.m1.2.4.4"><mi id="A5.E3Xa.2.1.1.m1.2.4.4.2">p</mi><mi mathvariant="normal" id="A5.E3Xa.2.1.1.m1.2.4.4.3">ℓ</mi></msub><mo rspace="0.167em" stretchy="false" id="A5.E3Xa.2.1.1.m1.2.4.5">)</mo></mrow><mi id="A5.E3Xa.2.1.1.m1.2.5">log</mi><mrow id="A5.E3Xa.2.1.1.m1.2.6"><mo stretchy="false" id="A5.E3Xa.2.1.1.m1.2.6.1">(</mo><mi id="A5.E3Xa.2.1.1.m1.2.6.2">σ</mi><mrow id="A5.E3Xa.2.1.1.m1.2.6.3"><mo stretchy="false" id="A5.E3Xa.2.1.1.m1.2.6.3.1">(</mo><msub id="A5.E3Xa.2.1.1.m1.2.6.3.2"><mi id="A5.E3Xa.2.1.1.m1.2.6.3.2.2">r</mi><mi id="A5.E3Xa.2.1.1.m1.2.6.3.2.3">ϕ</mi></msub><mrow id="A5.E3Xa.2.1.1.m1.2.6.3.3"><mo stretchy="false" id="A5.E3Xa.2.1.1.m1.2.6.3.3.1">(</mo><mi id="A5.E3Xa.2.1.1.m1.1.1">x</mi><mo id="A5.E3Xa.2.1.1.m1.2.6.3.3.2">,</mo><msub id="A5.E3Xa.2.1.1.m1.2.6.3.3.3"><mi id="A5.E3Xa.2.1.1.m1.2.6.3.3.3.2">y</mi><mi id="A5.E3Xa.2.1.1.m1.2.6.3.3.3.3">r</mi></msub><mo stretchy="false" id="A5.E3Xa.2.1.1.m1.2.6.3.3.4">)</mo></mrow><mo id="A5.E3Xa.2.1.1.m1.2.6.3.4">−</mo><msub id="A5.E3Xa.2.1.1.m1.2.6.3.5"><mi id="A5.E3Xa.2.1.1.m1.2.6.3.5.2">r</mi><mi id="A5.E3Xa.2.1.1.m1.2.6.3.5.3">ϕ</mi></msub><mrow id="A5.E3Xa.2.1.1.m1.2.6.3.6"><mo stretchy="false" id="A5.E3Xa.2.1.1.m1.2.6.3.6.1">(</mo><mi id="A5.E3Xa.2.1.1.m1.2.2">x</mi><mo id="A5.E3Xa.2.1.1.m1.2.6.3.6.2">,</mo><msub id="A5.E3Xa.2.1.1.m1.2.6.3.6.3"><mi id="A5.E3Xa.2.1.1.m1.2.6.3.6.3.2">y</mi><mi id="A5.E3Xa.2.1.1.m1.2.6.3.6.3.3">c</mi></msub><mo stretchy="false" id="A5.E3Xa.2.1.1.m1.2.6.3.6.4">)</mo></mrow><mo stretchy="false" id="A5.E3Xa.2.1.1.m1.2.6.3.7">)</mo></mrow><mo lspace="0em" id="A5.E3Xa.2.1.1.m1.2.6.4">.</mo></mrow></mrow><annotation encoding="application/x-tex" id="A5.E3Xa.2.1.1.m1.2c">\displaystyle-(1-p_{\ell})\log(\sigma(r_{\phi}(x,y_{r})-r_{\phi}(x,y_{c})).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="A5.SS1.SSS0.Px1.p1.14" class="ltx_p">The target level <math id="A5.SS1.SSS0.Px1.p1.13.m1.1" class="ltx_Math" alttext="p_{\ell}" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.13.m1.1a"><msub id="A5.SS1.SSS0.Px1.p1.13.m1.1.1" xref="A5.SS1.SSS0.Px1.p1.13.m1.1.1.cmml"><mi id="A5.SS1.SSS0.Px1.p1.13.m1.1.1.2" xref="A5.SS1.SSS0.Px1.p1.13.m1.1.1.2.cmml">p</mi><mi mathvariant="normal" id="A5.SS1.SSS0.Px1.p1.13.m1.1.1.3" xref="A5.SS1.SSS0.Px1.p1.13.m1.1.1.3.cmml">ℓ</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.13.m1.1b"><apply id="A5.SS1.SSS0.Px1.p1.13.m1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.13.m1.1.1"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.13.m1.1.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.13.m1.1.1">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.13.m1.1.1.2.cmml" xref="A5.SS1.SSS0.Px1.p1.13.m1.1.1.2">𝑝</ci><ci id="A5.SS1.SSS0.Px1.p1.13.m1.1.1.3.cmml" xref="A5.SS1.SSS0.Px1.p1.13.m1.1.1.3">ℓ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.13.m1.1c">p_{\ell}</annotation></semantics></math> is a hyperparameter in our algorithm and should take larger value if the preference level is higher. In our experiments, we choose <math id="A5.SS1.SSS0.Px1.p1.14.m2.4" class="ltx_Math" alttext="p_{\ell}=0.95,~{}0.85,~{}0.75,~{}0.65" display="inline"><semantics id="A5.SS1.SSS0.Px1.p1.14.m2.4a"><mrow id="A5.SS1.SSS0.Px1.p1.14.m2.4.5" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.cmml"><msub id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.cmml"><mi id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.2" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.2.cmml">p</mi><mi mathvariant="normal" id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.3" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.3.cmml">ℓ</mi></msub><mo id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.1" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.1.cmml">=</mo><mrow id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.2" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.1.cmml"><mn id="A5.SS1.SSS0.Px1.p1.14.m2.1.1" xref="A5.SS1.SSS0.Px1.p1.14.m2.1.1.cmml">0.95</mn><mo rspace="0.497em" id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.2.1" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.1.cmml">,</mo><mn id="A5.SS1.SSS0.Px1.p1.14.m2.2.2" xref="A5.SS1.SSS0.Px1.p1.14.m2.2.2.cmml">0.85</mn><mo rspace="0.497em" id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.2.2" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.1.cmml">,</mo><mn id="A5.SS1.SSS0.Px1.p1.14.m2.3.3" xref="A5.SS1.SSS0.Px1.p1.14.m2.3.3.cmml">0.75</mn><mo rspace="0.497em" id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.2.3" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.1.cmml">,</mo><mn id="A5.SS1.SSS0.Px1.p1.14.m2.4.4" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.4.cmml">0.65</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS1.SSS0.Px1.p1.14.m2.4b"><apply id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5"><eq id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.1.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.1"></eq><apply id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2"><csymbol cd="ambiguous" id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.1.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2">subscript</csymbol><ci id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.2.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.2">𝑝</ci><ci id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.3.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.2.3">ℓ</ci></apply><list id="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.1.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.5.3.2"><cn type="float" id="A5.SS1.SSS0.Px1.p1.14.m2.1.1.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.1.1">0.95</cn><cn type="float" id="A5.SS1.SSS0.Px1.p1.14.m2.2.2.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.2.2">0.85</cn><cn type="float" id="A5.SS1.SSS0.Px1.p1.14.m2.3.3.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.3.3">0.75</cn><cn type="float" id="A5.SS1.SSS0.Px1.p1.14.m2.4.4.cmml" xref="A5.SS1.SSS0.Px1.p1.14.m2.4.4">0.65</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS1.SSS0.Px1.p1.14.m2.4c">p_{\ell}=0.95,~{}0.85,~{}0.75,~{}0.65</annotation></semantics></math> for significantly better, better, slightly better, and negligibly better, respectively.</p>
</div>
</section>
<section id="A5.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Single-sided grading as regularization.</h5>

<div id="A5.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="A5.SS1.SSS0.Px2.p1.1" class="ltx_p">We also leverage the single-sided gradings as regularization terms in our reward model. The intuition is that with these gradings as regularization terms, we can learn a better embedding to capture human preferences. The regularization loss is</p>
<table id="A5.E4" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="A5.E4X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A5.E4X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle L_{\text{regu}}(\phi)=" display="inline"><semantics id="A5.E4X.2.1.1.m1.1a"><mrow id="A5.E4X.2.1.1.m1.1.2" xref="A5.E4X.2.1.1.m1.1.2.cmml"><mrow id="A5.E4X.2.1.1.m1.1.2.2" xref="A5.E4X.2.1.1.m1.1.2.2.cmml"><msub id="A5.E4X.2.1.1.m1.1.2.2.2" xref="A5.E4X.2.1.1.m1.1.2.2.2.cmml"><mi id="A5.E4X.2.1.1.m1.1.2.2.2.2" xref="A5.E4X.2.1.1.m1.1.2.2.2.2.cmml">L</mi><mtext id="A5.E4X.2.1.1.m1.1.2.2.2.3" xref="A5.E4X.2.1.1.m1.1.2.2.2.3a.cmml">regu</mtext></msub><mo lspace="0em" rspace="0em" id="A5.E4X.2.1.1.m1.1.2.2.1" xref="A5.E4X.2.1.1.m1.1.2.2.1.cmml">​</mo><mrow id="A5.E4X.2.1.1.m1.1.2.2.3.2" xref="A5.E4X.2.1.1.m1.1.2.2.cmml"><mo stretchy="false" id="A5.E4X.2.1.1.m1.1.2.2.3.2.1" xref="A5.E4X.2.1.1.m1.1.2.2.cmml">(</mo><mi id="A5.E4X.2.1.1.m1.1.1" xref="A5.E4X.2.1.1.m1.1.1.cmml">ϕ</mi><mo stretchy="false" id="A5.E4X.2.1.1.m1.1.2.2.3.2.2" xref="A5.E4X.2.1.1.m1.1.2.2.cmml">)</mo></mrow></mrow><mo id="A5.E4X.2.1.1.m1.1.2.1" xref="A5.E4X.2.1.1.m1.1.2.1.cmml">=</mo><mi id="A5.E4X.2.1.1.m1.1.2.3" xref="A5.E4X.2.1.1.m1.1.2.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A5.E4X.2.1.1.m1.1b"><apply id="A5.E4X.2.1.1.m1.1.2.cmml" xref="A5.E4X.2.1.1.m1.1.2"><eq id="A5.E4X.2.1.1.m1.1.2.1.cmml" xref="A5.E4X.2.1.1.m1.1.2.1"></eq><apply id="A5.E4X.2.1.1.m1.1.2.2.cmml" xref="A5.E4X.2.1.1.m1.1.2.2"><times id="A5.E4X.2.1.1.m1.1.2.2.1.cmml" xref="A5.E4X.2.1.1.m1.1.2.2.1"></times><apply id="A5.E4X.2.1.1.m1.1.2.2.2.cmml" xref="A5.E4X.2.1.1.m1.1.2.2.2"><csymbol cd="ambiguous" id="A5.E4X.2.1.1.m1.1.2.2.2.1.cmml" xref="A5.E4X.2.1.1.m1.1.2.2.2">subscript</csymbol><ci id="A5.E4X.2.1.1.m1.1.2.2.2.2.cmml" xref="A5.E4X.2.1.1.m1.1.2.2.2.2">𝐿</ci><ci id="A5.E4X.2.1.1.m1.1.2.2.2.3a.cmml" xref="A5.E4X.2.1.1.m1.1.2.2.2.3"><mtext mathsize="70%" id="A5.E4X.2.1.1.m1.1.2.2.2.3.cmml" xref="A5.E4X.2.1.1.m1.1.2.2.2.3">regu</mtext></ci></apply><ci id="A5.E4X.2.1.1.m1.1.1.cmml" xref="A5.E4X.2.1.1.m1.1.1">italic-ϕ</ci></apply><csymbol cd="latexml" id="A5.E4X.2.1.1.m1.1.2.3.cmml" xref="A5.E4X.2.1.1.m1.1.2.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.E4X.2.1.1.m1.1c">\displaystyle L_{\text{regu}}(\phi)=</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A5.E4X.3.2.2.m1.1" class="ltx_math_unparsed" alttext="\displaystyle\sum_{\text{grade}\in{\text{if,verb,truth,harm}}}\Big{(}\mathsf{cross\_entropy}(u_{\phi}^{\text{grade}}(x,y_{c}),z^{\text{grade}}_{c})" display="inline"><semantics id="A5.E4X.3.2.2.m1.1a"><mrow id="A5.E4X.3.2.2.m1.1b"><mstyle displaystyle="true" id="A5.E4X.3.2.2.m1.1.2"><munder id="A5.E4X.3.2.2.m1.1.2a"><mo movablelimits="false" id="A5.E4X.3.2.2.m1.1.2.2">∑</mo><mrow id="A5.E4X.3.2.2.m1.1.2.3"><mtext id="A5.E4X.3.2.2.m1.1.2.3.2">grade</mtext><mo id="A5.E4X.3.2.2.m1.1.2.3.1">∈</mo><mtext id="A5.E4X.3.2.2.m1.1.2.3.3">if,verb,truth,harm</mtext></mrow></munder></mstyle><mrow id="A5.E4X.3.2.2.m1.1.3"><mo maxsize="160%" minsize="160%" id="A5.E4X.3.2.2.m1.1.3.1">(</mo><mi id="A5.E4X.3.2.2.m1.1.3.2">𝖼𝗋𝗈𝗌𝗌</mi><mi mathvariant="normal" id="A5.E4X.3.2.2.m1.1.3.3">_</mi><mi id="A5.E4X.3.2.2.m1.1.3.4">𝖾𝗇𝗍𝗋𝗈𝗉𝗒</mi><mrow id="A5.E4X.3.2.2.m1.1.3.5"><mo stretchy="false" id="A5.E4X.3.2.2.m1.1.3.5.1">(</mo><msubsup id="A5.E4X.3.2.2.m1.1.3.5.2"><mi id="A5.E4X.3.2.2.m1.1.3.5.2.2.2">u</mi><mi id="A5.E4X.3.2.2.m1.1.3.5.2.2.3">ϕ</mi><mtext id="A5.E4X.3.2.2.m1.1.3.5.2.3">grade</mtext></msubsup><mrow id="A5.E4X.3.2.2.m1.1.3.5.3"><mo stretchy="false" id="A5.E4X.3.2.2.m1.1.3.5.3.1">(</mo><mi id="A5.E4X.3.2.2.m1.1.1">x</mi><mo id="A5.E4X.3.2.2.m1.1.3.5.3.2">,</mo><msub id="A5.E4X.3.2.2.m1.1.3.5.3.3"><mi id="A5.E4X.3.2.2.m1.1.3.5.3.3.2">y</mi><mi id="A5.E4X.3.2.2.m1.1.3.5.3.3.3">c</mi></msub><mo stretchy="false" id="A5.E4X.3.2.2.m1.1.3.5.3.4">)</mo></mrow><mo id="A5.E4X.3.2.2.m1.1.3.5.4">,</mo><msubsup id="A5.E4X.3.2.2.m1.1.3.5.5"><mi id="A5.E4X.3.2.2.m1.1.3.5.5.2.2">z</mi><mi id="A5.E4X.3.2.2.m1.1.3.5.5.3">c</mi><mtext id="A5.E4X.3.2.2.m1.1.3.5.5.2.3">grade</mtext></msubsup><mo stretchy="false" id="A5.E4X.3.2.2.m1.1.3.5.6">)</mo></mrow></mrow></mrow><annotation encoding="application/x-tex" id="A5.E4X.3.2.2.m1.1c">\displaystyle\sum_{\text{grade}\in{\text{if,verb,truth,harm}}}\Big{(}\mathsf{cross\_entropy}(u_{\phi}^{\text{grade}}(x,y_{c}),z^{\text{grade}}_{c})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(4)</span></td>
</tr>
<tr id="A5.E4Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A5.E4Xa.2.1.1.m1.1" class="ltx_math_unparsed" alttext="\displaystyle+\mathsf{cross\_entropy}(u_{\phi}^{\text{grade}}(x,y_{r}),z^{\text{grade}}_{r})\Big{)}." display="inline"><semantics id="A5.E4Xa.2.1.1.m1.1a"><mrow id="A5.E4Xa.2.1.1.m1.1b"><mo id="A5.E4Xa.2.1.1.m1.1.2">+</mo><mi id="A5.E4Xa.2.1.1.m1.1.3">𝖼𝗋𝗈𝗌𝗌</mi><mi mathvariant="normal" id="A5.E4Xa.2.1.1.m1.1.4">_</mi><mi id="A5.E4Xa.2.1.1.m1.1.5">𝖾𝗇𝗍𝗋𝗈𝗉𝗒</mi><mrow id="A5.E4Xa.2.1.1.m1.1.6"><mo stretchy="false" id="A5.E4Xa.2.1.1.m1.1.6.1">(</mo><msubsup id="A5.E4Xa.2.1.1.m1.1.6.2"><mi id="A5.E4Xa.2.1.1.m1.1.6.2.2.2">u</mi><mi id="A5.E4Xa.2.1.1.m1.1.6.2.2.3">ϕ</mi><mtext id="A5.E4Xa.2.1.1.m1.1.6.2.3">grade</mtext></msubsup><mrow id="A5.E4Xa.2.1.1.m1.1.6.3"><mo stretchy="false" id="A5.E4Xa.2.1.1.m1.1.6.3.1">(</mo><mi id="A5.E4Xa.2.1.1.m1.1.1">x</mi><mo id="A5.E4Xa.2.1.1.m1.1.6.3.2">,</mo><msub id="A5.E4Xa.2.1.1.m1.1.6.3.3"><mi id="A5.E4Xa.2.1.1.m1.1.6.3.3.2">y</mi><mi id="A5.E4Xa.2.1.1.m1.1.6.3.3.3">r</mi></msub><mo stretchy="false" id="A5.E4Xa.2.1.1.m1.1.6.3.4">)</mo></mrow><mo id="A5.E4Xa.2.1.1.m1.1.6.4">,</mo><msubsup id="A5.E4Xa.2.1.1.m1.1.6.5"><mi id="A5.E4Xa.2.1.1.m1.1.6.5.2.2">z</mi><mi id="A5.E4Xa.2.1.1.m1.1.6.5.3">r</mi><mtext id="A5.E4Xa.2.1.1.m1.1.6.5.2.3">grade</mtext></msubsup><mo stretchy="false" id="A5.E4Xa.2.1.1.m1.1.6.6">)</mo></mrow><mo maxsize="160%" minsize="160%" id="A5.E4Xa.2.1.1.m1.1.7">)</mo><mo lspace="0em" id="A5.E4Xa.2.1.1.m1.1.8">.</mo></mrow><annotation encoding="application/x-tex" id="A5.E4Xa.2.1.1.m1.1c">\displaystyle+\mathsf{cross\_entropy}(u_{\phi}^{\text{grade}}(x,y_{r}),z^{\text{grade}}_{r})\Big{)}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="A5.SS1.SSS0.Px2.p1.2" class="ltx_p">Overall, the reward model training loss that we use is</p>
<table id="A6.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A5.E5"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A5.E5.m1.3" class="ltx_Math" alttext="\displaystyle L_{\text{ranking}}(\phi)+\lambda L_{\text{regu}}(\phi)." display="inline"><semantics id="A5.E5.m1.3a"><mrow id="A5.E5.m1.3.3.1" xref="A5.E5.m1.3.3.1.1.cmml"><mrow id="A5.E5.m1.3.3.1.1" xref="A5.E5.m1.3.3.1.1.cmml"><mrow id="A5.E5.m1.3.3.1.1.2" xref="A5.E5.m1.3.3.1.1.2.cmml"><msub id="A5.E5.m1.3.3.1.1.2.2" xref="A5.E5.m1.3.3.1.1.2.2.cmml"><mi id="A5.E5.m1.3.3.1.1.2.2.2" xref="A5.E5.m1.3.3.1.1.2.2.2.cmml">L</mi><mtext id="A5.E5.m1.3.3.1.1.2.2.3" xref="A5.E5.m1.3.3.1.1.2.2.3a.cmml">ranking</mtext></msub><mo lspace="0em" rspace="0em" id="A5.E5.m1.3.3.1.1.2.1" xref="A5.E5.m1.3.3.1.1.2.1.cmml">​</mo><mrow id="A5.E5.m1.3.3.1.1.2.3.2" xref="A5.E5.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="A5.E5.m1.3.3.1.1.2.3.2.1" xref="A5.E5.m1.3.3.1.1.2.cmml">(</mo><mi id="A5.E5.m1.1.1" xref="A5.E5.m1.1.1.cmml">ϕ</mi><mo stretchy="false" id="A5.E5.m1.3.3.1.1.2.3.2.2" xref="A5.E5.m1.3.3.1.1.2.cmml">)</mo></mrow></mrow><mo id="A5.E5.m1.3.3.1.1.1" xref="A5.E5.m1.3.3.1.1.1.cmml">+</mo><mrow id="A5.E5.m1.3.3.1.1.3" xref="A5.E5.m1.3.3.1.1.3.cmml"><mi id="A5.E5.m1.3.3.1.1.3.2" xref="A5.E5.m1.3.3.1.1.3.2.cmml">λ</mi><mo lspace="0em" rspace="0em" id="A5.E5.m1.3.3.1.1.3.1" xref="A5.E5.m1.3.3.1.1.3.1.cmml">​</mo><msub id="A5.E5.m1.3.3.1.1.3.3" xref="A5.E5.m1.3.3.1.1.3.3.cmml"><mi id="A5.E5.m1.3.3.1.1.3.3.2" xref="A5.E5.m1.3.3.1.1.3.3.2.cmml">L</mi><mtext id="A5.E5.m1.3.3.1.1.3.3.3" xref="A5.E5.m1.3.3.1.1.3.3.3a.cmml">regu</mtext></msub><mo lspace="0em" rspace="0em" id="A5.E5.m1.3.3.1.1.3.1a" xref="A5.E5.m1.3.3.1.1.3.1.cmml">​</mo><mrow id="A5.E5.m1.3.3.1.1.3.4.2" xref="A5.E5.m1.3.3.1.1.3.cmml"><mo stretchy="false" id="A5.E5.m1.3.3.1.1.3.4.2.1" xref="A5.E5.m1.3.3.1.1.3.cmml">(</mo><mi id="A5.E5.m1.2.2" xref="A5.E5.m1.2.2.cmml">ϕ</mi><mo stretchy="false" id="A5.E5.m1.3.3.1.1.3.4.2.2" xref="A5.E5.m1.3.3.1.1.3.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="A5.E5.m1.3.3.1.2" xref="A5.E5.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A5.E5.m1.3b"><apply id="A5.E5.m1.3.3.1.1.cmml" xref="A5.E5.m1.3.3.1"><plus id="A5.E5.m1.3.3.1.1.1.cmml" xref="A5.E5.m1.3.3.1.1.1"></plus><apply id="A5.E5.m1.3.3.1.1.2.cmml" xref="A5.E5.m1.3.3.1.1.2"><times id="A5.E5.m1.3.3.1.1.2.1.cmml" xref="A5.E5.m1.3.3.1.1.2.1"></times><apply id="A5.E5.m1.3.3.1.1.2.2.cmml" xref="A5.E5.m1.3.3.1.1.2.2"><csymbol cd="ambiguous" id="A5.E5.m1.3.3.1.1.2.2.1.cmml" xref="A5.E5.m1.3.3.1.1.2.2">subscript</csymbol><ci id="A5.E5.m1.3.3.1.1.2.2.2.cmml" xref="A5.E5.m1.3.3.1.1.2.2.2">𝐿</ci><ci id="A5.E5.m1.3.3.1.1.2.2.3a.cmml" xref="A5.E5.m1.3.3.1.1.2.2.3"><mtext mathsize="70%" id="A5.E5.m1.3.3.1.1.2.2.3.cmml" xref="A5.E5.m1.3.3.1.1.2.2.3">ranking</mtext></ci></apply><ci id="A5.E5.m1.1.1.cmml" xref="A5.E5.m1.1.1">italic-ϕ</ci></apply><apply id="A5.E5.m1.3.3.1.1.3.cmml" xref="A5.E5.m1.3.3.1.1.3"><times id="A5.E5.m1.3.3.1.1.3.1.cmml" xref="A5.E5.m1.3.3.1.1.3.1"></times><ci id="A5.E5.m1.3.3.1.1.3.2.cmml" xref="A5.E5.m1.3.3.1.1.3.2">𝜆</ci><apply id="A5.E5.m1.3.3.1.1.3.3.cmml" xref="A5.E5.m1.3.3.1.1.3.3"><csymbol cd="ambiguous" id="A5.E5.m1.3.3.1.1.3.3.1.cmml" xref="A5.E5.m1.3.3.1.1.3.3">subscript</csymbol><ci id="A5.E5.m1.3.3.1.1.3.3.2.cmml" xref="A5.E5.m1.3.3.1.1.3.3.2">𝐿</ci><ci id="A5.E5.m1.3.3.1.1.3.3.3a.cmml" xref="A5.E5.m1.3.3.1.1.3.3.3"><mtext mathsize="70%" id="A5.E5.m1.3.3.1.1.3.3.3.cmml" xref="A5.E5.m1.3.3.1.1.3.3.3">regu</mtext></ci></apply><ci id="A5.E5.m1.2.2.cmml" xref="A5.E5.m1.2.2">italic-ϕ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.E5.m1.3c">\displaystyle L_{\text{ranking}}(\phi)+\lambda L_{\text{regu}}(\phi).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
</div>
</section>
</section>
<section id="A5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection"><span id="A5.SS2.2.1.1" class="ltx_text ltx_font_sansserif">E.2</span> </span><span id="A5.SS2.3.2" class="ltx_text ltx_font_sansserif">Online RL algorithm</span>
</h3>

<div id="A5.SS2.p1" class="ltx_para">
<p id="A5.SS2.p1.1" class="ltx_p">In this section, we present more details of our online RLHF algorithm, MDLOO.</p>
</div>
<section id="A5.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Leave-One-Out (LOO) estimator of the advantage.</h5>

<div id="A5.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="A5.SS2.SSS0.Px1.p1.9" class="ltx_p">In each iteration of the algorithm, we have a data collection stage and a policy updating stage. Let <math id="A5.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\theta_{k}" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.1.m1.1a"><msub id="A5.SS2.SSS0.Px1.p1.1.m1.1.1" xref="A5.SS2.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="A5.SS2.SSS0.Px1.p1.1.m1.1.1.2" xref="A5.SS2.SSS0.Px1.p1.1.m1.1.1.2.cmml">θ</mi><mi id="A5.SS2.SSS0.Px1.p1.1.m1.1.1.3" xref="A5.SS2.SSS0.Px1.p1.1.m1.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.1.m1.1b"><apply id="A5.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.1.m1.1.1">subscript</csymbol><ci id="A5.SS2.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="A5.SS2.SSS0.Px1.p1.1.m1.1.1.2">𝜃</ci><ci id="A5.SS2.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="A5.SS2.SSS0.Px1.p1.1.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.1.m1.1c">\theta_{k}</annotation></semantics></math> be the model parameter at the beginning of the <math id="A5.SS2.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.2.m2.1a"><mi id="A5.SS2.SSS0.Px1.p1.2.m2.1.1" xref="A5.SS2.SSS0.Px1.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.2.m2.1b"><ci id="A5.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.2.m2.1c">k</annotation></semantics></math>-th iteration. We sample a batch of <math id="A5.SS2.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.3.m3.1a"><mi id="A5.SS2.SSS0.Px1.p1.3.m3.1.1" xref="A5.SS2.SSS0.Px1.p1.3.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.3.m3.1b"><ci id="A5.SS2.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.3.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.3.m3.1c">n</annotation></semantics></math> prompts from our prompt set, and for each prompt, we sample <math id="A5.SS2.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="K" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.4.m4.1a"><mi id="A5.SS2.SSS0.Px1.p1.4.m4.1.1" xref="A5.SS2.SSS0.Px1.p1.4.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.4.m4.1b"><ci id="A5.SS2.SSS0.Px1.p1.4.m4.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.4.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.4.m4.1c">K</annotation></semantics></math> responses according to the policy <math id="A5.SS2.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="\pi_{\theta_{k}}" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.5.m5.1a"><msub id="A5.SS2.SSS0.Px1.p1.5.m5.1.1" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.cmml"><mi id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.2" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.2.cmml">π</mi><msub id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.cmml"><mi id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.2" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.2.cmml">θ</mi><mi id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.3" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.3.cmml">k</mi></msub></msub><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.5.m5.1b"><apply id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1">subscript</csymbol><ci id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.2.cmml" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.2">𝜋</ci><apply id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.cmml" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.1.cmml" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3">subscript</csymbol><ci id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.2.cmml" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.2">𝜃</ci><ci id="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.3.cmml" xref="A5.SS2.SSS0.Px1.p1.5.m5.1.1.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.5.m5.1c">\pi_{\theta_{k}}</annotation></semantics></math>, and thus collecting a total of <math id="A5.SS2.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="nK" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.6.m6.1a"><mrow id="A5.SS2.SSS0.Px1.p1.6.m6.1.1" xref="A5.SS2.SSS0.Px1.p1.6.m6.1.1.cmml"><mi id="A5.SS2.SSS0.Px1.p1.6.m6.1.1.2" xref="A5.SS2.SSS0.Px1.p1.6.m6.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A5.SS2.SSS0.Px1.p1.6.m6.1.1.1" xref="A5.SS2.SSS0.Px1.p1.6.m6.1.1.1.cmml">​</mo><mi id="A5.SS2.SSS0.Px1.p1.6.m6.1.1.3" xref="A5.SS2.SSS0.Px1.p1.6.m6.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.6.m6.1b"><apply id="A5.SS2.SSS0.Px1.p1.6.m6.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.6.m6.1.1"><times id="A5.SS2.SSS0.Px1.p1.6.m6.1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.6.m6.1.1.1"></times><ci id="A5.SS2.SSS0.Px1.p1.6.m6.1.1.2.cmml" xref="A5.SS2.SSS0.Px1.p1.6.m6.1.1.2">𝑛</ci><ci id="A5.SS2.SSS0.Px1.p1.6.m6.1.1.3.cmml" xref="A5.SS2.SSS0.Px1.p1.6.m6.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.6.m6.1c">nK</annotation></semantics></math> data points in each iteration. Let <math id="A5.SS2.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.7.m7.1a"><mi id="A5.SS2.SSS0.Px1.p1.7.m7.1.1" xref="A5.SS2.SSS0.Px1.p1.7.m7.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.7.m7.1b"><ci id="A5.SS2.SSS0.Px1.p1.7.m7.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.7.m7.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.7.m7.1c">x</annotation></semantics></math> be a prompt and <math id="A5.SS2.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.8.m8.1a"><msub id="A5.SS2.SSS0.Px1.p1.8.m8.1.1" xref="A5.SS2.SSS0.Px1.p1.8.m8.1.1.cmml"><mi id="A5.SS2.SSS0.Px1.p1.8.m8.1.1.2" xref="A5.SS2.SSS0.Px1.p1.8.m8.1.1.2.cmml">y</mi><mi id="A5.SS2.SSS0.Px1.p1.8.m8.1.1.3" xref="A5.SS2.SSS0.Px1.p1.8.m8.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.8.m8.1b"><apply id="A5.SS2.SSS0.Px1.p1.8.m8.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.8.m8.1.1"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px1.p1.8.m8.1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.8.m8.1.1">subscript</csymbol><ci id="A5.SS2.SSS0.Px1.p1.8.m8.1.1.2.cmml" xref="A5.SS2.SSS0.Px1.p1.8.m8.1.1.2">𝑦</ci><ci id="A5.SS2.SSS0.Px1.p1.8.m8.1.1.3.cmml" xref="A5.SS2.SSS0.Px1.p1.8.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.8.m8.1c">y_{i}</annotation></semantics></math> be one of the responses. Since we consider the bandit setting, by definition, the <em id="A5.SS2.SSS0.Px1.p1.9.1" class="ltx_emph ltx_font_italic">advantage</em> of <math id="A5.SS2.SSS0.Px1.p1.9.m9.2" class="ltx_Math" alttext="(x,y_{i})" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.9.m9.2a"><mrow id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.2.cmml"><mo stretchy="false" id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.2" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.2.cmml">(</mo><mi id="A5.SS2.SSS0.Px1.p1.9.m9.1.1" xref="A5.SS2.SSS0.Px1.p1.9.m9.1.1.cmml">x</mi><mo id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.3" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.2.cmml">,</mo><msub id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.cmml"><mi id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.2" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.2.cmml">y</mi><mi id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.3" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.4" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.9.m9.2b"><interval closure="open" id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.2.cmml" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1"><ci id="A5.SS2.SSS0.Px1.p1.9.m9.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.9.m9.1.1">𝑥</ci><apply id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1">subscript</csymbol><ci id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.2.cmml" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.2">𝑦</ci><ci id="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.3.cmml" xref="A5.SS2.SSS0.Px1.p1.9.m9.2.2.1.1.3">𝑖</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.9.m9.2c">(x,y_{i})</annotation></semantics></math> is</p>
<table id="A6.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A5.E6"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A5.E6.m1.5" class="ltx_math_unparsed" alttext="\displaystyle A_{k}(x,y_{i})=R(x,y_{i})-\mathbb{E}_{y\sim\pi_{\theta_{k}}(\cdot|x)}[R(x,y)]." display="inline"><semantics id="A5.E6.m1.5a"><mrow id="A5.E6.m1.5.5.1"><mrow id="A5.E6.m1.5.5.1.1"><mrow id="A5.E6.m1.5.5.1.1.1"><msub id="A5.E6.m1.5.5.1.1.1.3"><mi id="A5.E6.m1.5.5.1.1.1.3.2">A</mi><mi id="A5.E6.m1.5.5.1.1.1.3.3">k</mi></msub><mo lspace="0em" rspace="0em" id="A5.E6.m1.5.5.1.1.1.2">​</mo><mrow id="A5.E6.m1.5.5.1.1.1.1.1"><mo stretchy="false" id="A5.E6.m1.5.5.1.1.1.1.1.2">(</mo><mi id="A5.E6.m1.1.1">x</mi><mo id="A5.E6.m1.5.5.1.1.1.1.1.3">,</mo><msub id="A5.E6.m1.5.5.1.1.1.1.1.1"><mi id="A5.E6.m1.5.5.1.1.1.1.1.1.2">y</mi><mi id="A5.E6.m1.5.5.1.1.1.1.1.1.3">i</mi></msub><mo stretchy="false" id="A5.E6.m1.5.5.1.1.1.1.1.4">)</mo></mrow></mrow><mo id="A5.E6.m1.5.5.1.1.4">=</mo><mrow id="A5.E6.m1.5.5.1.1.3"><mrow id="A5.E6.m1.5.5.1.1.2.1"><mi id="A5.E6.m1.5.5.1.1.2.1.3">R</mi><mo lspace="0em" rspace="0em" id="A5.E6.m1.5.5.1.1.2.1.2">​</mo><mrow id="A5.E6.m1.5.5.1.1.2.1.1.1"><mo stretchy="false" id="A5.E6.m1.5.5.1.1.2.1.1.1.2">(</mo><mi id="A5.E6.m1.2.2">x</mi><mo id="A5.E6.m1.5.5.1.1.2.1.1.1.3">,</mo><msub id="A5.E6.m1.5.5.1.1.2.1.1.1.1"><mi id="A5.E6.m1.5.5.1.1.2.1.1.1.1.2">y</mi><mi id="A5.E6.m1.5.5.1.1.2.1.1.1.1.3">i</mi></msub><mo stretchy="false" id="A5.E6.m1.5.5.1.1.2.1.1.1.4">)</mo></mrow></mrow><mo id="A5.E6.m1.5.5.1.1.3.3">−</mo><mrow id="A5.E6.m1.5.5.1.1.3.2"><msub id="A5.E6.m1.5.5.1.1.3.2.3"><mi id="A5.E6.m1.5.5.1.1.3.2.3.2">𝔼</mi><mrow id="A5.E6.m1.5.5.1.1.3.2.3.3"><mi id="A5.E6.m1.5.5.1.1.3.2.3.3.1">y</mi><mo id="A5.E6.m1.5.5.1.1.3.2.3.3.2">∼</mo><msub id="A5.E6.m1.5.5.1.1.3.2.3.3.3"><mi id="A5.E6.m1.5.5.1.1.3.2.3.3.3.2">π</mi><msub id="A5.E6.m1.5.5.1.1.3.2.3.3.3.3"><mi id="A5.E6.m1.5.5.1.1.3.2.3.3.3.3.2">θ</mi><mi id="A5.E6.m1.5.5.1.1.3.2.3.3.3.3.3">k</mi></msub></msub><mrow id="A5.E6.m1.5.5.1.1.3.2.3.3.4"><mo stretchy="false" id="A5.E6.m1.5.5.1.1.3.2.3.3.4.1">(</mo><mo lspace="0em" rspace="0em" id="A5.E6.m1.5.5.1.1.3.2.3.3.4.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E6.m1.5.5.1.1.3.2.3.3.4.3">|</mo><mi id="A5.E6.m1.5.5.1.1.3.2.3.3.4.4">x</mi><mo stretchy="false" id="A5.E6.m1.5.5.1.1.3.2.3.3.4.5">)</mo></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="A5.E6.m1.5.5.1.1.3.2.2">​</mo><mrow id="A5.E6.m1.5.5.1.1.3.2.1.1"><mo stretchy="false" id="A5.E6.m1.5.5.1.1.3.2.1.1.2">[</mo><mrow id="A5.E6.m1.5.5.1.1.3.2.1.1.1"><mi id="A5.E6.m1.5.5.1.1.3.2.1.1.1.2">R</mi><mo lspace="0em" rspace="0em" id="A5.E6.m1.5.5.1.1.3.2.1.1.1.1">​</mo><mrow id="A5.E6.m1.5.5.1.1.3.2.1.1.1.3.2"><mo stretchy="false" id="A5.E6.m1.5.5.1.1.3.2.1.1.1.3.2.1">(</mo><mi id="A5.E6.m1.3.3">x</mi><mo id="A5.E6.m1.5.5.1.1.3.2.1.1.1.3.2.2">,</mo><mi id="A5.E6.m1.4.4">y</mi><mo stretchy="false" id="A5.E6.m1.5.5.1.1.3.2.1.1.1.3.2.3">)</mo></mrow></mrow><mo stretchy="false" id="A5.E6.m1.5.5.1.1.3.2.1.1.3">]</mo></mrow></mrow></mrow></mrow><mo lspace="0em" id="A5.E6.m1.5.5.1.2">.</mo></mrow><annotation encoding="application/x-tex" id="A5.E6.m1.5b">\displaystyle A_{k}(x,y_{i})=R(x,y_{i})-\mathbb{E}_{y\sim\pi_{\theta_{k}}(\cdot|x)}[R(x,y)].</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="A5.SS2.SSS0.Px1.p1.12" class="ltx_p">We use the leave-one-out (LOO) method <cite class="ltx_cite ltx_citemacro_citep">[Kool et al., <a href="#bib.bib28" title="" class="ltx_ref">2019</a>]</cite> to estimate <math id="A5.SS2.SSS0.Px1.p1.10.m1.2" class="ltx_Math" alttext="A_{k}(x,y_{i})" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.10.m1.2a"><mrow id="A5.SS2.SSS0.Px1.p1.10.m1.2.2" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.cmml"><msub id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.cmml"><mi id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.2" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.2.cmml">A</mi><mi id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.3" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.2" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.2.cmml">​</mo><mrow id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.2.cmml"><mo stretchy="false" id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.2" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.2.cmml">(</mo><mi id="A5.SS2.SSS0.Px1.p1.10.m1.1.1" xref="A5.SS2.SSS0.Px1.p1.10.m1.1.1.cmml">x</mi><mo id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.3" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.2.cmml">,</mo><msub id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.cmml"><mi id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.2" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.2.cmml">y</mi><mi id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.3" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.4" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.10.m1.2b"><apply id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2"><times id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.2.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.2"></times><apply id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.1.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3">subscript</csymbol><ci id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.2.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.2">𝐴</ci><ci id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.3.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.3.3">𝑘</ci></apply><interval closure="open" id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.2.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1"><ci id="A5.SS2.SSS0.Px1.p1.10.m1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.1.1">𝑥</ci><apply id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1">subscript</csymbol><ci id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.2.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.2">𝑦</ci><ci id="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.3.cmml" xref="A5.SS2.SSS0.Px1.p1.10.m1.2.2.1.1.1.3">𝑖</ci></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.10.m1.2c">A_{k}(x,y_{i})</annotation></semantics></math>. Namely, we estimate the mean reward given the prompt <math id="A5.SS2.SSS0.Px1.p1.11.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.11.m2.1a"><mi id="A5.SS2.SSS0.Px1.p1.11.m2.1.1" xref="A5.SS2.SSS0.Px1.p1.11.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.11.m2.1b"><ci id="A5.SS2.SSS0.Px1.p1.11.m2.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.11.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.11.m2.1c">x</annotation></semantics></math> with the other <math id="A5.SS2.SSS0.Px1.p1.12.m3.1" class="ltx_Math" alttext="K-1" display="inline"><semantics id="A5.SS2.SSS0.Px1.p1.12.m3.1a"><mrow id="A5.SS2.SSS0.Px1.p1.12.m3.1.1" xref="A5.SS2.SSS0.Px1.p1.12.m3.1.1.cmml"><mi id="A5.SS2.SSS0.Px1.p1.12.m3.1.1.2" xref="A5.SS2.SSS0.Px1.p1.12.m3.1.1.2.cmml">K</mi><mo id="A5.SS2.SSS0.Px1.p1.12.m3.1.1.1" xref="A5.SS2.SSS0.Px1.p1.12.m3.1.1.1.cmml">−</mo><mn id="A5.SS2.SSS0.Px1.p1.12.m3.1.1.3" xref="A5.SS2.SSS0.Px1.p1.12.m3.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px1.p1.12.m3.1b"><apply id="A5.SS2.SSS0.Px1.p1.12.m3.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.12.m3.1.1"><minus id="A5.SS2.SSS0.Px1.p1.12.m3.1.1.1.cmml" xref="A5.SS2.SSS0.Px1.p1.12.m3.1.1.1"></minus><ci id="A5.SS2.SSS0.Px1.p1.12.m3.1.1.2.cmml" xref="A5.SS2.SSS0.Px1.p1.12.m3.1.1.2">𝐾</ci><cn type="integer" id="A5.SS2.SSS0.Px1.p1.12.m3.1.1.3.cmml" xref="A5.SS2.SSS0.Px1.p1.12.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px1.p1.12.m3.1c">K-1</annotation></semantics></math> responses, i.e.,</p>
<table id="A6.EGx4" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A5.E7"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A5.E7.m1.4" class="ltx_Math" alttext="\displaystyle\widehat{A}_{k}(x,y_{i})=R(x,y_{i})-\frac{1}{K-1}\sum_{j\neq i}R(x,y_{j})." display="inline"><semantics id="A5.E7.m1.4a"><mrow id="A5.E7.m1.4.4.1" xref="A5.E7.m1.4.4.1.1.cmml"><mrow id="A5.E7.m1.4.4.1.1" xref="A5.E7.m1.4.4.1.1.cmml"><mrow id="A5.E7.m1.4.4.1.1.1" xref="A5.E7.m1.4.4.1.1.1.cmml"><msub id="A5.E7.m1.4.4.1.1.1.3" xref="A5.E7.m1.4.4.1.1.1.3.cmml"><mover accent="true" id="A5.E7.m1.4.4.1.1.1.3.2" xref="A5.E7.m1.4.4.1.1.1.3.2.cmml"><mi id="A5.E7.m1.4.4.1.1.1.3.2.2" xref="A5.E7.m1.4.4.1.1.1.3.2.2.cmml">A</mi><mo id="A5.E7.m1.4.4.1.1.1.3.2.1" xref="A5.E7.m1.4.4.1.1.1.3.2.1.cmml">^</mo></mover><mi id="A5.E7.m1.4.4.1.1.1.3.3" xref="A5.E7.m1.4.4.1.1.1.3.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="A5.E7.m1.4.4.1.1.1.2" xref="A5.E7.m1.4.4.1.1.1.2.cmml">​</mo><mrow id="A5.E7.m1.4.4.1.1.1.1.1" xref="A5.E7.m1.4.4.1.1.1.1.2.cmml"><mo stretchy="false" id="A5.E7.m1.4.4.1.1.1.1.1.2" xref="A5.E7.m1.4.4.1.1.1.1.2.cmml">(</mo><mi id="A5.E7.m1.1.1" xref="A5.E7.m1.1.1.cmml">x</mi><mo id="A5.E7.m1.4.4.1.1.1.1.1.3" xref="A5.E7.m1.4.4.1.1.1.1.2.cmml">,</mo><msub id="A5.E7.m1.4.4.1.1.1.1.1.1" xref="A5.E7.m1.4.4.1.1.1.1.1.1.cmml"><mi id="A5.E7.m1.4.4.1.1.1.1.1.1.2" xref="A5.E7.m1.4.4.1.1.1.1.1.1.2.cmml">y</mi><mi id="A5.E7.m1.4.4.1.1.1.1.1.1.3" xref="A5.E7.m1.4.4.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A5.E7.m1.4.4.1.1.1.1.1.4" xref="A5.E7.m1.4.4.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="A5.E7.m1.4.4.1.1.4" xref="A5.E7.m1.4.4.1.1.4.cmml">=</mo><mrow id="A5.E7.m1.4.4.1.1.3" xref="A5.E7.m1.4.4.1.1.3.cmml"><mrow id="A5.E7.m1.4.4.1.1.2.1" xref="A5.E7.m1.4.4.1.1.2.1.cmml"><mi id="A5.E7.m1.4.4.1.1.2.1.3" xref="A5.E7.m1.4.4.1.1.2.1.3.cmml">R</mi><mo lspace="0em" rspace="0em" id="A5.E7.m1.4.4.1.1.2.1.2" xref="A5.E7.m1.4.4.1.1.2.1.2.cmml">​</mo><mrow id="A5.E7.m1.4.4.1.1.2.1.1.1" xref="A5.E7.m1.4.4.1.1.2.1.1.2.cmml"><mo stretchy="false" id="A5.E7.m1.4.4.1.1.2.1.1.1.2" xref="A5.E7.m1.4.4.1.1.2.1.1.2.cmml">(</mo><mi id="A5.E7.m1.2.2" xref="A5.E7.m1.2.2.cmml">x</mi><mo id="A5.E7.m1.4.4.1.1.2.1.1.1.3" xref="A5.E7.m1.4.4.1.1.2.1.1.2.cmml">,</mo><msub id="A5.E7.m1.4.4.1.1.2.1.1.1.1" xref="A5.E7.m1.4.4.1.1.2.1.1.1.1.cmml"><mi id="A5.E7.m1.4.4.1.1.2.1.1.1.1.2" xref="A5.E7.m1.4.4.1.1.2.1.1.1.1.2.cmml">y</mi><mi id="A5.E7.m1.4.4.1.1.2.1.1.1.1.3" xref="A5.E7.m1.4.4.1.1.2.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A5.E7.m1.4.4.1.1.2.1.1.1.4" xref="A5.E7.m1.4.4.1.1.2.1.1.2.cmml">)</mo></mrow></mrow><mo id="A5.E7.m1.4.4.1.1.3.3" xref="A5.E7.m1.4.4.1.1.3.3.cmml">−</mo><mrow id="A5.E7.m1.4.4.1.1.3.2" xref="A5.E7.m1.4.4.1.1.3.2.cmml"><mstyle displaystyle="true" id="A5.E7.m1.4.4.1.1.3.2.3" xref="A5.E7.m1.4.4.1.1.3.2.3.cmml"><mfrac id="A5.E7.m1.4.4.1.1.3.2.3a" xref="A5.E7.m1.4.4.1.1.3.2.3.cmml"><mn id="A5.E7.m1.4.4.1.1.3.2.3.2" xref="A5.E7.m1.4.4.1.1.3.2.3.2.cmml">1</mn><mrow id="A5.E7.m1.4.4.1.1.3.2.3.3" xref="A5.E7.m1.4.4.1.1.3.2.3.3.cmml"><mi id="A5.E7.m1.4.4.1.1.3.2.3.3.2" xref="A5.E7.m1.4.4.1.1.3.2.3.3.2.cmml">K</mi><mo id="A5.E7.m1.4.4.1.1.3.2.3.3.1" xref="A5.E7.m1.4.4.1.1.3.2.3.3.1.cmml">−</mo><mn id="A5.E7.m1.4.4.1.1.3.2.3.3.3" xref="A5.E7.m1.4.4.1.1.3.2.3.3.3.cmml">1</mn></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A5.E7.m1.4.4.1.1.3.2.2" xref="A5.E7.m1.4.4.1.1.3.2.2.cmml">​</mo><mrow id="A5.E7.m1.4.4.1.1.3.2.1" xref="A5.E7.m1.4.4.1.1.3.2.1.cmml"><mstyle displaystyle="true" id="A5.E7.m1.4.4.1.1.3.2.1.2" xref="A5.E7.m1.4.4.1.1.3.2.1.2.cmml"><munder id="A5.E7.m1.4.4.1.1.3.2.1.2a" xref="A5.E7.m1.4.4.1.1.3.2.1.2.cmml"><mo movablelimits="false" id="A5.E7.m1.4.4.1.1.3.2.1.2.2" xref="A5.E7.m1.4.4.1.1.3.2.1.2.2.cmml">∑</mo><mrow id="A5.E7.m1.4.4.1.1.3.2.1.2.3" xref="A5.E7.m1.4.4.1.1.3.2.1.2.3.cmml"><mi id="A5.E7.m1.4.4.1.1.3.2.1.2.3.2" xref="A5.E7.m1.4.4.1.1.3.2.1.2.3.2.cmml">j</mi><mo id="A5.E7.m1.4.4.1.1.3.2.1.2.3.1" xref="A5.E7.m1.4.4.1.1.3.2.1.2.3.1.cmml">≠</mo><mi id="A5.E7.m1.4.4.1.1.3.2.1.2.3.3" xref="A5.E7.m1.4.4.1.1.3.2.1.2.3.3.cmml">i</mi></mrow></munder></mstyle><mrow id="A5.E7.m1.4.4.1.1.3.2.1.1" xref="A5.E7.m1.4.4.1.1.3.2.1.1.cmml"><mi id="A5.E7.m1.4.4.1.1.3.2.1.1.3" xref="A5.E7.m1.4.4.1.1.3.2.1.1.3.cmml">R</mi><mo lspace="0em" rspace="0em" id="A5.E7.m1.4.4.1.1.3.2.1.1.2" xref="A5.E7.m1.4.4.1.1.3.2.1.1.2.cmml">​</mo><mrow id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.2.cmml"><mo stretchy="false" id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.2" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.2.cmml">(</mo><mi id="A5.E7.m1.3.3" xref="A5.E7.m1.3.3.cmml">x</mi><mo id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.3" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.2.cmml">,</mo><msub id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.cmml"><mi id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.2" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.2.cmml">y</mi><mi id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.3" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.3.cmml">j</mi></msub><mo stretchy="false" id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.4" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow></mrow><mo lspace="0em" id="A5.E7.m1.4.4.1.2" xref="A5.E7.m1.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A5.E7.m1.4b"><apply id="A5.E7.m1.4.4.1.1.cmml" xref="A5.E7.m1.4.4.1"><eq id="A5.E7.m1.4.4.1.1.4.cmml" xref="A5.E7.m1.4.4.1.1.4"></eq><apply id="A5.E7.m1.4.4.1.1.1.cmml" xref="A5.E7.m1.4.4.1.1.1"><times id="A5.E7.m1.4.4.1.1.1.2.cmml" xref="A5.E7.m1.4.4.1.1.1.2"></times><apply id="A5.E7.m1.4.4.1.1.1.3.cmml" xref="A5.E7.m1.4.4.1.1.1.3"><csymbol cd="ambiguous" id="A5.E7.m1.4.4.1.1.1.3.1.cmml" xref="A5.E7.m1.4.4.1.1.1.3">subscript</csymbol><apply id="A5.E7.m1.4.4.1.1.1.3.2.cmml" xref="A5.E7.m1.4.4.1.1.1.3.2"><ci id="A5.E7.m1.4.4.1.1.1.3.2.1.cmml" xref="A5.E7.m1.4.4.1.1.1.3.2.1">^</ci><ci id="A5.E7.m1.4.4.1.1.1.3.2.2.cmml" xref="A5.E7.m1.4.4.1.1.1.3.2.2">𝐴</ci></apply><ci id="A5.E7.m1.4.4.1.1.1.3.3.cmml" xref="A5.E7.m1.4.4.1.1.1.3.3">𝑘</ci></apply><interval closure="open" id="A5.E7.m1.4.4.1.1.1.1.2.cmml" xref="A5.E7.m1.4.4.1.1.1.1.1"><ci id="A5.E7.m1.1.1.cmml" xref="A5.E7.m1.1.1">𝑥</ci><apply id="A5.E7.m1.4.4.1.1.1.1.1.1.cmml" xref="A5.E7.m1.4.4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A5.E7.m1.4.4.1.1.1.1.1.1.1.cmml" xref="A5.E7.m1.4.4.1.1.1.1.1.1">subscript</csymbol><ci id="A5.E7.m1.4.4.1.1.1.1.1.1.2.cmml" xref="A5.E7.m1.4.4.1.1.1.1.1.1.2">𝑦</ci><ci id="A5.E7.m1.4.4.1.1.1.1.1.1.3.cmml" xref="A5.E7.m1.4.4.1.1.1.1.1.1.3">𝑖</ci></apply></interval></apply><apply id="A5.E7.m1.4.4.1.1.3.cmml" xref="A5.E7.m1.4.4.1.1.3"><minus id="A5.E7.m1.4.4.1.1.3.3.cmml" xref="A5.E7.m1.4.4.1.1.3.3"></minus><apply id="A5.E7.m1.4.4.1.1.2.1.cmml" xref="A5.E7.m1.4.4.1.1.2.1"><times id="A5.E7.m1.4.4.1.1.2.1.2.cmml" xref="A5.E7.m1.4.4.1.1.2.1.2"></times><ci id="A5.E7.m1.4.4.1.1.2.1.3.cmml" xref="A5.E7.m1.4.4.1.1.2.1.3">𝑅</ci><interval closure="open" id="A5.E7.m1.4.4.1.1.2.1.1.2.cmml" xref="A5.E7.m1.4.4.1.1.2.1.1.1"><ci id="A5.E7.m1.2.2.cmml" xref="A5.E7.m1.2.2">𝑥</ci><apply id="A5.E7.m1.4.4.1.1.2.1.1.1.1.cmml" xref="A5.E7.m1.4.4.1.1.2.1.1.1.1"><csymbol cd="ambiguous" id="A5.E7.m1.4.4.1.1.2.1.1.1.1.1.cmml" xref="A5.E7.m1.4.4.1.1.2.1.1.1.1">subscript</csymbol><ci id="A5.E7.m1.4.4.1.1.2.1.1.1.1.2.cmml" xref="A5.E7.m1.4.4.1.1.2.1.1.1.1.2">𝑦</ci><ci id="A5.E7.m1.4.4.1.1.2.1.1.1.1.3.cmml" xref="A5.E7.m1.4.4.1.1.2.1.1.1.1.3">𝑖</ci></apply></interval></apply><apply id="A5.E7.m1.4.4.1.1.3.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2"><times id="A5.E7.m1.4.4.1.1.3.2.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2.2"></times><apply id="A5.E7.m1.4.4.1.1.3.2.3.cmml" xref="A5.E7.m1.4.4.1.1.3.2.3"><divide id="A5.E7.m1.4.4.1.1.3.2.3.1.cmml" xref="A5.E7.m1.4.4.1.1.3.2.3"></divide><cn type="integer" id="A5.E7.m1.4.4.1.1.3.2.3.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2.3.2">1</cn><apply id="A5.E7.m1.4.4.1.1.3.2.3.3.cmml" xref="A5.E7.m1.4.4.1.1.3.2.3.3"><minus id="A5.E7.m1.4.4.1.1.3.2.3.3.1.cmml" xref="A5.E7.m1.4.4.1.1.3.2.3.3.1"></minus><ci id="A5.E7.m1.4.4.1.1.3.2.3.3.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2.3.3.2">𝐾</ci><cn type="integer" id="A5.E7.m1.4.4.1.1.3.2.3.3.3.cmml" xref="A5.E7.m1.4.4.1.1.3.2.3.3.3">1</cn></apply></apply><apply id="A5.E7.m1.4.4.1.1.3.2.1.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1"><apply id="A5.E7.m1.4.4.1.1.3.2.1.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.2"><csymbol cd="ambiguous" id="A5.E7.m1.4.4.1.1.3.2.1.2.1.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.2">subscript</csymbol><sum id="A5.E7.m1.4.4.1.1.3.2.1.2.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.2.2"></sum><apply id="A5.E7.m1.4.4.1.1.3.2.1.2.3.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.2.3"><neq id="A5.E7.m1.4.4.1.1.3.2.1.2.3.1.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.2.3.1"></neq><ci id="A5.E7.m1.4.4.1.1.3.2.1.2.3.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.2.3.2">𝑗</ci><ci id="A5.E7.m1.4.4.1.1.3.2.1.2.3.3.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.2.3.3">𝑖</ci></apply></apply><apply id="A5.E7.m1.4.4.1.1.3.2.1.1.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.1"><times id="A5.E7.m1.4.4.1.1.3.2.1.1.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.1.2"></times><ci id="A5.E7.m1.4.4.1.1.3.2.1.1.3.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.1.3">𝑅</ci><interval closure="open" id="A5.E7.m1.4.4.1.1.3.2.1.1.1.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.1"><ci id="A5.E7.m1.3.3.cmml" xref="A5.E7.m1.3.3">𝑥</ci><apply id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.1.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1">subscript</csymbol><ci id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.2.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.2">𝑦</ci><ci id="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.3.cmml" xref="A5.E7.m1.4.4.1.1.3.2.1.1.1.1.1.3">𝑗</ci></apply></interval></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.E7.m1.4c">\displaystyle\widehat{A}_{k}(x,y_{i})=R(x,y_{i})-\frac{1}{K-1}\sum_{j\neq i}R(x,y_{j}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
<p id="A5.SS2.SSS0.Px1.p1.13" class="ltx_p">As shown in recent works <cite class="ltx_cite ltx_citemacro_citep">[Ahmadian et al., <a href="#bib.bib3" title="" class="ltx_ref">2024</a>]</cite>, this advantage estimation is beneficial for RLHF. Empirically, we find that using LOO estimator leads to more stable training and better results compared to directly using the reward as the advantage estimation or using the difference between the reward and a running average baseline <cite class="ltx_cite ltx_citemacro_citep">[Williams, <a href="#bib.bib55" title="" class="ltx_ref">1992</a>]</cite>.</p>
</div>
</section>
<section id="A5.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_font_sansserif ltx_title_paragraph">Mirror descent policy optimization (MDPO).</h5>

<div id="A5.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="A5.SS2.SSS0.Px2.p1.1" class="ltx_p">Our policy optimization approach belongs to a widely used class of trust-region policy optimization algorithms <cite class="ltx_cite ltx_citemacro_citep">[Schulman et al., <a href="#bib.bib44" title="" class="ltx_ref">2015</a>]</cite>. The basic idea in these algorithms is that in each policy iteration, we apply a regularization method to prevent the policy from changing too much in an iteration. The regularization can be achieved by adding KL regularization <cite class="ltx_cite ltx_citemacro_citep">[Tomar et al., <a href="#bib.bib50" title="" class="ltx_ref">2020</a>; Abbasi-Yadkori et al., <a href="#bib.bib1" title="" class="ltx_ref">2019</a>; Lazic et al., <a href="#bib.bib29" title="" class="ltx_ref">2021</a>]</cite> and using clipping for the probability ratio such as in PPO <cite class="ltx_cite ltx_citemacro_citep">[Schulman et al., <a href="#bib.bib45" title="" class="ltx_ref">2017</a>]</cite>. In this work, we use KL regularization as in Mirror Descent Policy Optimization (MDPO) <cite class="ltx_cite ltx_citemacro_citep">[Tomar et al., <a href="#bib.bib50" title="" class="ltx_ref">2020</a>]</cite>.</p>
</div>
<div id="A5.SS2.SSS0.Px2.p2" class="ltx_para">
<p id="A5.SS2.SSS0.Px2.p2.3" class="ltx_p">In particular, in the <math id="A5.SS2.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS2.SSS0.Px2.p2.1.m1.1a"><mi id="A5.SS2.SSS0.Px2.p2.1.m1.1.1" xref="A5.SS2.SSS0.Px2.p2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px2.p2.1.m1.1b"><ci id="A5.SS2.SSS0.Px2.p2.1.m1.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px2.p2.1.m1.1c">k</annotation></semantics></math>-th iteration, with the data (prompts along with the <math id="A5.SS2.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="A5.SS2.SSS0.Px2.p2.2.m2.1a"><mi id="A5.SS2.SSS0.Px2.p2.2.m2.1.1" xref="A5.SS2.SSS0.Px2.p2.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px2.p2.2.m2.1b"><ci id="A5.SS2.SSS0.Px2.p2.2.m2.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px2.p2.2.m2.1c">K</annotation></semantics></math> responses sampled according to <math id="A5.SS2.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="\pi_{\theta_{k}}" display="inline"><semantics id="A5.SS2.SSS0.Px2.p2.3.m3.1a"><msub id="A5.SS2.SSS0.Px2.p2.3.m3.1.1" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.cmml"><mi id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.2" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.2.cmml">π</mi><msub id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.cmml"><mi id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.2" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.cmml">θ</mi><mi id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.3" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.cmml">k</mi></msub></msub><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px2.p2.3.m3.1b"><apply id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1">subscript</csymbol><ci id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.2">𝜋</ci><apply id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.cmml" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.1.cmml" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3">subscript</csymbol><ci id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.cmml" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.2">𝜃</ci><ci id="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.cmml" xref="A5.SS2.SSS0.Px2.p2.3.m3.1.1.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px2.p2.3.m3.1c">\pi_{\theta_{k}}</annotation></semantics></math> for each prompt), we aim to optimize the following regularized advantage maximization problem:</p>
<table id="A6.EGx5" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A5.E8"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A5.E8.m1.3" class="ltx_math_unparsed" alttext="\displaystyle\max_{\theta}\Psi(\theta):=\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta_{k}}(\cdot|x)}[A_{k}(x,y)]-\gamma D_{\text{KL}}(\pi_{\theta}(\cdot|x)||\pi_{\theta_{k}}(\cdot|x))\right]." display="inline"><semantics id="A5.E8.m1.3a"><mrow id="A5.E8.m1.3b"><munder id="A5.E8.m1.3.4"><mi id="A5.E8.m1.3.4.2">max</mi><mi id="A5.E8.m1.3.4.3">θ</mi></munder><mi mathvariant="normal" id="A5.E8.m1.3.5">Ψ</mi><mrow id="A5.E8.m1.3.6"><mo stretchy="false" id="A5.E8.m1.3.6.1">(</mo><mi id="A5.E8.m1.1.1">θ</mi><mo rspace="0.278em" stretchy="false" id="A5.E8.m1.3.6.2">)</mo></mrow><mo rspace="0.278em" id="A5.E8.m1.3.7">:=</mo><msub id="A5.E8.m1.3.8"><mi id="A5.E8.m1.3.8.2">𝔼</mi><mrow id="A5.E8.m1.3.8.3"><mi id="A5.E8.m1.3.8.3.2">x</mi><mo id="A5.E8.m1.3.8.3.1">∼</mo><mi class="ltx_font_mathcaligraphic" id="A5.E8.m1.3.8.3.3">𝒟</mi></mrow></msub><mrow id="A5.E8.m1.3.9"><mo id="A5.E8.m1.3.9.1">[</mo><msub id="A5.E8.m1.3.9.2"><mi id="A5.E8.m1.3.9.2.2">𝔼</mi><mrow id="A5.E8.m1.3.9.2.3"><mi id="A5.E8.m1.3.9.2.3.1">y</mi><mo id="A5.E8.m1.3.9.2.3.2">∼</mo><msub id="A5.E8.m1.3.9.2.3.3"><mi id="A5.E8.m1.3.9.2.3.3.2">π</mi><msub id="A5.E8.m1.3.9.2.3.3.3"><mi id="A5.E8.m1.3.9.2.3.3.3.2">θ</mi><mi id="A5.E8.m1.3.9.2.3.3.3.3">k</mi></msub></msub><mrow id="A5.E8.m1.3.9.2.3.4"><mo stretchy="false" id="A5.E8.m1.3.9.2.3.4.1">(</mo><mo lspace="0em" rspace="0em" id="A5.E8.m1.3.9.2.3.4.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E8.m1.3.9.2.3.4.3">|</mo><mi id="A5.E8.m1.3.9.2.3.4.4">x</mi><mo stretchy="false" id="A5.E8.m1.3.9.2.3.4.5">)</mo></mrow></mrow></msub><mrow id="A5.E8.m1.3.9.3"><mo stretchy="false" id="A5.E8.m1.3.9.3.1">[</mo><msub id="A5.E8.m1.3.9.3.2"><mi id="A5.E8.m1.3.9.3.2.2">A</mi><mi id="A5.E8.m1.3.9.3.2.3">k</mi></msub><mrow id="A5.E8.m1.3.9.3.3"><mo stretchy="false" id="A5.E8.m1.3.9.3.3.1">(</mo><mi id="A5.E8.m1.2.2">x</mi><mo id="A5.E8.m1.3.9.3.3.2">,</mo><mi id="A5.E8.m1.3.3">y</mi><mo stretchy="false" id="A5.E8.m1.3.9.3.3.3">)</mo></mrow><mo stretchy="false" id="A5.E8.m1.3.9.3.4">]</mo></mrow><mo id="A5.E8.m1.3.9.4">−</mo><mi id="A5.E8.m1.3.9.5">γ</mi><msub id="A5.E8.m1.3.9.6"><mi id="A5.E8.m1.3.9.6.2">D</mi><mtext id="A5.E8.m1.3.9.6.3">KL</mtext></msub><mrow id="A5.E8.m1.3.9.7"><mo stretchy="false" id="A5.E8.m1.3.9.7.1">(</mo><msub id="A5.E8.m1.3.9.7.2"><mi id="A5.E8.m1.3.9.7.2.2">π</mi><mi id="A5.E8.m1.3.9.7.2.3">θ</mi></msub><mrow id="A5.E8.m1.3.9.7.3"><mo stretchy="false" id="A5.E8.m1.3.9.7.3.1">(</mo><mo lspace="0em" rspace="0em" id="A5.E8.m1.3.9.7.3.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E8.m1.3.9.7.3.3">|</mo><mi id="A5.E8.m1.3.9.7.3.4">x</mi><mo stretchy="false" id="A5.E8.m1.3.9.7.3.5">)</mo></mrow><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E8.m1.3.9.7.4">|</mo><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E8.m1.3.9.7.5">|</mo><msub id="A5.E8.m1.3.9.7.6"><mi id="A5.E8.m1.3.9.7.6.2">π</mi><msub id="A5.E8.m1.3.9.7.6.3"><mi id="A5.E8.m1.3.9.7.6.3.2">θ</mi><mi id="A5.E8.m1.3.9.7.6.3.3">k</mi></msub></msub><mrow id="A5.E8.m1.3.9.7.7"><mo stretchy="false" id="A5.E8.m1.3.9.7.7.1">(</mo><mo lspace="0em" rspace="0em" id="A5.E8.m1.3.9.7.7.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E8.m1.3.9.7.7.3">|</mo><mi id="A5.E8.m1.3.9.7.7.4">x</mi><mo stretchy="false" id="A5.E8.m1.3.9.7.7.5">)</mo></mrow><mo stretchy="false" id="A5.E8.m1.3.9.7.8">)</mo></mrow><mo id="A5.E8.m1.3.9.8">]</mo></mrow><mo lspace="0em" id="A5.E8.m1.3.10">.</mo></mrow><annotation encoding="application/x-tex" id="A5.E8.m1.3c">\displaystyle\max_{\theta}\Psi(\theta):=\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi_{\theta_{k}}(\cdot|x)}[A_{k}(x,y)]-\gamma D_{\text{KL}}(\pi_{\theta}(\cdot|x)||\pi_{\theta_{k}}(\cdot|x))\right].</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="A5.SS2.SSS0.Px2.p2.5" class="ltx_p">Note that here the KL regularization term is different from the one in Eq. (<a href="#S4.E1" title="In 4.3.3 Online RLHF algorithm: MDLOO ‣ 4.3 Reinforcement learning from human feedback (RLHF) ‣ 4 Post-Training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>). The KL regularization in Eq. (<a href="#S4.E1" title="In 4.3.3 Online RLHF algorithm: MDLOO ‣ 4.3 Reinforcement learning from human feedback (RLHF) ‣ 4 Post-Training ‣ Apple Intelligence Foundation Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) is between the policy model and the reference model; whereas the KL regularization term in Eq. (<a href="#A5.E8" title="In Mirror descent policy optimization (MDPO). ‣ E.2 Online RL algorithm ‣ Appendix E Technical details for RLHF ‣ Apple Intelligence Foundation Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>) is between the policy model and the policy at the beginning of the <math id="A5.SS2.SSS0.Px2.p2.4.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS2.SSS0.Px2.p2.4.m1.1a"><mi id="A5.SS2.SSS0.Px2.p2.4.m1.1.1" xref="A5.SS2.SSS0.Px2.p2.4.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px2.p2.4.m1.1b"><ci id="A5.SS2.SSS0.Px2.p2.4.m1.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.4.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px2.p2.4.m1.1c">k</annotation></semantics></math>-th iteration. Then we can obtain the gradient of <math id="A5.SS2.SSS0.Px2.p2.5.m2.1" class="ltx_Math" alttext="\Psi(\theta)" display="inline"><semantics id="A5.SS2.SSS0.Px2.p2.5.m2.1a"><mrow id="A5.SS2.SSS0.Px2.p2.5.m2.1.2" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.2.cmml"><mi mathvariant="normal" id="A5.SS2.SSS0.Px2.p2.5.m2.1.2.2" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.2.2.cmml">Ψ</mi><mo lspace="0em" rspace="0em" id="A5.SS2.SSS0.Px2.p2.5.m2.1.2.1" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.2.1.cmml">​</mo><mrow id="A5.SS2.SSS0.Px2.p2.5.m2.1.2.3.2" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.2.cmml"><mo stretchy="false" id="A5.SS2.SSS0.Px2.p2.5.m2.1.2.3.2.1" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.2.cmml">(</mo><mi id="A5.SS2.SSS0.Px2.p2.5.m2.1.1" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.1.cmml">θ</mi><mo stretchy="false" id="A5.SS2.SSS0.Px2.p2.5.m2.1.2.3.2.2" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px2.p2.5.m2.1b"><apply id="A5.SS2.SSS0.Px2.p2.5.m2.1.2.cmml" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.2"><times id="A5.SS2.SSS0.Px2.p2.5.m2.1.2.1.cmml" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.2.1"></times><ci id="A5.SS2.SSS0.Px2.p2.5.m2.1.2.2.cmml" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.2.2">Ψ</ci><ci id="A5.SS2.SSS0.Px2.p2.5.m2.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.5.m2.1.1">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px2.p2.5.m2.1c">\Psi(\theta)</annotation></semantics></math> as</p>
<table id="A5.E9" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="A5.E9X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A5.E9X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\nabla\Psi(\theta)=" display="inline"><semantics id="A5.E9X.2.1.1.m1.1a"><mrow id="A5.E9X.2.1.1.m1.1.2" xref="A5.E9X.2.1.1.m1.1.2.cmml"><mrow id="A5.E9X.2.1.1.m1.1.2.2" xref="A5.E9X.2.1.1.m1.1.2.2.cmml"><mrow id="A5.E9X.2.1.1.m1.1.2.2.2" xref="A5.E9X.2.1.1.m1.1.2.2.2.cmml"><mo rspace="0.167em" id="A5.E9X.2.1.1.m1.1.2.2.2.1" xref="A5.E9X.2.1.1.m1.1.2.2.2.1.cmml">∇</mo><mi mathvariant="normal" id="A5.E9X.2.1.1.m1.1.2.2.2.2" xref="A5.E9X.2.1.1.m1.1.2.2.2.2.cmml">Ψ</mi></mrow><mo lspace="0em" rspace="0em" id="A5.E9X.2.1.1.m1.1.2.2.1" xref="A5.E9X.2.1.1.m1.1.2.2.1.cmml">​</mo><mrow id="A5.E9X.2.1.1.m1.1.2.2.3.2" xref="A5.E9X.2.1.1.m1.1.2.2.cmml"><mo stretchy="false" id="A5.E9X.2.1.1.m1.1.2.2.3.2.1" xref="A5.E9X.2.1.1.m1.1.2.2.cmml">(</mo><mi id="A5.E9X.2.1.1.m1.1.1" xref="A5.E9X.2.1.1.m1.1.1.cmml">θ</mi><mo stretchy="false" id="A5.E9X.2.1.1.m1.1.2.2.3.2.2" xref="A5.E9X.2.1.1.m1.1.2.2.cmml">)</mo></mrow></mrow><mo id="A5.E9X.2.1.1.m1.1.2.1" xref="A5.E9X.2.1.1.m1.1.2.1.cmml">=</mo><mi id="A5.E9X.2.1.1.m1.1.2.3" xref="A5.E9X.2.1.1.m1.1.2.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A5.E9X.2.1.1.m1.1b"><apply id="A5.E9X.2.1.1.m1.1.2.cmml" xref="A5.E9X.2.1.1.m1.1.2"><eq id="A5.E9X.2.1.1.m1.1.2.1.cmml" xref="A5.E9X.2.1.1.m1.1.2.1"></eq><apply id="A5.E9X.2.1.1.m1.1.2.2.cmml" xref="A5.E9X.2.1.1.m1.1.2.2"><times id="A5.E9X.2.1.1.m1.1.2.2.1.cmml" xref="A5.E9X.2.1.1.m1.1.2.2.1"></times><apply id="A5.E9X.2.1.1.m1.1.2.2.2.cmml" xref="A5.E9X.2.1.1.m1.1.2.2.2"><ci id="A5.E9X.2.1.1.m1.1.2.2.2.1.cmml" xref="A5.E9X.2.1.1.m1.1.2.2.2.1">∇</ci><ci id="A5.E9X.2.1.1.m1.1.2.2.2.2.cmml" xref="A5.E9X.2.1.1.m1.1.2.2.2.2">Ψ</ci></apply><ci id="A5.E9X.2.1.1.m1.1.1.cmml" xref="A5.E9X.2.1.1.m1.1.1">𝜃</ci></apply><csymbol cd="latexml" id="A5.E9X.2.1.1.m1.1.2.3.cmml" xref="A5.E9X.2.1.1.m1.1.2.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.E9X.2.1.1.m1.1c">\displaystyle\nabla\Psi(\theta)=</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A5.E9X.3.2.2.m1.5" class="ltx_math_unparsed" alttext="\displaystyle\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta_{k}}(\cdot|x)}\left[\frac{\pi_{\theta}(y|x)}{\pi_{\theta_{k}}(y|x)}A_{k}(x,y)\nabla\log\pi_{\theta}(y|x)\right]" display="inline"><semantics id="A5.E9X.3.2.2.m1.5a"><mrow id="A5.E9X.3.2.2.m1.5.5"><msub id="A5.E9X.3.2.2.m1.5.5.3"><mi id="A5.E9X.3.2.2.m1.5.5.3.2">𝔼</mi><mrow id="A5.E9X.3.2.2.m1.5.5.3.3"><mi id="A5.E9X.3.2.2.m1.5.5.3.3.1">x</mi><mo id="A5.E9X.3.2.2.m1.5.5.3.3.2">∼</mo><mi class="ltx_font_mathcaligraphic" id="A5.E9X.3.2.2.m1.5.5.3.3.3">𝒟</mi><mo id="A5.E9X.3.2.2.m1.5.5.3.3.4">,</mo><mi id="A5.E9X.3.2.2.m1.5.5.3.3.5">y</mi><mo id="A5.E9X.3.2.2.m1.5.5.3.3.6">∼</mo><msub id="A5.E9X.3.2.2.m1.5.5.3.3.7"><mi id="A5.E9X.3.2.2.m1.5.5.3.3.7.2">π</mi><msub id="A5.E9X.3.2.2.m1.5.5.3.3.7.3"><mi id="A5.E9X.3.2.2.m1.5.5.3.3.7.3.2">θ</mi><mi id="A5.E9X.3.2.2.m1.5.5.3.3.7.3.3">k</mi></msub></msub><mrow id="A5.E9X.3.2.2.m1.5.5.3.3.8"><mo stretchy="false" id="A5.E9X.3.2.2.m1.5.5.3.3.8.1">(</mo><mo lspace="0em" rspace="0em" id="A5.E9X.3.2.2.m1.5.5.3.3.8.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E9X.3.2.2.m1.5.5.3.3.8.3">|</mo><mi id="A5.E9X.3.2.2.m1.5.5.3.3.8.4">x</mi><mo stretchy="false" id="A5.E9X.3.2.2.m1.5.5.3.3.8.5">)</mo></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="A5.E9X.3.2.2.m1.5.5.2">​</mo><mrow id="A5.E9X.3.2.2.m1.5.5.1.1"><mo id="A5.E9X.3.2.2.m1.5.5.1.1.2">[</mo><mrow id="A5.E9X.3.2.2.m1.5.5.1.1.1"><mstyle displaystyle="true" id="A5.E9X.3.2.2.m1.2.2"><mfrac id="A5.E9X.3.2.2.m1.2.2a"><mrow id="A5.E9X.3.2.2.m1.1.1.1"><msub id="A5.E9X.3.2.2.m1.1.1.1.3"><mi id="A5.E9X.3.2.2.m1.1.1.1.3.2">π</mi><mi id="A5.E9X.3.2.2.m1.1.1.1.3.3">θ</mi></msub><mo lspace="0em" rspace="0em" id="A5.E9X.3.2.2.m1.1.1.1.2">​</mo><mrow id="A5.E9X.3.2.2.m1.1.1.1.1.1"><mo stretchy="false" id="A5.E9X.3.2.2.m1.1.1.1.1.1.2">(</mo><mrow id="A5.E9X.3.2.2.m1.1.1.1.1.1.1"><mi id="A5.E9X.3.2.2.m1.1.1.1.1.1.1.2">y</mi><mo fence="false" id="A5.E9X.3.2.2.m1.1.1.1.1.1.1.1">|</mo><mi id="A5.E9X.3.2.2.m1.1.1.1.1.1.1.3">x</mi></mrow><mo stretchy="false" id="A5.E9X.3.2.2.m1.1.1.1.1.1.3">)</mo></mrow></mrow><mrow id="A5.E9X.3.2.2.m1.2.2.2"><msub id="A5.E9X.3.2.2.m1.2.2.2.3"><mi id="A5.E9X.3.2.2.m1.2.2.2.3.2">π</mi><msub id="A5.E9X.3.2.2.m1.2.2.2.3.3"><mi id="A5.E9X.3.2.2.m1.2.2.2.3.3.2">θ</mi><mi id="A5.E9X.3.2.2.m1.2.2.2.3.3.3">k</mi></msub></msub><mo lspace="0em" rspace="0em" id="A5.E9X.3.2.2.m1.2.2.2.2">​</mo><mrow id="A5.E9X.3.2.2.m1.2.2.2.1.1"><mo stretchy="false" id="A5.E9X.3.2.2.m1.2.2.2.1.1.2">(</mo><mrow id="A5.E9X.3.2.2.m1.2.2.2.1.1.1"><mi id="A5.E9X.3.2.2.m1.2.2.2.1.1.1.2">y</mi><mo fence="false" id="A5.E9X.3.2.2.m1.2.2.2.1.1.1.1">|</mo><mi id="A5.E9X.3.2.2.m1.2.2.2.1.1.1.3">x</mi></mrow><mo stretchy="false" id="A5.E9X.3.2.2.m1.2.2.2.1.1.3">)</mo></mrow></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A5.E9X.3.2.2.m1.5.5.1.1.1.2">​</mo><msub id="A5.E9X.3.2.2.m1.5.5.1.1.1.3"><mi id="A5.E9X.3.2.2.m1.5.5.1.1.1.3.2">A</mi><mi id="A5.E9X.3.2.2.m1.5.5.1.1.1.3.3">k</mi></msub><mo lspace="0em" rspace="0em" id="A5.E9X.3.2.2.m1.5.5.1.1.1.2a">​</mo><mrow id="A5.E9X.3.2.2.m1.5.5.1.1.1.4.2"><mo stretchy="false" id="A5.E9X.3.2.2.m1.5.5.1.1.1.4.2.1">(</mo><mi id="A5.E9X.3.2.2.m1.3.3">x</mi><mo id="A5.E9X.3.2.2.m1.5.5.1.1.1.4.2.2">,</mo><mi id="A5.E9X.3.2.2.m1.4.4">y</mi><mo stretchy="false" id="A5.E9X.3.2.2.m1.5.5.1.1.1.4.2.3">)</mo></mrow><mo lspace="0.167em" rspace="0em" id="A5.E9X.3.2.2.m1.5.5.1.1.1.2b">​</mo><mrow id="A5.E9X.3.2.2.m1.5.5.1.1.1.5"><mrow id="A5.E9X.3.2.2.m1.5.5.1.1.1.5.1"><mo rspace="0.167em" id="A5.E9X.3.2.2.m1.5.5.1.1.1.5.1.1">∇</mo><mi id="A5.E9X.3.2.2.m1.5.5.1.1.1.5.1.2">log</mi></mrow><mo lspace="0.167em" id="A5.E9X.3.2.2.m1.5.5.1.1.1.5a">⁡</mo><msub id="A5.E9X.3.2.2.m1.5.5.1.1.1.5.2"><mi id="A5.E9X.3.2.2.m1.5.5.1.1.1.5.2.2">π</mi><mi id="A5.E9X.3.2.2.m1.5.5.1.1.1.5.2.3">θ</mi></msub></mrow><mo lspace="0em" rspace="0em" id="A5.E9X.3.2.2.m1.5.5.1.1.1.2c">​</mo><mrow id="A5.E9X.3.2.2.m1.5.5.1.1.1.1.1"><mo stretchy="false" id="A5.E9X.3.2.2.m1.5.5.1.1.1.1.1.2">(</mo><mrow id="A5.E9X.3.2.2.m1.5.5.1.1.1.1.1.1"><mi id="A5.E9X.3.2.2.m1.5.5.1.1.1.1.1.1.2">y</mi><mo fence="false" id="A5.E9X.3.2.2.m1.5.5.1.1.1.1.1.1.1">|</mo><mi id="A5.E9X.3.2.2.m1.5.5.1.1.1.1.1.1.3">x</mi></mrow><mo stretchy="false" id="A5.E9X.3.2.2.m1.5.5.1.1.1.1.1.3">)</mo></mrow></mrow><mo id="A5.E9X.3.2.2.m1.5.5.1.1.3">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="A5.E9X.3.2.2.m1.5b">\displaystyle\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta_{k}}(\cdot|x)}\left[\frac{\pi_{\theta}(y|x)}{\pi_{\theta_{k}}(y|x)}A_{k}(x,y)\nabla\log\pi_{\theta}(y|x)\right]</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(9)</span></td>
</tr>
<tr id="A5.E9Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A5.E9Xa.2.1.1.m1.1" class="ltx_math_unparsed" alttext="\displaystyle-\gamma\mathbb{E}_{x\sim\mathcal{D}}\left[\nabla D_{\text{KL}}\pi_{\theta}(\cdot|x)||\pi_{\theta_{k}}(\cdot|x)\right]." display="inline"><semantics id="A5.E9Xa.2.1.1.m1.1a"><mrow id="A5.E9Xa.2.1.1.m1.1b"><mo id="A5.E9Xa.2.1.1.m1.1.1">−</mo><mi id="A5.E9Xa.2.1.1.m1.1.2">γ</mi><msub id="A5.E9Xa.2.1.1.m1.1.3"><mi id="A5.E9Xa.2.1.1.m1.1.3.2">𝔼</mi><mrow id="A5.E9Xa.2.1.1.m1.1.3.3"><mi id="A5.E9Xa.2.1.1.m1.1.3.3.2">x</mi><mo id="A5.E9Xa.2.1.1.m1.1.3.3.1">∼</mo><mi class="ltx_font_mathcaligraphic" id="A5.E9Xa.2.1.1.m1.1.3.3.3">𝒟</mi></mrow></msub><mrow id="A5.E9Xa.2.1.1.m1.1.4"><mo id="A5.E9Xa.2.1.1.m1.1.4.1">[</mo><mo rspace="0.167em" id="A5.E9Xa.2.1.1.m1.1.4.2">∇</mo><msub id="A5.E9Xa.2.1.1.m1.1.4.3"><mi id="A5.E9Xa.2.1.1.m1.1.4.3.2">D</mi><mtext id="A5.E9Xa.2.1.1.m1.1.4.3.3">KL</mtext></msub><msub id="A5.E9Xa.2.1.1.m1.1.4.4"><mi id="A5.E9Xa.2.1.1.m1.1.4.4.2">π</mi><mi id="A5.E9Xa.2.1.1.m1.1.4.4.3">θ</mi></msub><mrow id="A5.E9Xa.2.1.1.m1.1.4.5"><mo stretchy="false" id="A5.E9Xa.2.1.1.m1.1.4.5.1">(</mo><mo lspace="0em" rspace="0em" id="A5.E9Xa.2.1.1.m1.1.4.5.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E9Xa.2.1.1.m1.1.4.5.3">|</mo><mi id="A5.E9Xa.2.1.1.m1.1.4.5.4">x</mi><mo stretchy="false" id="A5.E9Xa.2.1.1.m1.1.4.5.5">)</mo></mrow><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E9Xa.2.1.1.m1.1.4.6">|</mo><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E9Xa.2.1.1.m1.1.4.7">|</mo><msub id="A5.E9Xa.2.1.1.m1.1.4.8"><mi id="A5.E9Xa.2.1.1.m1.1.4.8.2">π</mi><msub id="A5.E9Xa.2.1.1.m1.1.4.8.3"><mi id="A5.E9Xa.2.1.1.m1.1.4.8.3.2">θ</mi><mi id="A5.E9Xa.2.1.1.m1.1.4.8.3.3">k</mi></msub></msub><mrow id="A5.E9Xa.2.1.1.m1.1.4.9"><mo stretchy="false" id="A5.E9Xa.2.1.1.m1.1.4.9.1">(</mo><mo lspace="0em" rspace="0em" id="A5.E9Xa.2.1.1.m1.1.4.9.2">⋅</mo><mo fence="false" rspace="0.167em" stretchy="false" id="A5.E9Xa.2.1.1.m1.1.4.9.3">|</mo><mi id="A5.E9Xa.2.1.1.m1.1.4.9.4">x</mi><mo stretchy="false" id="A5.E9Xa.2.1.1.m1.1.4.9.5">)</mo></mrow><mo id="A5.E9Xa.2.1.1.m1.1.4.10">]</mo></mrow><mo lspace="0em" id="A5.E9Xa.2.1.1.m1.1.5">.</mo></mrow><annotation encoding="application/x-tex" id="A5.E9Xa.2.1.1.m1.1c">\displaystyle-\gamma\mathbb{E}_{x\sim\mathcal{D}}\left[\nabla D_{\text{KL}}\pi_{\theta}(\cdot|x)||\pi_{\theta_{k}}(\cdot|x)\right].</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="A5.SS2.SSS0.Px2.p2.9" class="ltx_p">The MDLOO algorithm can be derived by replacing the expectations in Eq. (<a href="#A5.E9" title="In Mirror descent policy optimization (MDPO). ‣ E.2 Online RL algorithm ‣ Appendix E Technical details for RLHF ‣ Apple Intelligence Foundation Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>) with the <math id="A5.SS2.SSS0.Px2.p2.6.m1.1" class="ltx_Math" alttext="nK" display="inline"><semantics id="A5.SS2.SSS0.Px2.p2.6.m1.1a"><mrow id="A5.SS2.SSS0.Px2.p2.6.m1.1.1" xref="A5.SS2.SSS0.Px2.p2.6.m1.1.1.cmml"><mi id="A5.SS2.SSS0.Px2.p2.6.m1.1.1.2" xref="A5.SS2.SSS0.Px2.p2.6.m1.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A5.SS2.SSS0.Px2.p2.6.m1.1.1.1" xref="A5.SS2.SSS0.Px2.p2.6.m1.1.1.1.cmml">​</mo><mi id="A5.SS2.SSS0.Px2.p2.6.m1.1.1.3" xref="A5.SS2.SSS0.Px2.p2.6.m1.1.1.3.cmml">K</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px2.p2.6.m1.1b"><apply id="A5.SS2.SSS0.Px2.p2.6.m1.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.6.m1.1.1"><times id="A5.SS2.SSS0.Px2.p2.6.m1.1.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.6.m1.1.1.1"></times><ci id="A5.SS2.SSS0.Px2.p2.6.m1.1.1.2.cmml" xref="A5.SS2.SSS0.Px2.p2.6.m1.1.1.2">𝑛</ci><ci id="A5.SS2.SSS0.Px2.p2.6.m1.1.1.3.cmml" xref="A5.SS2.SSS0.Px2.p2.6.m1.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px2.p2.6.m1.1c">nK</annotation></semantics></math> samples collected with <math id="A5.SS2.SSS0.Px2.p2.7.m2.1" class="ltx_Math" alttext="\pi_{\theta_{k}}" display="inline"><semantics id="A5.SS2.SSS0.Px2.p2.7.m2.1a"><msub id="A5.SS2.SSS0.Px2.p2.7.m2.1.1" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.cmml"><mi id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.2" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.2.cmml">π</mi><msub id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.cmml"><mi id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.2" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.2.cmml">θ</mi><mi id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.3" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.3.cmml">k</mi></msub></msub><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px2.p2.7.m2.1b"><apply id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1">subscript</csymbol><ci id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.2.cmml" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.2">𝜋</ci><apply id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.cmml" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.1.cmml" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3">subscript</csymbol><ci id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.2.cmml" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.2">𝜃</ci><ci id="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.3.cmml" xref="A5.SS2.SSS0.Px2.p2.7.m2.1.1.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px2.p2.7.m2.1c">\pi_{\theta_{k}}</annotation></semantics></math>, and the advantage <math id="A5.SS2.SSS0.Px2.p2.8.m3.2" class="ltx_Math" alttext="A_{k}(x,y)" display="inline"><semantics id="A5.SS2.SSS0.Px2.p2.8.m3.2a"><mrow id="A5.SS2.SSS0.Px2.p2.8.m3.2.3" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.cmml"><msub id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.cmml"><mi id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.2" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.2.cmml">A</mi><mi id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.3" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.1" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.1.cmml">​</mo><mrow id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.2" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.1.cmml"><mo stretchy="false" id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.2.1" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.1.cmml">(</mo><mi id="A5.SS2.SSS0.Px2.p2.8.m3.1.1" xref="A5.SS2.SSS0.Px2.p2.8.m3.1.1.cmml">x</mi><mo id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.2.2" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.1.cmml">,</mo><mi id="A5.SS2.SSS0.Px2.p2.8.m3.2.2" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.2.cmml">y</mi><mo stretchy="false" id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.2.3" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px2.p2.8.m3.2b"><apply id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.cmml" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3"><times id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.1.cmml" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.1"></times><apply id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.cmml" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.1.cmml" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2">subscript</csymbol><ci id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.2.cmml" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.2">𝐴</ci><ci id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.3.cmml" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.2.3">𝑘</ci></apply><interval closure="open" id="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.1.cmml" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.3.3.2"><ci id="A5.SS2.SSS0.Px2.p2.8.m3.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.8.m3.1.1">𝑥</ci><ci id="A5.SS2.SSS0.Px2.p2.8.m3.2.2.cmml" xref="A5.SS2.SSS0.Px2.p2.8.m3.2.2">𝑦</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px2.p2.8.m3.2c">A_{k}(x,y)</annotation></semantics></math> with the LOO estimator <math id="A5.SS2.SSS0.Px2.p2.9.m4.2" class="ltx_Math" alttext="\widehat{A}_{k}(x,y)" display="inline"><semantics id="A5.SS2.SSS0.Px2.p2.9.m4.2a"><mrow id="A5.SS2.SSS0.Px2.p2.9.m4.2.3" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.cmml"><msub id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.cmml"><mover accent="true" id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.cmml"><mi id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.2" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.2.cmml">A</mi><mo id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.1" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.1.cmml">^</mo></mover><mi id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.3" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.1" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.1.cmml">​</mo><mrow id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.2" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.1.cmml"><mo stretchy="false" id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.2.1" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.1.cmml">(</mo><mi id="A5.SS2.SSS0.Px2.p2.9.m4.1.1" xref="A5.SS2.SSS0.Px2.p2.9.m4.1.1.cmml">x</mi><mo id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.2.2" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.1.cmml">,</mo><mi id="A5.SS2.SSS0.Px2.p2.9.m4.2.2" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.2.cmml">y</mi><mo stretchy="false" id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.2.3" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.SS2.SSS0.Px2.p2.9.m4.2b"><apply id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3"><times id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.1.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.1"></times><apply id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2"><csymbol cd="ambiguous" id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.1.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2">subscript</csymbol><apply id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2"><ci id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.1.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.1">^</ci><ci id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.2.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.2.2">𝐴</ci></apply><ci id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.3.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.2.3">𝑘</ci></apply><interval closure="open" id="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.1.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.3.3.2"><ci id="A5.SS2.SSS0.Px2.p2.9.m4.1.1.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.1.1">𝑥</ci><ci id="A5.SS2.SSS0.Px2.p2.9.m4.2.2.cmml" xref="A5.SS2.SSS0.Px2.p2.9.m4.2.2">𝑦</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.SS2.SSS0.Px2.p2.9.m4.2c">\widehat{A}_{k}(x,y)</annotation></semantics></math> in Eq. (<a href="#A5.E7" title="In Leave-One-Out (LOO) estimator of the advantage. ‣ E.2 Online RL algorithm ‣ Appendix E Technical details for RLHF ‣ Apple Intelligence Foundation Language Models" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>). Empirically, we find that MDLOO works better than the popular PPO <cite class="ltx_cite ltx_citemacro_citep">[Schulman et al., <a href="#bib.bib45" title="" class="ltx_ref">2017</a>]</cite> algorithm in our setting.</p>
</div>
</section>
</section>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Accuracy-recovery adapters ablation</h2>

<div id="A6.p1" class="ltx_para">
<p id="A6.p1.1" class="ltx_p">In this section, we present the evaluation results on unquantized, quantized, and accuracy-recovered models. As shown in <a href="#A6.T10" title="Table 10 ‣ Appendix F Accuracy-recovery adapters ablation ‣ Apple Intelligence Foundation Language Models" class="ltx_ref ltx_refmacro_autoref"><span class="ltx_text ltx_ref_tag">Table 10</span></a>, the quantized models have huge quality drops in both pre-train and post-train metrics. By using accuracy-recovery LoRA adapters with only rank 16, Alpaca win rate can be improved by 7-18%, GMS8K accuracy is boosted by 5-10%.
The recovered models perform much closer to the original unquantized model while achieving significant reductions on the model size.
More interestingly, we observe that when the quantization scheme becomes more aggressive (from 3.7 to 3.5 bpw), the adapters also recover more quality back.</p>
</div>
<figure id="A6.T10" class="ltx_table">
<div id="A6.T10.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:107.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-1.0pt,0.3pt) scale(0.995367248003808,0.995367248003808) ;">
<table id="A6.T10.1.1" class="ltx_tabular ltx_align_middle">
<tr id="A6.T10.1.1.1" class="ltx_tr">
<td id="A6.T10.1.1.1.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">BPW</span></td>
<td id="A6.T10.1.1.1.2" class="ltx_td ltx_align_left ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.1.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Models</span></td>
<td id="A6.T10.1.1.1.3" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.1.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">IFEval Instruction-Level</span></td>
<td id="A6.T10.1.1.1.4" class="ltx_td ltx_align_right ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.1.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">AlpacaEval 2.0 LC</span></td>
<td id="A6.T10.1.1.1.5" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.1.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">GSM8K (8-shot CoT)</span></td>
</tr>
<tr id="A6.T10.1.1.2" class="ltx_tr">
<td id="A6.T10.1.1.2.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">16</span></td>
<td id="A6.T10.1.1.2.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.2.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">AFM-on-device</span></td>
<td id="A6.T10.1.1.2.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.2.3.1" class="ltx_text" style="font-size:90%;">100.0%</span></td>
<td id="A6.T10.1.1.2.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.2.4.1" class="ltx_text" style="font-size:90%;">100.0%</span></td>
<td id="A6.T10.1.1.2.5" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.2.5.1" class="ltx_text" style="font-size:90%;">100.0%</span></td>
</tr>
<tr id="A6.T10.1.1.3" class="ltx_tr">
<td id="A6.T10.1.1.3.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;" rowspan="2"><span id="A6.T10.1.1.3.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">3.5</span></td>
<td id="A6.T10.1.1.3.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.3.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">quantized</span></td>
<td id="A6.T10.1.1.3.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.3.3.1" class="ltx_text" style="font-size:90%;">98.4%</span></td>
<td id="A6.T10.1.1.3.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.3.4.1" class="ltx_text" style="font-size:90%;">76.7%</span></td>
<td id="A6.T10.1.1.3.5" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.3.5.1" class="ltx_text" style="font-size:90%;">82.2%</span></td>
</tr>
<tr id="A6.T10.1.1.4" class="ltx_tr">
<td id="A6.T10.1.1.4.1" class="ltx_td ltx_align_left" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.4.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Acc.-recovered (rank 16)</span></td>
<td id="A6.T10.1.1.4.2" class="ltx_td ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.4.2.1" class="ltx_text" style="font-size:90%;">98.8%</span></td>
<td id="A6.T10.1.1.4.3" class="ltx_td ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.4.3.1" class="ltx_text" style="font-size:90%;">94.7%</span></td>
<td id="A6.T10.1.1.4.4" class="ltx_td ltx_nopad_r ltx_align_right" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.4.4.1" class="ltx_text" style="font-size:90%;">92.1%</span></td>
</tr>
<tr id="A6.T10.1.1.5" class="ltx_tr">
<td id="A6.T10.1.1.5.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;" rowspan="2"><span id="A6.T10.1.1.5.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">3.7</span></td>
<td id="A6.T10.1.1.5.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.5.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">quantized</span></td>
<td id="A6.T10.1.1.5.3" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.5.3.1" class="ltx_text" style="font-size:90%;">97.9%</span></td>
<td id="A6.T10.1.1.5.4" class="ltx_td ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.5.4.1" class="ltx_text" style="font-size:90%;">87.3%</span></td>
<td id="A6.T10.1.1.5.5" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.5.5.1" class="ltx_text" style="font-size:90%;">91.3%</span></td>
</tr>
<tr id="A6.T10.1.1.6" class="ltx_tr">
<td id="A6.T10.1.1.6.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.6.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Acc.-recovered (rank 16)</span></td>
<td id="A6.T10.1.1.6.2" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.6.2.1" class="ltx_text" style="font-size:90%;">100.6%</span></td>
<td id="A6.T10.1.1.6.3" class="ltx_td ltx_align_right ltx_border_bb" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.6.3.1" class="ltx_text" style="font-size:90%;">94.8%</span></td>
<td id="A6.T10.1.1.6.4" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb" style="padding-top:0.9pt;padding-bottom:0.9pt;"><span id="A6.T10.1.1.6.4.1" class="ltx_text" style="font-size:90%;">96.0%</span></td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 10: </span>Evaluation results for quantized and accuracy-recovered models. Numbers are normalized to the unquantized version.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section><div class="ltx_rdf" about="" property="dcterms:creator" content="Apple"></div>
<div class="ltx_rdf" about="" property="dcterms:title" content="Apple Intelligence Foundation Language Models"></div>

</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.21074" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.21075" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.21075">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.21075" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.21076" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 14:53:26 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.21075
We present foundation language models developed to power Apple Intelligence features, including a3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute.
These models are designed to perform a wide range of tasks efficiently, accurately, and responsibly.
This report describes the model architecture, the data used to train the model, the training process, how the models are optimized for inference, and the evaluation results. We highlight our focus on Responsible AI and how the principles are applied throughout the model development.
abstract
Apple Intelligence Foundation Language Models
Abstract
2407.21075
1IntroductionAt the 2024 Worldwide Developers Conference, we introduced Apple Intelligence, a personal intelligence system integrated deeply into iOS 18, iPadOS 18, and macOS Sequoia.Apple Intelligence consists of multiple highly-capable generative models that are fast, efficient, specialized for our users’ everyday tasks, and can adapt on the fly for their current activity. The foundation models built into Apple Intelligence have been fine-tuned for user experiences such as writing and refining text, prioritizing and summarizing notifications, creating playful images for conversations with family and friends, and taking in-app actions to simplify interactions across apps.Figure 1:Modeling overview for the Apple foundation models.In this report we will detail how two of these models—AFM-on-device (AFM stands forApple Foundation Model) , a3 billion parameter language model, and AFM-server, a larger server-based language model—have been built and adapted to perform specialized tasks efficiently, accurately, and responsibly (Figure 1) . These two foundation models are part of a larger family of generative models created by Apple to support users and developers; this includes a coding model (based on an AFM language model)  to build intelligence into Xcode, as well as a diffusion model to help users express themselves visually, for example, in the Messages app.Apple Intelligence is designed with Apple’s core values at every step and built on a foundation of industry-lead privacy protection. Additionally, we have created Responsible AI principles to guide how we develop AI tools, as well as the models that underpin them:1.Empower users with intelligent tools: We identify areas where AI can be used responsibly to create tools for addressing specific user needs. We respect how our users choose to use these tools to accomplish their goals.2.Represent our users: We build deeply personal products with the goal of representing users around the globe authentically. We work continuously to avoid perpetuating stereotypes and systemic biases across our AI tools and models.3.Design with care: We take precautions at every stage of our process, including design, model training, feature development, and quality evaluation to identify how our AI tools may be misused or lead to potential harm. We will continuously and proactively improve our AI tools with the help of user feedback.4.Protect privacy: We protect our users’ privacy with powerful on-device processing and groundbreaking infrastructure like Private Cloud Compute. We do not use our users’ private personal data or user interactions when training our foundation models.These principles are reflected at every stage of the architecture that enables Apple Intelligence and connects features and tools with specialized models.In the remainder of this report, we provide details on decisions such as: how we develop models that are highly capable, fast, and power-efficient; how we approach training these models; how our adapters are fine-tuned for specific user needs; and how we evaluate model performance for both helpfulness and unintended harm.
introduction
Apple Intelligence Foundation Language Models
1Introduction
2407.21075
2ArchitectureThe AFM base models are dense decoder-only models that build on the Transformer architecture, with the following design choices:•A shared input/output embedding matrixto reduce memory usage for parameters.•Pre-Normalizationwith RMSNormfor training stability.•Query/key normalizationto improve training stability.•Grouped-query attention (GQA) with 8 key-value heads to reduce the KV-cache memory footprint.•The SwiGLU activationfor higher efficiency.•RoPEpositional embeddings with the base frequency set tofor long-context support.Table 1provides some details about AFM-on-device dimensions.Model dimensionHead dimensionNum query headsNum key/value headsNum layersNum non-embedding params (B) Num embedding params (B) Table 1:AFM-on-device dimensions.
body
Apple Intelligence Foundation Language Models
2Architecture
2407.21075
3Pre-trainingOur AFM pre-training process plays a critical role in developing highly capable language models to power a host of Apple Intelligence features that can help and support users. We focus on efficiency and data quality at every step in order to pre-train for a high-quality end-to-end user experience with efficient and low-latency models.3.1DataThe AFM pre-training dataset consists of a diverse and high quality data mixture.
This includes data we have licensed from publishers, curated publicly-available or open-sourced datasets, and publicly available information crawled by our web-crawler, Applebot. We respect the right of webpages to opt out of being crawled by Applebot, using standard robots.txt directives.Given our focus on protecting user privacy, we note that no private Apple user data is included in the data mixture. Additionally, extensive efforts have been made to exclude profanity, unsafe material, and personally identifiable information from publicly available data (seeSection 7for more details) . Rigorous decontamination is also performed against many common evaluation benchmarks.We find that data quality, much more so than quantity, is the key determining factor of downstream model performance. In the following, we provide more details about key components of the data mixture.3.1.1Web pagesWe crawl publicly available information using our web crawler, Applebot, and respect the rights of web publishers to opt out of Applebot using standard robots.txt directives.
Plus, we take steps to exclude pages containing profanity and apply filters to remove certain categories of personally identifiable information (PII) .
The remaining documents are then processed by a pipeline which performs quality filtering and plain-text extraction, more specifically:1.Body extraction is performed using a combination of Safari’s reader mode and the Boilerpipealgorithm.2.Safety and profanity filtering, using heuristics and model-based classifiers.3.Global fuzzy de-duplication using locality-sensitive n-gram hashing.4.Extensive quality filtering using heuristics and model-based classifiers.5.Decontamination against 811 common pre-training benchmarks, filtering entire documents upon 4-13 gram collisions with any of the benchmark datasets, unless the collision-count for a given n-gram reaches a “common-usage" threshold of 1000.3.1.2Licensed datasetsWe go to lengths to identify and license a limited amount of high-quality data from publishers.
These licensed datasets provide a natural source of diverse and high quality long-context data, so we include them as part of the data mixture for the continued and context-lengthening stages of pre-training (seeSection 3.2.2and3.2.3for more details) .
We decontaminate sections of publisher licensed data the same way we decontaminate web pages (Section 3.1.1) .3.1.3CodeCode data is obtained from license-filtered111Using MIT, Apache, BSD, CC0, CC-BY, Unlicensed, ISC, and Artistic Licenses.open source repositories on GitHub. The bulk of the code data covers 14 common programming languages, including: Swift, Python, C, Objective-C, C++, JavaScript, Java, and Go.
The data is de-duplicated, further filtered for PII and quality, and decontaminated in the same fashion as inSection 3.1.1.3.1.4MathWe integrate two categories of high-quality data sourced from the web. The first category is a Math Q&A dataset, comprising 3 billion tokens from 20 web domains rich in math content. We extract the questions and answers by identifying relevant tags from HTML pages. The second category is a collection of 14 billion tokens from web pages such as math forums, blogs, tutorials, and seminars. To filter these web pages, we used a specialized pipeline that includes a math tag filter with a collection of 40 strings to identify mathematical templates, a math symbol filter with a collection of 350 Unicode and LaTeX symbols to identify math content, a quality filter powered by a language model classifier specifically designed for math, and a domain filter that processes all web pages from domains manually labeled by humans. We applied these filters, followed by deduplication, decontamination, and PII removal to produce the final dataset.3.1.5Public datasetsWe evaluated and selected a number of high-quality publicly-available datasets with licenses that permit use for training language models.
Then, we filtered the datasets to remove personally identifiable information before including them in the pre-training mixture.3.1.6TokenizerWe use abyte-pair encoding(BPE)  tokenizer, following the implementation from SentencePiece. All numbers are split into individual digits and we use byte-fallback to decompose unknown UTF-8 characters into byte tokens. We do not enable Unicode normalization.
The total vocabulary size is 100k and 49k tokens for AFM-server and AFM-on-device, respectively.3.2RecipeWe break AFM pre-training into three distinct stages: 1.corewhich consumes most of the compute budget, 2.continued, where we down-weight the lower-quality bulk web-crawl data, favoring a higher code and math weight instead combined with inclusion of the licensed data described inSection 3.1.2, 3.context-lengtheningwhich is similar to anothercontinuedpre-training stage, but conducted at longer sequence length and with synthetic long-context data included in the mixture.Details about model quality after each of the three pre-training stages (alongside additional metrics for AFM derived from our internal benchmark implementations)  are inAppendix C, andAppendix Dexamines AFM-server’s long-context capabilities.All three stages use decoupled weight decayfor regularization, as well as a simplified version ofParam, similar to what is described asParam (simple)  in. Thus far we have not found more sophisticated parameter norm controls to be necessary at these scales. All stages maintain sharded model and optimizer states infloat32, casting tobfloat16for the forward and backward passes for efficiency.3.2.1Core pre-trainingAFM-server core training is conducted from scratch, while AFM-on-device is distilled and pruned from a larger model.AFM-server:We train AFM-server from scratch for 6.3T tokens on 8192 TPUv4 chips, using a sequence length of 4096 and a batch-size of 4096 sequences. The batch size was determined using a scaling law fit to model size and compute budget, however we find that downstream results are relatively insensitive to a fairly wide range of batch sizes, and expect that any value betweenandthe predicted batch size would have yielded similar results (the predicted optimum was in fact3072, but 4096 allowed for better chip utilization) .
We perform a learning rate sweep using a proxy model with a model dimension of 768, finding that the optimum learning rate spans 0.01-0.02, so we choose 0.01 to be conservative. Linear layers will have an effective learning rate scaled by0.1 due to the use ofParam (simple) .222In scaling law experiments we find thatParam (simple)  stabilizes the optimal learning rate as model size increases, although extrapolating to very significantly deeper and/or larger models does exhibit a slight left-shift beyond what is accounted for.We use a tuned decoupled weight decay of, finding that it works well across all tested model sizes and compute budgets. The learning rate schedule includes a linear warm-up for 5000 steps, followed by cosine decay to 0.005 of the peak over the remainder of training. For further details on the optimizer, seeSection 3.2.4.Appendix Acompares the AFM core pre-training recipe to a more typical configuration.AFM-on-device:For the on-device model, we found that knowledge distillationand structural pruning are effective ways to improve model performance and training efficiency. These two methods are complementary to each other and work in different ways. More specifically, before training AFM-on-device, we initialize it from a pruned 6.4B model (trained from scratch using the same recipe as AFM-server) , using pruning masks that are learned through a method similar to what is described in. The key differences are: (1)  we only prune the hidden dimension in the feed-forward layers; (2)  we use Soft-Top-K maskinginstead of HardConcrete masking; (3)  we employ the same pre-training data mixture as the core phase to learn the mask, training for 188B tokens. Then, during the core pre-training of AFM-on-device, a distillation loss is used by replacing the target labels with a convex combination of the true labels and the teacher model’s top-1 predictions, (with 0.9 weight assigned to the teacher’s labels) , training for a full 6.3T tokens. We observe that initializing from a pruned model improves both data efficiency and the final benchmark results by 0-2%, whilst adding distillation boosts MMLU and GSM8K by about 5% and 3% respectively. More detailed ablation results can be found inAppendix B. All training hyper-parameters except for batch-size are kept the same as AFM-server.3.2.2Continued pre-trainingFor both models we perform continued pre-training at a sequence length of 8192, with another 1T tokens from a mixture that upweights math and code, and down-weights the bulk web-crawl. We also include the licensed data described inSection 3.1.2. We use a peak learning rate ofand decoupled weight decay of, and 1000 warm-up steps with a final learning rate decay to 0.001 of peak, differently to core pre-training. Other settings (batch size, etc)  are carried over. We did not find a distillation loss to be helpful here for AFM-on-device, unlike in core pre-training, so the recipe is identical to that used for AFM-server.3.2.3Context lengtheningFinally, we conduct a further 100B tokens of continued pre-training at a sequence length oftokens, using the data mixture from the continued pre-training stage, augmented with synthetic long-context Q&A data. We also increase the RoPE base frequency from 500k to, following the scaling laws described in, with the expectation that this will allow for better short-to-long generalization—which is desirable given that the majority of our pre-training data is comprised of documents that are significantly shorter than 32k tokens long. The recipe is similar to that used for continued pre-training. We examine the long-context performance of AFM-server inAppendix D.3.2.4OptimizerWe choose to use a variant of RMSPropwith momentum for AFM pre-training. In particular, we divide the raw gradient by the square-root of a bias-corrected exponential moving average of the squared gradient to produce an instantaneous update, which is clipped to a maximum norm ofper parameter block, before then further smoothing this estimate over steps with an exponential moving average without bias-correction to produce the net update. Unless otherwise noted, the smoothing constants for both the squared gradient ()  and the update ()  are set to. A small constantis added to the instantaneous squared gradient prior to smoothing, for numerical stability.The smoothed updates are scaled by the learning rate, weight-decay is added, and then scheduled decay is applied to form the final weight delta. As an additional guard for stability, prior to the optimizer we clip the global gradient norm to. For a recipe ablation against a more typical configuration, seeAppendix A.3.3Training infrastructureThe AFM models are pre-trained on v4 and v5p Cloud TPU clusters with the AXLearn framework, a JAXbased deep learning library designed for the public cloud.
Training is conducted using a combination of tensor, fully-sharded-data-parallel, and sequence parallelism, allowing training to scale to a large number of model parameters and sequence lengths at high utilization.
This system allows us to train the AFM models efficiently and scalably, including AFM-on-device, AFM-server, and larger models.AFM-server was trained on 8192 TPUv4 chips provisioned aschip slices, where slices are connected together by the data-center network (DCN) . Only data-parallelism crosses the slice boundary, other types of state sharding are within-slice only as the within-slice interconnect bandwidth is orders of magnitude higher than the DCN. The sustained model-flop-utilization (MFU)  for this training run was approximately 52%. AFM-on-device was trained on one slice of 2048 TPUv5p chips.
body
Apple Intelligence Foundation Language Models
3Pre-training
2407.21075
4Post-TrainingWhile Apple Intelligence features are powered through adapters on top of the base model (seeSection 5for a deep-dive on the adapter architecture) , empirically we found that improving the general-purpose post-training lifts the performance of all features, as the models have stronger capabilities on instruction following, reasoning, and writing.We conduct extensive research in post-training methods to instill general-purpose instruction following and conversation capabilities to the pre-trained AFM models.
Our goal is to ensure these model capabilities are aligned with Apple’s core values and principles, including our commitment to protecting user privacy, and our Responsible AI principles.
Our post-training efforts include a series of data collection and generation, instruction tuning, and alignment innovations.
Our post-training process contains two stages: supervised fine-tuning (SFT)  and reinforcement learning from human feedback (RLHF) .
We present two new post-training algorithms: (1)  a rejection sampling fine-tuning algorithm with teacher committee (iTeC) , and (2)  a reinforcement learning from human feedback (RLHF)  algorithm with mirror descent policy optimization and a leave-one-out advantage estimator (MDLOO)  that are used on our reinforcement learning iterations and lead to significant model quality improvements.4.1DataWe use a hybrid data strategy in our post-training pipeline, which consists of both human annotated and synthetic data.
Throughout our data collection and experiment process, we have found data quality to be the key to model success and thus have conducted extensive data curation and filtering procedures.4.1.1Human annotationsDemonstration dataTo fuel the instruction fine-tuning of AFM,
we collect high-quality human annotated demonstration datasets from various sources.
This dialogue-style data consists of both system-level and task-level instructions (a.k.a. prompts) ,
as well as their corresponding responses.
Similar to,
we observe quality to weigh more importantly than quantity in our experiments.
As a result,
we focus on key data quality criteria including helpfulness, harmlessness, presentation, and response accuracy,
in addition to targeting a diverse task distribution covering Apple Intelligence features.
To protect user privacy,
we take steps to verify no personally identifiable information is present in our data, and we do not include any personal data stored by users with Apple.Human preference feedbackTo iteratively improve AFM’s capabilities,
we further collect human feedback for reinforcement learning.
In particular,
we instruct human annotators to compare and rank two model responses for the same prompt to collect side-by-side preference labels.
In addition, we also use single-side questions to guide this process.
These questions inform raters to grade the model response quality of various aspects including instruction following, safety, factuality, and presentation,
and we also retain these labels for model training.
We emphasize Apple values and standards in the process.
Similar to demonstration data,
we find data quality to be crucial for feedback data,
and thus we iterate data and model qualities jointly to improve them in a unified flywheel.4.1.2Synthetic dataIn addition to human annotations, we delve into enhancing data quality and diversity through synthetic data generation. Our findings suggest that when guided by our robust reward models, AFMs are capable of generating high quality responses and for some specific domains, these responses are found to be on par with, or even superior to, human annotations. Therefore, we extend our prompt set to increase the diversity and find that those generated responses can benefit AFMs themselves. In the following, we discuss three domains where we generate synthetic data for AFM post-training: mathematics, tool use, and coding.MathematicsIn the field of mathematics, the wide-ranging subjects and difficulty level make it exceptionally resource-intensive for collecting human demonstrations, since it requires expert knowledge from the human writers. It also becomes impractical to solely rely on human-written content as the model continuously improves. As a consequence, exploring the potential of synthetic data becomes essential to effectively address the challenges.The creation of synthetic data for mathematics involves two primary stages: generating synthetic math problems and producing their corresponding solutions. For math problem synthesis, we employ several “evolution" strategies where a seed set of prompts are transformed into a much larger set of diverse prompts:Problem rephrase and reversion.Following the approach in, we prompt AFM to rephrase seed math questions, and curate reverse questions to derive a specific number in a raw problem statement when provided with the final answer.Problem evolution.Inspired by the instruction evolving technique, given a seed problem setwe prompt AFM to generate two distinct sets of math problems, i.e., and. The in-depth evolution enhances instructions by adding complexities while the in-breadth evolution improves the topic coverage.
For bothand, we first perform de-duplication with an embedding model, and subsequently prompt LLMs to ensure the coherence and solvability of the math problems. In addition, fora difficulty level is assigned and we only select math problems that score above a specified threshold.With an augmented set of math questions, we then prompt AFM to synthesizeresponses with chain-of-thought per question. If the initial seed data has ground truth, they can be used as an “outcome reward signal” to filter synthesized answers.
For problems that require less reasoning steps, we observe that a correct final answer often gets associated with correct intermediate steps. If direct answer checking is unsuccessful or ground truth is unavailable, we instead assess the response correctness by querying an LLM judge. We find that the filtered answers, when fed into the training data, boost our models’ math capabilities by a large margin.Tool useWe develop tool-use capabilities such as function call, code interpreter, and browsing through a mixture of synthetic and human data.
The model capabilities are first bootstrapped with synthetic data, which focuses on single-tool use cases.
We then collect human annotations to improve model capabilities that involve multi-tool and multi-step scenarios.
We further augment the human curated function call data by mixing the oracle tool with other similar tools to increase the difficulty of tool selection.
In addition, we synthesize parallel function call from human curated function call data to enable the new capability and tool intent detection data based on human curated function call and general SFT data to mitigate tool call over-triggering issues.CodingThe generation of a synthetic coding dataset involves a self-instruct method with rejection sampling. This approach enables the model to learn and generate data autonomously. Starting with 71 different programming topics as the seeds, the model is prompted to generate an initial pool of coding interview-like questions. For each question, the model generates a set of unit tests and a number of potential solutions. We then use an execution-based rejection sampling method to select the best solution. This involves compiling each potential solution with every unit test and executing them. The solution with the highest number of successful executions is chosen. This results in a collection of (question, test cases, solution)  triplets. At the end, we validate the quality of the dataset by filtering the triplets using the number of passed unit tests, resulting in 12K high quality triplets used in the SFT.4.2Supervised fine-tuning (SFT) It has been shownthat scaling multi-task instruction tuning dramatically enhances model performance on a wide variety of tasks. Similarly, we attempt to scale supervised fine-tuning data to achieve a strong base model for subsequent alignment. During SFT, we collect and train models on demonstration data of a given prompt333A prompt may consist of the most recent user instruction as well as all previous user-model-system interactions..
We carefully select and combine both human data and synthetic data to form a high quality mixture that covers various natural language use cases.Data selectionWe establish a series of quality guards of the data before onboarding them for model training, including ratings from in-house human labelers, automatic model-based filtering techniques, and deduplication with text embeddings. We also scale up the mixture size by a variety of synthetic data generation methods, as described inSection 4.1.2, and rejection sampling as described inSection 4.3.2.Tuning the mixture ratioIn order to tune the mixture weight, we treat it as an optimization problem. Specifically, given a set of weightswhererepresents the ratio of a specific component in the mixture, we train a model withand evaluate the quality change on a set of benchmarks.
We find that extensively running such experiments can effectively identify the best mixture and remove the least impactful data components.Training hyperparametersThe model is trained with a constant learning ratefor AFM-server andfor AFM-device models, as well as a drop out rate. Since the evaluation metrics fluctuate across different checkpoints, we run checkpoint selection based on automatic evaluation benchmarks and best-of-N selection with reward models to test the headroom for RL.4.3Reinforcement learning from human feedback (RLHF) We further use reinforcement learning with collected human preference data to improve model performance and quality.
This involves training a robust reward model and applying it in two algorithms of iTeC and MDLOO that we discuss below. We describe more details of our RLHF pipeline inAppendix E.4.3.1Reward modelingWe train reward models using the human preference data collected with the method inSection 4.1.1.
Each human preference data item contains one prompt and two responses along with human labels including:•The preferred response between the two and the preference level, i.e., whether the preferred response is significantly better, better, slightly better, or negligibly better than the rejected response.•The single-sided grading of each response, measuring the instruction following property, the conciseness, truthfulness, and harmlessness of each of the responses.Our reward model training follows the standard practice of reward modeling in RLHF with two main innovations:•We design a soft label loss function that takes the level of human preference into account.•We incorporate single-sided gradings as regularization terms in reward modeling.We employ the commonly used Bradley-Terry-Luce (BTL)  modelfor reward modeling in RLHF. In this model, the probability that a human annotator prefers one response over another is modeled as the sigmoid function of the difference of the rewards. Oursoft labelloss function encourages that this probability is high when the preference level is high, e.g., when one response is significantly better than the other, and vice versa. We note that this is different from themargin-based loss function in Llama 2, which also leverages the preference level. Empirically, we find that our method works better than the margin-based loss function. Moreover, we also find that using the single-sided gradings as regularization terms can effectively improve the accuracy of the reward model. More details of our reward modeling techniques can be found inSection E.1.4.3.2Iterative teaching committee (iTeC) To fully unlock the ability of our model with multiple rounds of RLHF,
we propose a novel iterative RLHF framework which effectively combines various preference optimization algorithms,
including rejection sampling (RS) , Direct Preference Optimization (DPO) and its variants such as IPO, and online reinforcement learning (RL) .
This enables us to bring the benefit of RLHF to AFM models across all sizes and improve their alignment at the same time.Iterative committeeOne of the most important lessons we learned from developing AFM RLHF
is to refresh online human preference data collection using a diverse set of the best performing models.
Specifically, for each batch of human preference data collection,
we set up a collection of latest promising models trained from SFT, RS, DPO/IPO, and RL,
as well as best models from the previous iterations,
which we refer to as “model committee”.
We collect pairwise human preference on responses sampled from the latest model committee.After acquiring each batch of human preference data,
we refresh our reward model, and further train a new set of models
using the collection of preference optimization algorithms.
We then continue the next round of iterative RLHF data collection with a new model committee.Committee distillationWe further run rejection sampling (distillation)  from the model committee with the latest reward model as a reranker.
Instead of reranking at global-level, i.e., picking a single best performing model from the committee and using it as a teacher model,
we rerank model responses at the prompt-level.
Specifically, for each prompt,
we sample multiple responses from each model in the committee,
and use the latest reward model to select the best response for each prompt.
This allows us to combine the advantages of models trained by different preference optimization algorithms. For instance,
we find that algorithms that leverage negative examples, e.g., online RLHF, DPO, IPO, to be better in improving reasoning skills such as math, while rejection sampling fine-tuning learns instruction following and writing skills more effectively.Scaling up distillationIn order to bring the RLHF improvements to AFM models across all sizes,
we scale up distillation from the model committee.
Different from larger models, where carefully iterating data and model quality matters much more than data quantity,
we find smaller models can achieve tremendous improvement when we scale up the number of prompts for distillation.
Our final AFM-on-device model is trained on more than 1M high quality responses generated from the model committee.4.3.3Online RLHF algorithm: MDLOOIn this section, we introduce our online reinforcement learning algorithm MDLOO,
where we decode responses during model training and apply RL algorithms to maximize the reward.We use the commonly adopted RLHF objective that maximizes the KL-penalized reward function:(1) whereis the prompt distribution,denotes the Kullback-Leibler divergence between two distributions, andis the coefficient that controls the divergence between the behavior policyand a reference policy, which is typically a model trained by SFT. In our RL training, we use the reward function(2) whose expectation is equivalent toEquation 1. We consider the bandit setting where the generation of the entire response is considered as one action, and we do not use the value network (a.k.a. the critic)  to obtain the per-token reward or advantage.Similar to commonly used RLHF algorithms such as PPO, we use a trust-region based policy iteration algorithm.
We made two main design choices in our online RL algorithm:•We use the Leave-One-Out (LOO)  estimator to estimate the advantage of a prompt-response pair, similar to a recent work.•We use Mirror Descent Policy Optimization (MDPO) to optimize the policy, differently from the more commonly used clipping-based PPO method.Thus, we name our online RL algorithmMirror Descent with Leave-One-Out estimation(MDLOO) . More specifically, during the decoding stage of the algorithm, we decode multiple responses for each prompt, and assign theadvantageof each response to be the difference of the reward of the (prompt, response)  pair and the mean reward of the other responses generated by the same prompt. Intuitively, this estimator aims to measure how much better a response is compared to a typical response. Empirically, we find this advantage estimator crucial for stabilizing the RL algorithm and achieving strong results.
Moreover, we use a KL-regularization-based trust region method, i.e. MDPO, to control the policy change in each iteration. We find that this algorithm is more effective than PPO in our setting. More details of our online RLHF algorithm can be found inSection E.2.
body
Apple Intelligence Foundation Language Models
4Post-Training
2407.21075
5Powering Apple Intelligence featuresOur foundation models are designed for Apple Intelligence, the personal intelligence system integrated into supported models of iPhone, iPad, and Mac.
We have built these models to be fast and efficient. And while we have achieved impressive levels of broad capability in our base model, the actual relevant measure of its quality is how it performs on specific tasks across our operating systems.Here we have found that we can elevate the performance of even small models to best-in-class performance through task-specific fine-tuning and have developed an architecture, based on runtime-swappable adapters, to enable the single foundation model to be specialized for dozens of such tasks. A high-level overview is presented inFigure 2.Figure 2:Architecture of Apple Intelligence with adapters for the language on-device and server models and the image models. In this report we are only describing the text models.5.1Adapter architectureOur foundation models are fine-tuned for users’ everyday activities, and can dynamically specialize themselves on-the-fly for the task at hand. We use LoRAadapters, small neural network modules that can be plugged into various layers of the base model, to fine-tune our models for specific tasks.
For each task, we adapt all of AFM’s linear projection matrices in the self-attention layers and the fully connected layers in the pointwise feedforward networks.
By fine-tuning only the adapters, the original parameters of the base pre-trained model remain unchanged, preserving the general knowledge of the model while tailoring the adapters to support specific tasks.We represent the values of the adapter parameters using 16 bits, and for the3 billion parameter on-device model, the parameters for a rank 16 adapter typically require 10s of megabytes. The adapter models can be dynamically loaded, temporarily cached in memory, and swapped—giving our foundation model the ability to specialize itself on the fly for the task at hand while efficiently managing memory and guaranteeing the operating system’s responsiveness.To facilitate the training of the adapters, we created an efficient infrastructure that allows us to rapidly add, retrain, test, and deploy adapters when the base model or the training data gets updated or new capabilities are required. It is worth noting that the adapter parameters are initialized using the accuracy-recovery adapter introduced inSection 5.2.5.2OptimizationsThe AFM models are designed to support our users throughout their daily activities, and both inference latency and power efficiency are important for the overall user experience.
We apply various optimization techniques to allow AFM to be efficiently deployed on-device and in Private Cloud Compute. These techniques significantly reduce memory, latency, and power usage while maintaining the overall model quality.In order to fit AFM into a constrained memory budget of edge devices and reduce inference cost, it is critical to apply model quantization techniques to reduce the effective bits per weight while maintaining the model quality. Previous works have found that 4-bit quantized models only have marginal loss of quality (typically measured in pre-training metrics)  compared to the original 32/16-bit float-point versions. Since AFM is expected to support a diverse set of product features, it is essential that the quantized model retains capabilities in specific domains critical to these use cases. To achieve an optimal trade-off between model capacity and inference performance, we have developed state-of-the-art quantization methods and a framework that utilizes accuracy-recovery adapters. This allows us to achieve near-lossless quantization that is on average less than 4 bit-per-weight, and provides flexible quantization scheme choices.MethodsThe model is compressed and quantized, on average under 4-bit-per-weight, after the post-training stages (details of the quantization scheme will be discussed later) . The quantized model often shows a moderate level of quality loss. Therefore, instead of directly passing the quantized model to application teams for feature development, we attach a set of parameter-efficient LoRA adapters for quality recovery. We make sure that these LoRA adapters training recipes are consistent with pre-training and post-training processes. Then, products will fine-tune their own feature-specific LoRA adapters by initializing the adapter weights from the accuracy-recovery adapters, while keeping the quantized base model frozen.It is noteworthy that training accuracy-recovery adapters is sample-efficient and can be considered as a mini-version of training the base model. During the pre-training stage of the adapters, we only require approximately 10 billion tokens (of base model training)  to fully recover the capacity for the quantized model. Since application adapters will fine tune from these accuracy-recovery adapters, they do not incur any additional memory usage or inference costs. Regarding adapter size, we found that adapter rank of 16 offers the optimal tradeoff between model capacity and inference performance. However, to provide flexibility for various use cases, we provide a suite of accuracy-recovery adapters in different ranksfor application teams to select from. InAppendix F, we provide detailed evaluation results across unquantized, quantized, and accuracy-recovered models and show that the recovered models perform much closer to the unquantized version.Quantization schemesAnother benefit brought by accuracy-recovery adapters is that they allow more flexible choices of quantization schemes. Previously when quantizing LLMs, people typically group the weights into small blocks, normalize each block by the corresponding maximal absolute values to filter out outliers, then apply quantization algorithms in a block basis. While a larger block size yields lower effective bits per weight and a higher throughput, the quantization loss would increase. In order to balance this tradeoff, it is common to set block size to a small value, like 64 or 32. In our experiments, we found that accuracy-recovery adapters can greatly improve the pareto frontier in the tradeoff. More errors will be recovered for more aggressive quantization schemes. As a result, we are able to use a highly-efficient quantization scheme for AFM without worrying about losing model capacity. Specifically, our AFM-on-device model running on Apple Neural Engine (ANE)  uses palettization: for projection weights, every 16 columns/rows share the same quantization constants (i.e., lookup tables)  and are quantized using K-means with 16 unique values (4-bit) . The quantization block size can be up to 100k. Besides, since AFM’s embedding layer is shared between the input and output, it is implemented differently from projection layers on ANE. Hence, we quantize the embedding using per-channel quantization with 8-bit integers for better efficiency.Mixed-precision quantizationResidual connections exist in every transformer block and every layer in AFM. So it is unlikely that all layers have the equal importance. Following this intuition, we further reduce the memory usage by pushing some layers to use 2-bit quantization (default is 4-bit) . On average, AFM-on-device can be compressed to only about 3.5 bits per weight (bpw)  without significant quality loss. We choose to use 3.7 bpw in production as it already meets the memory requirements.Interactive model analysisWe use an interactive model latency and power analysis tool, Talaria, to better guide the bit rate selection for each operation.More discussionsThe usage of quantized model and LoRA adapters look conceptually similar to QLoRA.
While QLoRA was designed to save computational resources during fine-tuning, our focus is on the ability to switch between different LoRA adapters to efficiently support high performance across various specific use cases.
Before feature-specific finetuning, we first train accuracy-recovery adapters on the same pretraining and post-training data, which is critical to preserve the model quality.
The accuracy-recovery framework can be combined with different quantization techniques, like GPTQand AWQ, since it does not depend directly on the quantization method itself.
The feature adapters described inSection 5are initialized from these accuracy-recovery adapters.5.3Case study: summarizationWe use the AFM-on-device model to power summarization features. We worked with our design teams to create specifications for summaries of Emails, Messages, and Notifications.While AFM-on-device is good at general summarization, we find it difficult to elicit summaries that strictly conform to the specification. Therefore, we fine tune a LoRA adapter on top of the quantized AFM-on-device for summarization. The adapter is initialized from the accuracy-recovery adapter as described inSection 5.2.
We use a data mixture consisting of input payloads covering Emails, Messages, and Notifications. These payloads include public dataset, vendor data, and internally generated and submitted examples. All the data have been approved to use for production. Vendor data and internally generated data have been anonymized to remove the user information.
Given these payloads, we generated synthetic summaries using AFM-server according to product’s requirements. These payloads and summaries are used for training.Synthetic summariesWe use AFM-server to generate synthetic summaries. We apply a series of rule-based filters followed by model based filters. Rule-based filters are based on heuristics such as length constraints, formatting constraints, points of view, voice, etc. Model-based filters are used to screen more challenging problems such as entailment.
Our synthetic data pipeline allows us to efficiently generate a large amount of training data and filter it out by an order of magnitude to retain high-quality examples for fine tuning.Prompt injectionWe find that AFM-on-device is prone to following instructions or answering questions that are present in the input content instead of summarizing it. To mitigate this issue, we identify a large set of examples with such content using heuristics, use AFM-server to generate summaries, as it does not exhibit similar behavior, and add this synthetic dataset to the fine tuning data mixture.
body
Apple Intelligence Foundation Language Models
5Powering Apple Intelligence features
2407.21075
6EvaluationWe evaluate the AFM models on pre-training (Section 6.1) , post-training (Section 6.2) , and most importantly, feature-specific (Section 6.3)  benchmarks.6.1Pre-training evaluationIn this section we present common few-shot pre-training evaluation metrics.
While these benchmarks are useful for tracking our progress on pre-training, we found that human evaluations on the post-trained models (Section 6.2)  and feature adapters (Section 6.3)  are more closely correlated to end-to-end user experience.We evaluate AFM pre-trained models with common open-sourced evaluation harnesses and benchmarks.Table 2presents the results of AFM-on-device and AFM-server on HELM MMLU v1.5.0, which tests 5-shot multiple-choice question answering across 57 subjects.
Also seeTable 3andTable 4for the results of AFM-server on a subset of the HuggingFace OpenLLM leaderboard V1and the HELM-Lite v1.5.0 benchmark suite, respectively.
These benchmarks show that the AFM pretrained models possess strong language and reasoning capabilities and provide a solid foundation for post-training and feature fine-tuning.AFM-on-deviceAFM-serverMMLU (5 shot) 61.475.4Table 2:HELM MMLU-5sv1.5.0 evaluation results.AFM-serverMMLU (5-shot) 75.3GSM8K (5-shot) 72.4ARC-c (25-shot) 69.7HellaSwag (10-shot) 86.9Winogrande (5-shot) 79.2Table 3:A subset of Open LLM LeaderboardV1 evaluation results.AFM-serverNarrative QA77.5Natural Questions (open) 73.8Natural Questions (closed) 43.1Openbook QA89.6MMLU67.2MATH-CoT55.4GSM8K72.3LegalBench67.9MedQA64.4WMT 201418.6Table 4:HELM-Lite v1.5.0pre-training evaluation results. N.B. Many benchmarks (e.g. MMLU)  differ significantly from commonly used settings.6.2Post-training evaluationWe evaluate post-training models on comprehensive benchmarks and
compare AFM models with various open-source models, as well as GPT-3.5 and GPT-4.111We compared against the following model versions: gpt-3.5-turbo-0125, gpt-4-0125-preview, Gemini-1.5-Pro-0514, DBRX Instruct, Phi-3-mini-4k-instruct, LLaMA 3 8B Instruct, LLaMA 3 70B Instruct, Mistral-7B-Instruct-v0.2, Mixtral-8x22B-Instruct-v0.1, Gemma-1.1-2B, and Gemma-1.1-7B.All results reported in this section are obtained using AFM-on-device and AFM-server base models without any adapter, inbfloat16precision. In this section, we first present human evaluation results that measure the AFMs general capabilities, and then present results for several specific capabilities and domains.6.2.1Human evaluationHuman evaluation simulates practical use cases and user feedback, and so often serves as the gold standard for language model evaluation.
Consequently, we conduct extensive human evaluations both while developing the model and to evaluate its final form.
We collect sets of evaluation prompts to test the model on different aspects, including both general capabilities and safety.
For each prompt, two model responses are presented to human raters anonymously for side-by-side comparisons.
Depending on the nature of the evaluation,
a detailed guideline containing grading principles and examples of single-response ratings and side-by-side preference ratings is provided to human raters to ensure consistent grading standards and evaluation quality.
Each pair of model responses is graded by multiple graders and their ratings are aggregated for final results.
Overall, we find human evaluation to align better with user experience and provide a better evaluation signal than some academic benchmarks that use LLMs as graders. In this section, we present the results for human evaluation on general capabilities, and the safety evaluation results are provided inSection 7.6.We collect a comprehensive set ofprompts to evaluate the general model capabilities.
These prompts are diverse across different difficulty levels and cover major categories including:
analytical reasoning,
brainstorming,
chatbot,
classification,
closed question answering,
coding,
extraction,
mathematical reasoning,
open question answering,
rewriting,
safety,
summarization,
and writing.
To prevent overfitting, when preparing training data, we conduct decontamination against our evaluation prompts.Figure 3:Side-by-side evaluation of AFM-on-device and AFM-server against comparable models. We find that our models are often preferred over competitor models by human graders.InFigure 3, we compare AFM with both open-source models (Phi-3, Gemma-1.1, Llama-3, Mistral, DBRX-Instruct)  and commercial models (GPT-3.5, and GPT-4) .
AFM models are preferred by human graders over competitor models.
In particular, AFM-on-device obtains a win rate of 47.7% when compared to Phi-3-mini despite being 25% smaller in model sizes,
and even outperforms open-source strong baselines Gemma-7B and Mistral-7B that are more than twice larger in the number of parameters.
When compared to closed-source models, AFM-server achieves competitive performance, scoring a win rate of more thanand a tie rate of 27.4% against GPT-3.5.6.2.2Instruction followingInstruction following (IF)  is the core capability we desire of language models,
as real-world prompts are often sophisticated and contain complex instructions.
We emphasize the importance of instruction following in both our RLHF data collection and human evaluation.
In this subsection, we evaluate our models’ IF skills using automated benchmarks.InFigure 4we evaluate AFM-on-device and AFM-server on the public IFEval benchmark, respectively.
This benchmark measures a language model’s capability to generate responses that precisely follow instructions in the prompt. The instructions in this benchmark typically include requirements on the response length, format, content, etc.
We find AFM-on-device and AFM-server to achieve superior performance on both instruction-level and prompt-level accuracy.
In addition,
we also benchmark AFM models on the AlpacaEval 2.0 LC benchmarkto measure general instruction-following capability, and results suggest that our models are highly competitive.Figure 4:Instruction-following capability (measured with IFEval)  for AFM models and relevant comparison models (higher is better) . The AlpacaEval 2.0 LC results for Mistral 7B, Llama3 8B, Llama3 70B, DBRX-Instruct, and Mixtral 8x22B are obtained fromthe AlpacaEval leaderboard. The Arena Hard results for comparison models are fromthe Arena-Hard-Auto leaderboard. All other results are from our own evaluations.6.2.3Tool useIn tool use applications,
given a user request and a list of potential tools with descriptions,
the model can choose to issue tool calls by providing a structured output specifying the name and parameter values of the tools to call. We expect the tool descriptions to follow the OpenAPI specification.444https://github.com/OAI/OpenAPI-SpecificationWe evaluate on the public Berkeley Function Calling Leaderboard benchmarksvia native support of function calling, using the AST metrics.As shown inFigure 5, AFM-server achieves the best overall accuracy, outperforming Gemini-1.5-Pro-Preview-0514 and GPT-4.Figure 5:Berkeley Function Calling Leaderboard Benchmark
evaluation results on Function Calling API, along-side relevant sampled comparisons.
Numbers were collected fromthe Gorilla leaderboard.6.2.4WritingWriting is one of the most critical abilities for large language models to have, as it empowers various downstream use cases such as changing-of-tone, rewriting, and summarization.
However, assessing writing quality is a non-trivial task, and not well-covered in the above public benchmarks.We evaluate AFM’s writing ability on our internal summarization and composition benchmarks,
consisting of a variety of writing instructions.
Following LLM-as-a-judge, we design a grading instruction
for each summarization and composition task,
and prompt GPT-4 Turbo to assign a score from 1 to 10 for model responses.555Due to the choice of using GPT-4 as judge, the score of GPT-4 Turbo can be overestimated.We note that there are certain limitations and biases associated with using an LLM as a grader, such as length bias.We compare AFM with a few of the most outstanding models,
along with smaller-scale open-source models.
As shown inFigure 6,
AFM-on-device can achieve comparable or superior performance when compared to Gemma-7B and Mistral-7B.
AFM-server significantly outperforms DBRX-Instruct and GPT3.5 and is comparable to GPT4.Figure 6:Writing ability on internal summarization and composition benchmarks (higher is better)  for AFM-on-device and AFM-server alongside
relevant sampled comparisons. We find that our models perform better or similar to related models.6.2.5MathInFigure 7, we compare post-training AFM’s performance on math benchmarks including GSM8Kand MATH. We use-shot chain-of-thought (CoT) prompt for GSM8K and-shot CoT promptfor MATH. We conduct all evaluations using an internal automated evaluation pipeline. We see that the AFM-on-device significantly outperforms Mistral-7B and Gemma-7B, even at less than half of their sizes.Figure 7:Math benchmarks for AFM-on-device and AFM-server alongside relevant sampled comparisons. GSM8K is 8-shot and MATH is 4-shot.
All results are collected with an internal automated evaluation pipeline.6.3Summarization feature evaluationThe product team specifications for summarizing Emails, Messages, and Notifications necessitated a tailor-made set of guidelines, metrics, and specialized graders to evaluate summarization quality against various open-source, licensed, and proprietary datasets.Datasets.We sampled abundant payloads carefully for each use case.
These evaluation datasets emphasize a diverse set of inputs which our product features are likely to face in production, and include a stratified mixture of single and stacked documents of varying content types and lengths. We developed a pipeline to build evaluation datasets that simulate real user inputs.Graders.We enlisted a pool of highly-trained, full-time, Apple-employed human graders with specialized writing and comprehension skills to evaluate summarization quality. To qualify for grading projects, each grader must pass a series of eligibility and training steps, which include a required bachelor’s degree in a writing-related discipline, customized training sessions, and consistently high performance against internal grading quality benchmarks.Grading guidelines.During the evaluation task, graders are presented with a specification for the summary, the original input content, and the output summary. Graders assess the summary on each the following sub-dimensions of quality using 3 point scales (“good”, “neutral”, or “poor” ) :Composition: Evaluates the overall readability of the summary considering grammar, punctuation, spelling, and brevity.Comprehensiveness: Evaluates how comprehensive the summary is in capturing the essential points or calling out any actions/conclusions for the user.Groundedness: Evaluates how grounded the summary is with respect to the original payload. Summaries that are not completely grounded may contain details that are exaggerated, inferred, inaccurate, or hallucinated.Following instructions: Evaluates whether the summary meets specific style and formatting requirements. Requirements are tailored to each feature and reflect specific product and design expectations.Harmfulness: Evaluates whether the summary contains content that is harmful or unsafe according to Apple’s safety taxonomy.A summary is classified as “poor” ifanyof the sub-dimensions are “poor” according to predefined product specifications. Likewise a summary is “good” only ifallsub-dimensions are good. These classifications are used to compute “Good/Poor Result Ratio” metrics defined as the percentage of good/poor summaries out of all summaries.Results.We ask human graders to evaluate the summarization quality of the AFM-on-device adapter, Phi-3-mini, Llama-3-8B, and Gemma-7B.Figure 8shows that AFM-on-device-adapter overall outperforms the other models.Figure 8:Ratio of “good” and “poor” responses for three summarization use cases relative to all responses. Summaries are classified as “good”, “neutral”, or “poor” along five dimensions. A result is classified as “good” ifallof the dimensions are good (higher is better) . A result is classified as “poor” ifanyof the dimensions are poor (lower is better) . Overall, our AFM-on-device adapter generates better summaries than comparable models.
body
Apple Intelligence Foundation Language Models
6Evaluation
2407.21075
7Responsible AI7.1OverviewApple Intelligence is developed responsibly and designed with care to empower our users, represent them authentically, and protect their privacy. Of primary importance to our Responsible AI approach is that we are ultimately delivering intelligent, well-defined tools that address specific user needs. Having a clear definition of what a feature is intended to do allows us to better identify any potential safety gaps.We have developed a safety taxonomy in order to be comprehensive and consistent in the design and evaluation of our generative AI-powered features. This taxonomy builds and extends Apple’s extensive experience in using artificial intelligence and machine learning to deliver helpful features to users around the world, and is updated regularly as we develop and test features. Currently, it consists of 12 primary categories comprised of 51 subcategories, including “Hate Speech, Stereotypes, and Slurs”, “Discrimination,
Marginalization, and Exclusion”, “Illegal Activities”, “Adult Sexual Material”, and “Graphic Violence.”The taxonomy serves as a structured way to consider potential issues and risks relative to each specific feature. As new or additional risks are identified, we develop and revise the associated policies that are contextualized to each individual feature, taking into account the specific needs that it serves, the content it produces, and the appropriate mitigations.
They are developed with extensive internal and external input from academics, AI ethicists, trust and safety, and legal experts to better identify and understand the relevant risks, the potential severity of such risks, and the potential disparate impact these risks may have on certain groups.
These policies guide our work in data collection, human annotation, model training, guardrails development, evaluation, and red teaming.Particularly, the taxonomy is not itself the sole determinant of our policy. For example, content that may fall within the safety taxonomy is not necessarily always blocked, as doing so unilaterally may be in conflict with other aspects of Apple’s Responsible AI development principles, such as “respecting how our users choose to use these tools to accomplish their goals.” Thus, features that operate as tools may be more permissive in the kinds of content they operate over and produce in order to effectively address the user’s intent. On the other hand, features that may generate content beyond a user’s specified intent may need to be more constrained. Regardless, we strive for some categories of harm to always be treated with special care (such as any content that relates to self harm)  while other categories will always be blocked (such as illegal content) .In addition, our Responsible AI principles are built into every stage of Apple Foundation Models and Apple Intelligence as well as the safety taxonomy, which helps us evaluate risks and formulate policies feature by feature. We include safety-oriented data as part of our fine-tuning of specific adapters tailored by use case. Furthermore, at the time of inference, we also run guardrail modelsas pre- and post-processing steps to evaluate potential harm at both the input and output level. Finally, we have mechanisms in place to continuously and proactively improve our AI tools with the help of ongoing user feedback.7.2Pre-TrainingAt the pre-training stage, we take several steps to ensure that the values as outlined above are upheld. We follow a strict data policy ensuring that no Apple user data is included, as well as conduct rigorous legal review for each component in the training corpus. Further, we perform safety filtering to reduce potentially harmful content, including NSFW content, profanity, spam, and PII or financial data.Because pre-training is a step which is shared among various downstream features, our safety mitigations aim to retain general capabilities that allow us to iterate on the taxonomy and policy at a per-feature level, without hurting the helpfulness of these downstream models. We take learnings from prior work to avoid overly aggressive filtering at the pre-training stage, which has potential benefits in safety alignment. Intuitively, the pre-trained model should be aware of content that downstream features and policies may require it to handle – in some cases with care, or in other cases operating over such content directly.7.3Post-TrainingIn the post-training phase, we aim to instill a baseline level of alignment with our Responsible AI principles to avoid necessitating the full complexities of post-training (such as RLHF)  in each downstream model that builds on top of the foundation model. In doing so, there are two key considerations:1.We must ensure our models produce output that is helpful to users, while minimizing potential harm.2.We must contextualize our safety taxonomy and policies on a feature by feature basis to deliver the best possible user experience.To balance helpfulness and harmlessness trade-off, our solution is to treat safety alignment as one of the many core post-training tasks that are evaluated and iterated on in tandem, instead of as a separate stage of training. Specifically, we include adversarial data into our SFT and RLHF training corpora that is curated according to our policy and values by partnering closely with trusted vendors. We also incorporate safety tasks and benchmarks into the automatic and human evaluations used during model development.In total, over 10% of the training data are adversarial or related to safety or sensitive topics, including single and multi-turn safety category annotations, pairwise and overall preference ratings, and annotator rewrites. This data is either used directly or as seed data for synthetic data generation, as described inSection 4.1.2.We do additional work to achieve appropriate safety behavior for each feature beyond baseline alignment. A primary way that we do this is by collecting safety-specific training data and including it when fine-tuning adapters. For instance,
in fine-tuning our summarization adapter we sought to improve aspects such as, improving robustness against malicious questions included within the content to be summarized, and reducing the likelihood that summaries would inadvertently amplify harmful or sensitive content to be summarized.7.4Guarding against malicious codeCode generation requires special care. Our code benchmarks involve actually executing the generated code to determine both syntactic and semantic correctness. Thus, responsible training of code models involves treating all generated code as unsafe by default – all code is always executed in a fully locked down environment with no access to the internet or any internal or external services. Specifically, the locked down environment is managed with FireCracker, with a FireCracker jailer at the cluster level.7.5Red teamingRed teaming attempts to elicit safety policy violating responses from models, or harmful responses for which no policy yet exists. These results inform both policy development as well as the focus and content of safety evaluation datasets. These in turn can influence design, engineering, and shipping readiness decisions.Red teaming is a fundamentally creative endeavor that requires red teamers to employ combinations of attack vectors to probe known model vulnerabilities, and try to discover new ones. Attack vectors used when engaging with language models include jailbreaks/prompt injections, persuasive techniques, and linguistic features known to cause model misbehavior (e.g. slang, code-switching, emojis, typos) .We employ both manual and automatic red-teamingto elicit potentially unknown failure modes of the aligned models. More recent workssuggest that automated processes can potentially generate even more diverse prompts than humans, previously seen as the “gold” standard for data collection. These automated processes can include using the language models themselves to identify gaps, some of which may be unintuitive or even surprising. Such examples can be used directly as synthetic training or evaluation data and to inform future data collection efforts.A basic human red teaming task schema is as follows: a red teamer is assigned a safety taxonomy category and attack vector(s) . They author an input to the model, using that attack vector, that is intended to elicit a response containing content from that category. If the response does not contain the target content, the red teamer can engage in a fixed number of conversational turns, after which they provide a final harmfulness rating of the model output and list the taxonomy categor(ies)  in it, if any. To ensure annotation quality, red teamers also provide an overall confidence score for their ratings.In addition to red teaming at the base model level, we also red team specific features. Red teaming projects at the feature level use feature-specific guidelines with attack vectors informed by the feature’s safety policy and engineering concerns. These projects can provide in-depth probing of known risks for that particular feature and also adversarially probe for unknown vulnerabilities.Our red teaming projects are run using internal and external crowds. To ensure responsible data collection, due to the sensitive nature of red teaming we: 1)  make red teaming completely voluntary; 2)  impose a strict time limit on how much each red teamer spends on the tasks per week; 3)  provide health and well-being resources available around the clock; and 4)  maintain an open line of communication with internal red teamers via weekly office hours and a Slack channel for them to communicate any concerns that arise.7.6EvaluationAs mentioned in previous sections, safety is one of the many axes iterated on during foundation model development, and therefore undergoes the same automatic and human evaluation cycles during post-training.Safety evaluation setTo reduce noise, cost, and turn-around time during human evaluations, we must ensure that our safety evaluation sets are clean, yet challenging and comprehensive. To that end, we filter out “easy" prompts which consistently yield low harmfulness responses across different versions of the model, and employ an embedding-based analysis to improve our evaluation prompt set coverage. Overall, we curate a set of over a thousand adversarial prompts to test AFM’s performance on harmful content, sensitive topics, and factuality according to our safety policy.Figure 9:Fraction of violating responses for harmful content, sensitive topics, and factuality (lower is better) . Our models are robust when faced with adversarial prompts.Figure 10:Fraction of preferred responses in side-by-side evaluation of Apple’s foundation model against comparable models on safety prompts. Human graders found our responses safer and more helpful.Safety evaluation resultsFigure 9summarizes the violation rates of different models evaluated by human graders on this safety evaluation set. Lower is better.
Both AFM-on-device and AFM-server are robust to adversarial prompts,
achieving violation rates significantly lower than open-source and commercial models.
In addition, we report side-by-side human preference on our safety evaluation prompts inFigure 10. AFM models are preferred by human graders as safe and helpful responses over competitor models.
body
Apple Intelligence Foundation Language Models
7Responsible AI
2407.21075
8ConclusionIn this report we introduced the foundation language models that power Apple Intelligence features, AFM-on-device and AFM-server.
The models are designed to be fast and run efficiently on iPhone, iPad, and Mac as well as on Apple silicon servers via Private Cloud Compute.
They are trained to be highly capable in tasks like language understanding, instruction following, reasoning, writing, and tool use.
We have developed an innovative model architecture to specialize these models for our users’ most common tasks.
On top of the foundation models, feature-specific adapters are fine-tuned to provide high-quality user experiences such as summarization of emails, messages, and notifications.
Our models have been created with the purpose of helping users do everyday activities across their Apple products, grounded in Apple’s core values, and rooted in our Responsible AI principles at every stage.
These foundation models are at the heart of Apple Intelligence, the personal intelligence system built by Apple to continue empowering our users and enriching their lives.
body
Apple Intelligence Foundation Language Models
8Conclusion
2407.21075
ContributorsWithin each section, contributors are listed in alphabetical order by first name.Foundation ModelsAndy Narayanan,
Aonan Zhang,
Bowen Zhang,
Chen Chen,
Chong Wang (inference efficiency lead) ,
Chung-Cheng Chiu,
David Qiu,
Deepak Gopinath,
Dian Ang Yap,
Dong Yin,
Feng Nan,
Floris Weers,
Guoli Yin,
Haoshuo Huang,
Jianyu Wang,
Jiarui Lu,
John Peebles,
Ke Ye,
Mark Lee,
Nan Du,
Qibin Chen,
Quentin Keunebroek,
Ruoming Pang (overall lead) ,
Sam Wiseman,
Syd Evans,
Tao Lei,
Tom Gunter (pre-train lead) ,
Vivek Rathod,
Xiang Kong,
Xianzhi Du,
Yanghao Li,
Yongqiang Wang,
Yuan Gao,
Zaid Ahmed,
Zhaoyang Xu,
Zhiyun Lu,
Zirui Wang (post-train lead) Data, Evaluation, and Responsible AIAl Rashid,
Albin Madappally Jose,
Alec Doane,
Alfredo Bencomo,
Allison Vanderby,
Andrew Hansen,
Ankur Jain,
Anupama Mann Anupama,
Areeba Kamal,
Bugu Wu,
Carolina Brum,
Charlie Maalouf,
Chinguun Erdenebileg,
Chris Dulhanty,
Dominik Moritz,
Doug Kang,
Eduardo Jimenez,
Evan Ladd,
Fangping Shi,
Felix Bai,
Frank Chu,
Fred Hohman,
Hadas Kotek,
Hannah Gillis Coleman,
Jane Li,
Jeffrey Bigham,
Jeffery Cao,
Jeff Lai,
Jessica Cheung,
Jiulong Shan,
Joe Zhou,
John Li,
Jun Qin,
Karanjeet Singh,
Karla Vega,
Ke Ye,
Kelvin Zou,
Laura Heckman,
Lauren Gardiner,
Margit Bowler,
Mark Lee,
Maria Cordell,
Meng Cao,
Nicole Hay,
Nilesh Shahdadpuri,
Otto Godwin,
Pranay Dighe,
Pushyami Rachapudi,
Ramsey Tantawi,
Roman Frigg,
Sam Davarnia,
Sanskruti Shah,
Saptarshi Guha,
Sasha Sirovica,
Shen Ma,
Shuang Ma,
Simon Wang,
Sulgi Kim,
Suma Jayaram,
Vaishaal Shankar,
Varsha Paidi,
Vivek Kumar,
Xiang Kong,
Xin Wang,
Xin Zheng,
Walker Cheng,
Yael Shrager,
Yang Ye,
Yasu Tanaka,
Yihao Guo,
Yunsong Meng,
Zhao Tang Luo,
Zhi Ouyang,
Zhiyun LuAdapters, Optimizations, and SummarizationAlp Aygar,
Alvin Wan,
Andrew Walkingshaw,
Andy Narayanan,
Antonie Lin,
Arsalan Farooq,
Brent Ramerth,
Chong Wang,
Colorado Reed,
Chris Bartels,
Chris Chaney,
David Riazati,
Eric Liang Yang,
Erin Feldman,
Gabriel Hochstrasser,
Guillaume Seguin,
Guoli Yin,
Irina Belousova,
Jianyu Wang,
Joris Pelemans,
Karen Yang,
Keivan Alizadeh Vahid,
Liangliang Cao,
Mahyar Najibi ,
Marco Zuliani,
Max Horton,
Minsik Cho,
Nikhil Bhendawade,
Patrick Dong,
Piotr Maj,
Pulkit Agrawal,
Qi Shan,
Qibin Chen,
Qichen Fu,
Regan Poston,
Sam Xu,
Shuangning Liu,
Sushma Rao,
Tashweena Heeramun,
Thomas Merth,
Uday Rayala,
Victor Cui,
Vivek Rangarajan Sridhar,
Vivek Rathod,
Wencong Zhang,
Wenqi Zhang,
Wentao Wu,
Xiang Kong,
Xingyu Zhou,
Xinwen Liu,
Yang Zhao,
Yin Xia,
Zhile Ren,
Zhongzheng Ren
conclusion
Apple Intelligence Foundation Language Models
Section 9