<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

<head>  <title>[2408.16673] Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
  <link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
  <link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20240822" />
  <link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
  <script language="javascript" src="/static/browse/0.3.4/js/accordion.js" /></script>
  
    <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/slider.css?v=1.14" />
    <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="/static/browse/0.3.4/js/donate.js?v=1.11"></script><link rel="canonical" href="/abs/2408.16673"/><meta property="og:type" content="website" />
<meta property="og:site_name" content="arXiv.org" />
<meta property="og:title" content="Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity" />
<meta property="og:url" content="https://arxiv.org/abs/2408.16673v1" />
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="700" />
<meta property="og:image:alt" content="arXiv logo"/>
<meta property="og:description" content="Large language models rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks. Cross Entropy (CE) loss is the de facto choice in SFT, but it often leads to overfitting and limited output diversity due to its aggressive updates to the data distribution. This paper aim to address these issues by introducing the maximum entropy principle, which favors models with flatter distributions that still effectively capture the data. Specifically, we develop a new distribution matching method called GEM, which solves reverse Kullback-Leibler divergence minimization with an entropy regularizer.
  For the SFT of Llama-3-8B models, GEM outperforms CE in several aspects. First, when applied to the UltraFeedback dataset to develop general instruction-following abilities, GEM exhibits reduced overfitting, evidenced by lower perplexity and better performance on the IFEval benchmark. Furthermore, GEM enhances output diversity, leading to performance gains of up to 7 points on math reasoning and code generation tasks using best-of-n sampling, even without domain-specific data. Second, when fine-tuning with domain-specific datasets for math reasoning and code generation, GEM also shows less overfitting and improvements of up to 10 points compared with CE."/>
<meta name="twitter:site" content="@arxiv"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Entropic Distribution Matching in Supervised Fine-tuning of LLMs:..."/>
<meta name="twitter:description" content="Large language models rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks. Cross Entropy (CE) loss is the de facto choice in SFT, but it often leads to overfitting and limited..."/>
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
<meta name="twitter:image:alt" content="arXiv logo"/>
  <link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/>  <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script>  <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
  <script src="/static/browse/0.3.4/js/toggle-labs.js?20230622" type="text/javascript"></script>
  <script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity" /><meta name="citation_author" content="Li, Ziniu" /><meta name="citation_author" content="Chen, Congliang" /><meta name="citation_author" content="Xu, Tian" /><meta name="citation_author" content="Qin, Zeyu" /><meta name="citation_author" content="Xiao, Jiancong" /><meta name="citation_author" content="Sun, Ruoyu" /><meta name="citation_author" content="Luo, Zhi-Quan" /><meta name="citation_date" content="2024/08/29" /><meta name="citation_online_date" content="2024/08/29" /><meta name="citation_pdf_url" content="http://arxiv.org/pdf/2408.16673" /><meta name="citation_arxiv_id" content="2408.16673" /><meta name="citation_abstract" content="Large language models rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks. Cross Entropy (CE) loss is the de facto choice in SFT, but it often leads to overfitting and limited output diversity due to its aggressive updates to the data distribution. This paper aim to address these issues by introducing the maximum entropy principle, which favors models with flatter distributions that still effectively capture the data. Specifically, we develop a new distribution matching method called GEM, which solves reverse Kullback-Leibler divergence minimization with an entropy regularizer. For the SFT of Llama-3-8B models, GEM outperforms CE in several aspects. First, when applied to the UltraFeedback dataset to develop general instruction-following abilities, GEM exhibits reduced overfitting, evidenced by lower perplexity and better performance on the IFEval benchmark. Furthermore, GEM enhances output diversity, leading to performance gains of up to 7 points on math reasoning and code generation tasks using best-of-n sampling, even without domain-specific data. Second, when fine-tuning with domain-specific datasets for math reasoning and code generation, GEM also shows less overfitting and improvements of up to 10 points compared with CE." />
</head>

<body  class="with-cu-identity">
    <aside class="slider-wrapper bps-banner forum green">
    <a class="close-slider bps-banner" href="#"><img src="/static/browse/0.3.4/images/icons/close-slider.png" alt="close this message"></a>
    <div class="columns">
      <img role="presentation" class="bps-banner-image" src="/static/browse/0.3.4/images/icons/smileybones-infinity-large.png" alt="arXiv Accessibility Forum 2024">
      <div class="copy-donation bps-banner">
        <h2>The countdown begins!</h2>
        <p>The <a href="https://accessibility2024.arxiv.org/" target="_blank">Accessibility Forum</a> starts next week! <a href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y" target="_blank">Sign up</a> and join us in September.</p>
      </div>
      <div class="amount-donation bps-banner">
        <div class="donate-cta"><a class="banner_link banner-btn-grad" target="_blank" href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y"><b>Sign Up</b></a></div>
      </div>
    </div>
  </aside>
  <div class="flex-wrap-footer">
    <header>
      <a href="#content" class="is-sr-only">Skip to main content</a>
      <!-- start desktop header -->
      <div class="columns is-vcentered is-hidden-mobile" id="cu-identity">
        <div class="column" id="cu-logo">
          <a href="https://www.cornell.edu/"><img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University" /></a>
        </div><div class="column banner-minimal forum">
            <!-- /from July 7 at 1:00 AM to Sept 2 at 1:00 AM --><p>arXiv's Accessibility Forum starts next month!</p>
            <a href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y" target="_blank">Sign Up</a>

            <!-- /from Sept 2 at 1:00 AM to Sept 13 at 11:30PM --></div><div class="column" id="support-ack">
          <span id="support-ack-url">We gratefully acknowledge support from the Simons Foundation, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and all contributors.</span>
          <a href="https://info.arxiv.org/about/donate.html" class="btn-header-donate">Donate</a>
        </div>
      </div>

      <div id="header" class="is-hidden-mobile">
<a aria-hidden="true" href="{url_path('ignore_me')}"></a>
  <div class="header-breadcrumbs is-hidden-mobile">
    <a href="/"><img src="/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg" alt="arxiv logo" style="height:40px;"/></a> <span>&gt;</span> <a href="/list/cs/recent">cs</a> <span>&gt;</span> arXiv:2408.16673
  </div>
          <div class="search-block level-right">
    <form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
      <div class="field has-addons">
        <div class="control">
          <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
          <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
        </div>
        <div class="control">
          <div class="select is-small">
            <select name="searchtype" aria-label="Field to search">
              <option value="all" selected="selected">All fields</option>
              <option value="title">Title</option>
              <option value="author">Author</option>
              <option value="abstract">Abstract</option>
              <option value="comments">Comments</option>
              <option value="journal_ref">Journal reference</option>
              <option value="acm_class">ACM classification</option>
              <option value="msc_class">MSC classification</option>
              <option value="report_num">Report number</option>
              <option value="paper_id">arXiv identifier</option>
              <option value="doi">DOI</option>
              <option value="orcid">ORCID</option>
              <option value="author_id">arXiv author ID</option>
              <option value="help">Help pages</option>
              <option value="full_text">Full text</option>
            </select>
          </div>
        </div>
        <input type="hidden" name="source" value="header">
        <button class="button is-small is-cul-darker">Search</button>
      </div>
    </form>
  </div>
     </div><!-- /end desktop header -->

      <div class="mobile-header">
        <div class="columns is-mobile">
          <div class="column logo-arxiv"><a href="https://arxiv.org/"><img src="/static/browse/0.3.4/images/arxiv-logomark-small-white.svg" alt="arXiv logo" style="height:60px;" /></a></div>
          <div class="column logo-cornell"><a href="https://www.cornell.edu/">
            <picture>
              <source media="(min-width: 501px)"
                srcset="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg  400w"
                sizes="400w" />
              <source srcset="/static/browse/0.3.4/images/icons/cu/cornell_seal_simple_black.svg 2x" />
              <img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University Logo" />
            </picture>
          </a></div>
          <div class="column nav" id="toggle-container" role="menubar">
            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-white"><title>open search</title><path d="M505 442.7L405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9 0 208 0S0 93.1 0 208s93.1 208 208 208c48.3 0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9 0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7 0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7 0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
                <div class="field has-addons">
                  <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
                  <input type="hidden" name="source" value="header">
                  <input type="hidden" name="searchtype" value="all">
                  <button class="button">GO</button>
                </div>
              </form>
            </div>

            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-white" role="menu"><title>open navigation menu</title><path d="M16 132h416c8.837 0 16-7.163 16-16V76c0-8.837-7.163-16-16-16H16C7.163 60 0 67.163 0 76v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16z"/ ></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <nav class="mobile-menu" aria-labelledby="mobilemenulabel">
                <h2 id="mobilemenulabel">quick links</h2>
                <ul>
                    <li><a href="https://arxiv.org/login">Login</a></li>
                    <li><a href="https://info.arxiv.org/help">Help Pages</a></li>
                    <li><a href="https://info.arxiv.org/about">About</a></li>
                </ul>
              </nav>
            </div>
          </div>
        </div>
      </div><!-- /end mobile-header -->
    </header>

    <main>
      <div id="content">
<!--
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
    <rdf:Description
        rdf:about="/abs/2408.16673"
        dc:identifier="/abs/2408.16673"
        dc:title="Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity"
        trackback:ping="/trackback/2408.16673" />
    </rdf:RDF>
--><div id="abs-outer">

  <div class="leftcolumn">
    <div class="subheader">
      <h1>Computer Science > Machine Learning</h1>
    </div>

    <div class="header-breadcrumbs-mobile">
      <strong>arXiv:2408.16673</strong> (cs)
    </div>
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
<div id="content-inner">
  <div id="abs">
    <div class="dateline">
  [Submitted on 29 Aug 2024]</div>
    <h1 class="title mathjax"><span class="descriptor">Title:</span>Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity</h1>
    <div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Li,+Z">Ziniu Li</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Chen,+C">Congliang Chen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Xu,+T">Tian Xu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Qin,+Z">Zeyu Qin</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Xiao,+J">Jiancong Xiao</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sun,+R">Ruoyu Sun</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Luo,+Z">Zhi-Quan Luo</a></div>            <div id="download-button-info" hidden>View a PDF of the paper titled Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity, by Ziniu Li and 6 other authors</div>
    <a class="mobile-submission-download" href="/pdf/2408.16673">View PDF</a>
    <a class="mobile-submission-download" href="https://arxiv.org/html/2408.16673v1">HTML (experimental)</a>



    <blockquote class="abstract mathjax">
            <span class="descriptor">Abstract:</span>Large language models rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks. Cross Entropy (CE) loss is the de facto choice in SFT, but it often leads to overfitting and limited output diversity due to its aggressive updates to the data distribution. This paper aim to address these issues by introducing the maximum entropy principle, which favors models with flatter distributions that still effectively capture the data. Specifically, we develop a new distribution matching method called GEM, which solves reverse Kullback-Leibler divergence minimization with an entropy regularizer.
<br>For the SFT of Llama-3-8B models, GEM outperforms CE in several aspects. First, when applied to the UltraFeedback dataset to develop general instruction-following abilities, GEM exhibits reduced overfitting, evidenced by lower perplexity and better performance on the IFEval benchmark. Furthermore, GEM enhances output diversity, leading to performance gains of up to 7 points on math reasoning and code generation tasks using best-of-n sampling, even without domain-specific data. Second, when fine-tuning with domain-specific datasets for math reasoning and code generation, GEM also shows less overfitting and improvements of up to 10 points compared with CE.
    </blockquote>

    <!--CONTEXT-->
    <div class="metatable">
      <table summary="Additional metadata"><tr>
          <td class="tablecell label">Subjects:</td>
          <td class="tablecell subjects">
            <span class="primary-subject">Machine Learning (cs.LG)</span>; Artificial Intelligence (cs.AI)</td>
        </tr><tr>
          <td class="tablecell label">Cite as:</td>
          <td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2408.16673">arXiv:2408.16673</a> [cs.LG]</span></td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxividv">(or <span class="arxivid">
              <a href="https://arxiv.org/abs/2408.16673v1">arXiv:2408.16673v1</a> [cs.LG]</span> for this version)
          </td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxivdoi">              <a href="https://doi.org/10.48550/arXiv.2408.16673" id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2408.16673</a><div class="button-and-tooltip">
              <button class="more-info" aria-describedby="more-info-desc-1">
                <svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
                <span class="visually-hidden">Focus to learn more</span>
              </button>
              <!-- tooltip description -->
              <div role="tooltip" id="more-info-desc-1">
                <span class="left-corner"></span>                  arXiv-issued DOI via DataCite (pending registration)</div>
            </div>
          </td>
        </tr></table>
    </div>
  </div>
</div>
    <div class="submission-history">
      <h2>Submission history</h2> From: Ziniu Li [<a href="/show-email/00e36ab2/2408.16673">view email</a>]      <br/>    <strong>[v1]</strong>
        Thu, 29 Aug 2024 16:21:00 UTC (549 KB)<br/>
</div>
  </div>
  <!--end leftcolumn-->
<div class="extra-services">    <div class="full-text">
      <a name="other"></a>
      <span class="descriptor">Full-text links:</span>
      <h2>Access Paper:</h2>
      <ul>
  <div id="download-button-info" hidden>
View a PDF of the paper titled Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity, by Ziniu Li and 6 other authors</div><li><a href="/pdf/2408.16673" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2408.16673v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2408.16673" class="abs-button download-eprint">TeX Source</a></li><li><a href="/format/2408.16673" class="abs-button download-format">Other Formats</a></li></ul>
      <div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
          <img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
          <span>view license</span>
        </a></div>
    </div>
    <!--end full-text-->    <div class="browse">
    Current browse context: <div class="current">cs.LG</div>

  <div class="prevnext">
<span class="arrow">
      <a class="abs-button prev-url" href="/prevnext?id=2408.16673&amp;function=prev&amp;context=cs.LG"
         accesskey="p" title="previous in cs.LG (accesskey p)">&lt;&nbsp;prev</a>
    </span>
    <span class="is-hidden-mobile">&nbsp; | &nbsp;</span>    <span class="arrow">
      <a class="abs-button next-url" href="/prevnext?id=2408.16673&amp;function=next&amp;context=cs.LG" accesskey="n"
         title="next in cs.LG (accesskey n)">next&nbsp;&gt;</a>
    </span><br/>
  </div><div class="list">
    <a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.LG/new">new</a>
    <span class="is-hidden-mobile"> | </span>
    <a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.LG/recent">recent</a>
    <span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.LG/2024-08">2024-08</a>
  </div><div class="abs-switch-cat">
    Change to browse by:
    <div class="switch context-change">
        <a href="/abs/2408.16673?context=cs">cs</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.16673?context=cs.AI">cs.AI</a><br class="is-hidden-mobile">
    </div>
  </div>

    </div>

    <div class="extra-ref-cite">
      <h3>References &amp; Citations</h3>
      <ul>
        <li><a  class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2408.16673">NASA ADS</a></li><li><a  class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2408.16673" target="_blank" rel="noopener">Google Scholar</a></li>
        <li><a  class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2408.16673" target="_blank" rel="noopener">Semantic Scholar</a></li>
      </ul>
      <div style="clear:both;"></div>
    </div>

<div class='extra-ref-cite'>
    <a id='bib-cite-css' hidden='true' href='/static/browse/0.3.4/css/cite.css'>a</a>

    <span id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</span>
    <span id='bib-cite-loading' hidden='true'>Loading...</span>
</div>

<div id='bib-cite-modal' class='bib-modal' hidden='true'>
    <div class='bib-modal-content'>
        <div class='bib-modal-title'>
            <h2>BibTeX formatted citation</h2>
            <span class='bib-modal-close' >&times;</span>
        </div>
        <div>
            <textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
        </div>
        <div>
            <span>Data provided by: </span>
            <a id='bib-cite-source-api'></a>
        </div>
    </div>
</div><div class="bookmarks">
  <div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2408.16673&amp;description=Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity"
     title="Bookmark on BibSonomy">
    <img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
         alt="BibSonomy logo"/>
  </a>
  <a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2408.16673&amp;title=Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity"
     title="Bookmark on Reddit">
    <img src="/static/browse/0.3.4/images/icons/social/reddit.png"
         alt="Reddit logo"/>
  </a>
</div>  </div>
  <!--end extra-services-->
<!-- LABS AREA -->
<div id="labstabs">
  <div class="labstabs"><input type="radio" name="tabs" id="tabone"checked="checked">
    <label for="tabone">Bibliographic Tools</label>
    <div class="tab labs-display-bib">
      <h1>Bibliographic and Citation Tools</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input id="bibex-toggle" type="checkbox" class="lab-toggle">
              <span class="slider"></span>
              <span class="is-sr-only">Bibliographic Explorer Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
          </div>
        </div>
<div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="litmaps-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
                aria-labelledby="label-for-litmaps">
              <span class="slider"></span>
              <span class="is-sr-only">Litmaps Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
          </div>
        </div>
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="scite-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
                aria-labelledby="label-for-scite">
              <span class="slider"></span>
              <span class="is-sr-only">scite.ai Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
          </div>
        </div>
      </div>
        <div class="labs-content-placeholder labs-display" style="display: none;"></div>
        <div style="min-height: 15px" id="litmaps-open-in"></div>
        <div style="min-height: 15px" id="scite-open-in"></div>
    </div>


    <input type="radio" name="tabs" id="tabtwo">
    <label for="tabtwo">Code, Data, Media</label>
    <div class="tab">
      <h1>Code, Data and Media Associated with this Article</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input        
                id="catalyzex-toggle"
                data-script-url="/static/browse/0.3.4/js/catalyzex.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="dagshub-toggle"
                data-script-url="/static/browse/0.3.4/js/dagshub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
              <span class="slider"></span>
              <span class="is-sr-only">DagsHub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
          </div>
        </div>
  
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="gotitpub-toggle"
                data-script-url="/static/browse/0.3.4/js/gotitpub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
              <span class="slider"></span>
              <span class="is-sr-only">GotitPub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="paperwithcode-toggle"
                data-script-url="/static/browse/0.3.4/js/paperswithcode.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-pwc">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-pwc">Papers with Code</span> <em>(<a href="https://paperswithcode.com/" target="_blank">What is Papers with Code?</a>)</em>
          </div>
        </div>


        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="sciencecast-toggle"
                data-script-url="/static/browse/0.3.4/js/sciencecast.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
              <span class="slider"></span>
              <span class="is-sr-only">ScienceCast Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
          </div>
        </div>
      </div>

      <div id="catalyzex-output" style="display:none"></div>
      <div id="dagshub-output" style="display:none"></div>
      <div id="gotitpub-output" style="display:none"></div>
      <div id="pwc-output" style="display:none"></div>
      <div id="pwc-data-output" style="display:none"></div>
      <div id="sciencecast-output" style="display:none"></div>
    </div>


      <input type="radio" name="tabs" id="labstabs-demos-input">
      <label for="labstabs-demos-input" id="labstabs-demos-label">Demos</label>
      <div class="tab">
        <h1>Demos</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="replicate-toggle"
                  data-script-url="/static/browse/0.3.4/js/replicate.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
                <span class="slider"></span>
                <span class="is-sr-only">Replicate Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="spaces-toggle"
                  data-script-url="/static/browse/0.3.4/js/spaces.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="txyz-toggle"
                  data-script-url="/static/browse/0.3.4/js/txyz.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
            </div>
          </div>
        </div>
        <div id="replicate-output"></div>
        <div id="spaces-output"></div>
        <div id="txyz-output"></div>
      </div>
      <input type="radio" name="tabs" id="tabfour">
      <label for="tabfour">Related Papers</label>
      <div class="tab">
        <h1>Recommenders and Search Tools</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="influenceflower-toggle"
                data-script-url="/static/browse/0.3.4/js/influenceflower.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
                <span class="slider"></span>
                <span class="is-sr-only">Link to Influence Flower</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="connectedpapers-toggle" type="checkbox" class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
                       aria-labelledby="label-for-connected-papers" >
                <span class="slider"></span>
                <span class="is-sr-only">Connected Papers Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
                <span class="slider"></span>
                <span class="is-sr-only">Core recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
            </div>
          </div>          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="iarxiv-toggle"
                       type="checkbox"
                       class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/iarxiv.js"
                       aria-labelledby="label-for-iarxiv">
                <span class="slider"></span>
                <span class="is-sr-only">IArxiv recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-iarxiv">IArxiv Recommender</span>
              <em>(<a href="https://iarxiv.org/about">What is IArxiv?</a>)</em>
            </div>
          </div>
</div>
        <div id="influenceflower-output"></div>
        <div id="influenceflower-output-graph" style="display:none">
          <ul class="flower-tabs">
            <li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
          </ul>
          <div class="flower-tab-content">
            <div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
            <div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
            <div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
            <div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
          </div>
        </div>
        <div style="min-height: 15px" id="connectedpapers-output"></div>
        <div id="coreRecommenderOutput"></div>
        <div id="iarxivOutput"></div>
      </div>

      <input type="radio" name="tabs" id="tabfive">
      <label for="tabfive">
        About arXivLabs
      </label>
      <div class="tab">
        <div class="columns">
          <div class="column">
            <h1>arXivLabs: experimental projects with community collaborators</h1>
            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
            <p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
          </div>
          <div class="column is-narrow is-full-mobile">
            <p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
          </div>
        </div>
      </div>

    </div>
</div>
<!-- END LABS AREA -->
  <div class="endorsers">
    <a href="/auth/show-endorsers/2408.16673" class="endorser-who">Which authors of this paper are endorsers?</a> |
    <a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
    <span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
  </div>
  <script type="text/javascript" language="javascript">mathjaxToggle();</script>
</div>
      </div>
    </main>

    <footer style="clear: both;">
      <div class="columns is-desktop" role="navigation" aria-label="Secondary" style="margin: -0.75em -0.75em 0.75em -0.75em">
        <!-- Macro-Column 1 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/about">About</a></li>
                <li><a href="https://info.arxiv.org/help">Help</a></li>
              </ul>
            </div>
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>contact arXiv</title><desc>Click here to contact arXiv</desc><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>
                  <a href="https://info.arxiv.org/help/contact.html"> Contact</a>
                </li>
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>subscribe to arXiv mailings</title><desc>Click here to subscribe</desc><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
                  <a href="https://info.arxiv.org/help/subscribe"> Subscribe</a>
                </li>
              </ul>
            </div>
          </div>
        </div>
        <!-- End Macro-Column 1 -->
        <!-- Macro-Column 2 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/license/index.html">Copyright</a></li>
                <li><a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy Policy</a></li>
              </ul>
            </div>
            <div class="column sorry-app-links">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/web_accessibility.html">Web Accessibility Assistance</a></li>
                <li>
                  <p class="help">
                    <a class="a11y-main-link" href="https://status.arxiv.org" target="_blank">arXiv Operational Status <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 512" class="icon filter-dark_grey" role="presentation"><path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"/></svg></a><br>
                    Get status notifications via
                    <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/email/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>email</a>
                    or <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/slack/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-black" role="presentation"><path d="M94.12 315.1c0 25.9-21.16 47.06-47.06 47.06S0 341 0 315.1c0-25.9 21.16-47.06 47.06-47.06h47.06v47.06zm23.72 0c0-25.9 21.16-47.06 47.06-47.06s47.06 21.16 47.06 47.06v117.84c0 25.9-21.16 47.06-47.06 47.06s-47.06-21.16-47.06-47.06V315.1zm47.06-188.98c-25.9 0-47.06-21.16-47.06-47.06S139 32 164.9 32s47.06 21.16 47.06 47.06v47.06H164.9zm0 23.72c25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06H47.06C21.16 243.96 0 222.8 0 196.9s21.16-47.06 47.06-47.06H164.9zm188.98 47.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06h-47.06V196.9zm-23.72 0c0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06V79.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06V196.9zM283.1 385.88c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06v-47.06h47.06zm0-23.72c-25.9 0-47.06-21.16-47.06-47.06 0-25.9 21.16-47.06 47.06-47.06h117.84c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06H283.1z"/></svg>slack</a>
                  </p>
                </li>
              </ul>
            </div>
          </div>
        </div> <!-- end MetaColumn 2 -->
        <!-- End Macro-Column 2 -->
      </div>
    </footer>
  </div>

  <script src="/static/base/1.0.1/js/member_acknowledgement.js"></script>

</body>

</html>
2408.16673
 Large language models rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks. Cross Entropy (CE) loss is the de facto choice in SFT, but it often leads to overfitting and limited output diversity due to its aggressive updates to the data distribution. This paper aim to address these issues by introducing the maximum entropy principle, which favors models with flatter distributions that still effectively capture the data. Specifically, we develop a new distribution matching method called GEM, which solves reverse Kullback-Leibler divergence minimization with an entropy regularizer.  For the SFT of Llama-3-8B models, GEM outperforms CE in several aspects. First, when applied to the UltraFeedback dataset to develop general instruction-following abilities, GEM exhibits reduced overfitting, evidenced by lower perplexity and better performance on the IFEval benchmark. Furthermore, GEM enhances output diversity, leading to performance gains of up to 7 points on math reasoning and code generation tasks using best-of-n sampling, even without domain-specific data. Second, when finetuning with domain-specific datasets for math reasoning and code generation, GEM also shows less overfitting and improvements of up to 10 points compared with CE. 
abstract
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
Abstract
2408.16673
 Large Language Models (LLMs) [38, 53, 52] are powerful generative models excelling in specialized tasks across various fields. Typically, LLMs first go through a pre-training stage [42, 7], where they learn to predict the next token from a large corpus of texts, such as books, scientific papers, and code.  Despite this extensive pre-training, LLMs often struggle to follow instructions and answer users' ∗Corresponding author.  ![1_image_1.png](1_image_1.png)  ![1_image_0.png](1_image_0.png)  Next-token Distribution queries effectively, because such scenarios are not commonly encountered during pre-training. To improve their performance in these tasks, instruction tuning [44, 60, 10], also known as Supervised Fine-Tuning (SFT) [39, 3], is employed. This process involves using high-quality labeled data (i.e., prompt-response pairs) and typically utilizes supervised learning with Cross Entropy (CE) loss to maximize the likelihood of the labeled data. Later on, these models may be further aligned with human preferences to ensure their outputs align with human values [39, 3].  SFT elicits the knowledge acquired from pre-training to answer various downstream questions and further paves the way for future developments, making it crucial part of the post-training pipeline [68, 54, 34, 65]. We expect models to generalize well by providing accurate answers and hope these answers are diverse as well. While the importance of generalization is clear, we also stress the significance of generation diversity. In creative writing, diversity sparks new ideas [12], and in chit-chat dialogues, users often appreciate having multiple output options to suit their preferences [30]. Many modern AI interfaces, such as ChatGPT and Claude AI, recognize this need by incorporating features like regeneration buttons. Additionally, generation diversity is vital when advanced generation algorithms are applied in LLMs to tackle complex tasks [50]. For example, the best-of-n sampling, commonly used in math reasoning [50] and code generation [8], benefits from selecting the optimal response from a diverse set of generated options.  However, models trained using CE loss in SFT often struggle with overfitting (see, e.g., [16, 17]) and poor generation diversity (see, e.g., [40, 37]). In theory, optimizing CE loss corresponds to minimizing the *forward* Kullback–Leibler (KL) divergence between the data distribution and the generative distribution of the LLM.1 This distribution matching process aggressively increases the likelihood of observed data, which often exhibit narrow coverage of real distributions that have diverse outcomes that we want the LLM to learn. However, the CE loss is unaware of this, and the optimized model biases toward low-entropy distributions, resulting in reduced output diversity.  One particular example is shown in Figure 1 (details are provided in Appendix E). Prior research [41, 14] links low-entropy predictive distributions with poor generalization, indicating that these issues are interconnected. While practitioners often use weight decay regularization with CE loss [39, 3], it does not fully address these problems, necessitating more principled approaches.  Our contributions. We formulate fine-tuning of LLMs as a distribution matching problem 1It is called "forward" KL because the loss is defined across the data distribution. We will later discuss *reverse* KL approaches, where the loss is defined across the generative model's distribution.  and propose two principles. The first principle advocates *generative* distribution matching methods to encourage the model learns from both ground truth supervision and its own generated mistakes.  This contrasts with the passive imitation of supervision used in the CE loss. The second principle is that the model should assign higher probabilities to the observed data while preventing overmemorization2, especially when dealing with the limited data. To implement these principles, we study the formulation of reverse KL divergence minimization with entropy regularization. However, this formulation is technically challenging and may require adversarial training techniques akin to those used in GANs [18]. Our main technical contribution is the development of a new training algorithm, referred to as GEM (Generative Entropy-regularized Matching of distributions), that addresses the above challenge and is tractable as the CE loss. By adhering to the proposed principles, GEM favors flatter distributions that effectively capture the data; see Figure 1 for an illustration.  We demonstrate the effectiveness of our model by fine-tuning the Llama-3-8B pre-trained model3 with two types of datasets. First, we fine-tune the model using the UltraFeedback dataset [13] to develop general instruction-following abilities. Our results show that GEM achieves lower evaluation perplexity than CE and better performance on the IFEval benchmark [69], indicating reduced overfitting. We also assess output diversity by evaluating the model's ability to generate varied content in creative tasks, such as poem and story writing [36]. Our findings show that GEM significantly enhances output diversity. This improved diversity translates into performance gains in math reasoning (GSM8K [11]) and code generation tasks (HumanEval [8] and MBPP [2]) when utilizing advanced generation strategies like Majority Voting (MV) and Best-Of-N (BON). In these tasks, GEM achieves performance improvements of up to 7 points. In a second experiment, we fine-tune the model on domain-specific datasets for specialized abilities in math reasoning (MetaMathQA dataset [66]) and code generation (MagicCoder-OSS-Instruct dataset [62]), seperately. In this setting, GEM can outperform CE by up to 10 points when using MV and BON, further demonstrating its effectiveness. Overall, these results suggest that GEM can mitigate overfitting and improve output diversity compared with CE. 
introduction
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**Introduction**
2408.16673
 Supervised Fine-tuning. SFT is the first stage of the post-training pipeline and plays an important role in subsequent developments. As mentioned in the introduction, using CE loss during the SFT stage often leads to overfitting and reduced output diversity. To address this, there is a line of research in scaling up SFT data (see, e.g., [66, 62, 67]), which, while effective, increases computational burden. Our work aims to develop training methods that more effectively leverage supervised data to mitigate overfitting and to enhance output diversity. While recent studies such as [9, 29] attempt to improve CE-trained models through techniques like self-play, we aim to address the limitations of CE loss and design methods that can improve the pre-trained models directly.  We also emphasize the importance of output diversity based on previous research. SFT-trained models are often further refined through Reinforcement Learning from Human Feedback (RLHF) to better align with human values [39, 3]. Xiao et al. [64] studied the impact of SFT models on preference learning in RLHF, showing that if an SFT model collapses (i.e., becomes biased toward certain outputs with near-certain probability), it may further lead to preference collapse in the alignment. Their findings highlight the need to address collapse during the SFT stage. Additionally, SFT-trained models are often used as synthetic data generators for self-improvement (see, e.g., [1, 15]). In this context, maintaining output diversity is crucial. It helps find better solutions and alleviate the mode collapse issue [19, 49]. For specialized applications, Wang et al. [59] highlighted the benefits of output diversity through majority voting in math reasoning. Recent research [6, 50] have explored enhancing search over multiple generated responses with a verifier, showing that scaling up test-time compute is effective. Unlike these studies focused on inference-time techniques, our paper improves SFT training methods to promote diversity.  Entropy Regularization. Dubey et al. [14] investigated the application of maximum entropy in visual fine-grained classification tasks, where the challenge lies in distinguishing between very similar categories of objects. Although their objective differs from ours, their insights remain valuable. They proposed that achieving zero CE loss is not essential for high accuracy. Instead, they suggested that a conditional probability distribution where the argmax corresponds to the correct class is sufficient for many applications. This concept motivates our use of entropy regularization, which allows for assigning probabilities to alternative options beyond the observed data. Prior to our work, Pereyra et al. [41] also explored entropy regularization in the context of neural network training. Their method closely resembles the CE with entropy regularization that we investigate in this paper, and they found that penalizing confident outputs improves generalization. It is important to note that Pereyra et al. [41] focused on image classification tasks, while our focus is on text generation where data is sequential in nature and is more challenging. In the context of LLMs, Hu et al. [23] also explored the maximum entropy regularization by using GFlowNet [4], but their methods require a reward function rather than supervised data. 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**Related Work**
2408.16673
 Large Language Models (LLMs). LLMs have a large vocabulary, denoted as [K] = {1, 2*, . . . , K*} and process text by splitting it into a series of tokens (x1*, . . . , x*T ), where each token xi ∈ [K] and T represents the sequence length. Let f be the generative distribution modeled by the language model. The notation f(·|x1*, . . . , x*t−1) specifies the categorical distribution over [K], conditioned on the context (x1*, . . . , x*t−1). Typically, f is parameterized by a neural network, often a transformer [57], with the parameter θ. For the i-th token at time step t, its prediction probability is given by  $f_{\theta}(i|x_{1},\ldots x_{t-1})=\text{softmax}(z_{t})=\frac{\exp(z_{t}[i])}{\sum_{i^{\prime}}\exp(z_{t}[i^{\prime}])}$.   where zt ∈ R K is the logit output from the neural network given the input (x1*, . . . , x*t−1), and zt[i] is i-th element of zt. This auto-regressive process specifies the joint probability of a sequence of tokens as fθ(x1*, . . . , x*T ) = QT t=1 fθ(xt|x1*, . . . , x*t−1).  LLMs are pre-trained to predict the next token in a sequence, thereby learning complex conditional probability distributions from vast amounts of data. In practical applications, LLMs are tasked with generating a response y to a given prompt x. However, these question-answer scenarios often differ from the textbook-like pre-training data, causing pre-trained LLMs to struggle in generating responses that follow human instructions effectively.  Supervised Fine-Tuning. To address the above issue, Supervised Fine-Tuning (SFT) is introduced. This process involves using a supervised dataset with high-quality prompt-response pairs {(x i, yi)} N i=1 for demonstration. The CE loss is the de facto training objective for this purpose: minθPN i=1 − log fθ(y i|x i). In theory, this corresponds to minimizing the *forward* KL divergence between the data distribution p and the generative distribution fθ: min θ DKL (*p, f*θ) ⇐⇒ max θ Ex∼ρ(·)Ey∼p(·|x)[log fθ(y|x)], where ρ is the prompt distribution, which is usually not modeled during the SFT stage. Thus, the distribution ρ can be treated as a constant and we omit it when the context is clear. In practice, many questions can correspond to multiple valid answers, and it is nearly impossible to collect a comprehensive high-quality dataset that encompasses all possibilities. As a result, the empirical data tends to be limited in size and often exhibits a narrower distribution than desired. In such scenarios, the CE loss function aggressively maximizes the likelihood of the available empirical data, adjusting the generative distribution fθ to closely align with it. However, this approach can lead to poor generation diversity and overfitting, as previously noted. 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**Preliminary**
2408.16673
 In this paper, we explore principled approaches to address limitations of existing SFT that uses the CE loss, especially when dealing with limited data. We present two core principles: the first focuses on the methodology of distribution matching, while the second offers guidance on learning from limited data.  Our first principle advocates for a *generative* approach to distribution matching. This approach encourages the model to learn from its own generated data and mistakes, rather than merely imitating supervised demonstrations. Unlike the traditional CE loss, which leads the model to imitate training data labels passively, a generative approach involves active learning through self-generated feedback.  This principle is grounded in cognitive science [47, 20], which demonstrates that children learn more effectively through exploration and experimentation, adjusting their understanding based on discrepancies between expectations and reality. Similarly, research on Generative Adversarial Networks (GANs) [18, 24] supports this notion by showing how models can learn to produce realistic data through iterative refinement. To summarize, we propose: Principle 1: The distribution matching approach should be "generative", meaning the model learns from both ground truth supervision and its own generated mistakes.  Our second principle addresses the challenge of overfitting. We draw inspiration from neuroscience, specifically the concept of avoiding over-memorization and achieving balanced learning. In neuroscience, synaptic plasticity, particularly homeostatic plasticity, underscores the importance of maintaining balance in learning processes [56, 55]. Overly strengthening certain neural connections can lead to rigid, maladaptive behaviors, analogous to how assigning excessively high probabilities to observed tokens can result in over-memorization in models, thereby limiting their ability to adapt and generalize. Based on these insights, especially for limited data, we propose: Principle 2: The model should assign higher probabilities to the observed data while preventing over-memorization. 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**Entropic Distribution Matching**
2408.16673
 To implement the two principles outlined above, we propose studying the formulation of *reverse* KL divergence minimization with maximum entropy regularization. The objective is defined as follows: max f Ex Ey∼f(·|x)[log p(y|x)] − Ey∼f(·|x)[log f(y|x)] | {z } =−DKL(f,p)  +γ · Ey∼f(·|x)[log f(y|x)] | {z } H(f) 	. (1) The first term in this formulation corresponds to the *reverse* KL divergence between the target distribution p and the model distribution f. This term supports Principle 1 by encouraging the model to learn from its generated data samples. The second term, entropy regularization, aligns with Principle 2 by preventing over-memorization, as it ensures that the probabilities for labeled data do not become excessively high. Furthermore, it brings another benefit: the output diversity can be improved. In this case, greedy sampling can reliably output the knowledge in the training data. We also note that adding entropy regularization to the CE loss supports Principle 2 but not Principle 1; its limitations are discussed in Appendix C, and we will show that it is inferior to the proposed approach in Section 5.  While the objective defined in Equation (1) appears promising, it presents significant challenges in practice. The main challenge is that we only have access to empirical data from the distribution p, not its full probability density function, making the reverse KL term impossible to compute directly. Additionally, calculating the expectation of the reverse KL across the model's generative distribution is not easy. This paper contributes a new algorithm to address these challenges. 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**Proposed Formulation: Reserve Kl With Entropy Regularization**
2408.16673
 In this section, we present a practical algorithm for solving the optimization problem of reverse KL with entropy regularization. As discussed earlier, the key is to obtain an estimate for the log probability density function log p. We first address the case where y is not sequential. We begin by outlining a conceptually simple but technically complicated solution. Building on this proposal, we then introduce a more tractable solution by new techniques.  An Initial Proposal. Drawing inspiration from GANs [18, 25], one may propose the following formulation for estimating the distribution p: $$\operatorname*{min}_{q}\operatorname*{max}_{r}\mathbb{E}_{x}\mathbb{E}_{y^{\mathrm{real}}\sim p(\cdot|x)}\mathbb{E}_{y^{\mathrm{sense}}\sim q(\cdot|x)}\left[h\left(r(x,y^{\mathrm{real}})-r(x,y^{\mathrm{sense}})\right)\right].$$ -hr(*x, y*real) − r(*x, y*gene) . (2) Here we use the subscript real to denote the supervised data and gene to denote the modelgenerated data for clarity. In addition, h is a monotonically increasing function (e.g., a linear function) that will be discussed later, and r : *X × Y →* R acts as a discriminator, designed to maximize the gap between samples drawn from the data distribution p and those from the opponent distribution q. The goal of the opponent q is to minimize the gap. When h is a linear function, it simplifies to the reward maximization problem maxq Ey gene∼q(·|x)[r(*x, y*gene)], a fact that shall be used later.  On its theoretical foundation, Jolicoeur-Martineau [25] proved that the inner maximization problem is to find a divergence function between the distributions p and q, and thus the outer minimization problem is to reduce this divergence. Therefore, the solved q serves as a good estimate of p. If we can solve problem (2), we can substitute log p = log q in Equation (1). While this offers a conceptually viable solution, it also presents several challenges. In particular, the optimization problems in Equations (1) and (2) involves three variables: r, q, and f, which are hard to solve: - The variables q and r are introduced to be adversarially trained in Equation (2); - The distribution f needs to be further solved in Equation (1) after q has been solved.  It is important to note that optimizing the reward function r and the distribution q can be particularly challenging when dealing with sequential data, often requiring Reinforcement Learning (RL) algorithms (see, e.g., [22]). Our initial attempts to implement the above proposal were not successful. We introduce a new tractable solution below.  Input: Dataset D = {(xi, yreal i)} 1: for iteration k = 1*, . . . ,* do 2: Set qk = softmax(1/β ∗ log fθk ) 3: Define the loss function  $$\mathcal{L}_{q}(f_{\theta})=\sum_{i}\sum_{y^{\mathsf{gen}}}q(y^{\mathsf{gen}}|x_{i})\cdot h\left(\left[\log f_{\theta}(y_{i}^{\mathsf{real}}|x_{i})-\log f_{\theta}(y^{\mathsf{gen}}|x_{i})\right]\right)\cdot$$ $$\cdot\cdot\cdot=\theta_{i}+n\cdot\nabla\circ\mathcal{L}_{q}(f_{\theta})\mid_{\theta=\theta_{i}}$$  ![6_image_0.png](6_image_0.png)  7 4: Update θk+1 = θk + η · ∇θLq(fθ) |θ=θk Output: Generative model fθ Proposed Solution. At a high level, our approach simplifies the process by solving a single-stage optimization problem, eliminating the need to first estimate the distribution p before solving for f. Our approach involves two key techniques:  - Reparameterization: We reparameterize the discriminator using the generative distribution f. Specifically, we define r as a real-valued function and parameterize it as log f(y|x), ensuring it remains real-valued.  - Direct entropy regularization: We introduce entropy regularization for the distribution q directly, deriving a closed-form solution for q. This technique offers two advantages. First, it establishes a connection between q and f, eliminating the need of solving problem (1) separately. Second, since q has a closed-form solution, it does not require explicit training. Specifically, our formulation is that:  $$\max_{f}\quad\mathcal{L}_{q}(f)\triangleq\mathbb{E}_{x}\mathbb{E}_{y^{\text{post}}\sim p(\cdot|x)}\mathbb{E}_{y^{\text{post}}\sim q(\cdot|x)}\left[h\left(\log f(y^{\text{post}}|x)-\log f(y^{\text{post}}|x)\right)\right]\tag{3}$$ $$\text{s.t.}\quad q=\underset{\pi}{\operatorname{argmax}}\mathbb{E}_{x}\mathbb{E}_{y\sim p(\cdot|x)}\left[\log f(y|x)\right]+1/\beta\cdot\mathcal{H}(\pi(\cdot|x))=\texttt{softmax}(1/\beta*\log f).\tag{4}$$  In this formulation, we optimize f as if we were optimizing the reward function r. Simultaneously, we optimize q with the reward given by log f(y|x), incorporating entropy regularization. Fortunately, this yields a closed-form solution, so we do not need to maintain or explicitly train q. We only need to optimize the distribution f in Equation (3). Note that although q is dependent on f generally, we do not calculate the gradient through q when calculating the gradient of f. This is similar to the target network used in RL [35, 32]. We have the following theoretical justification for this formulation.  Proposition 1. Assume that h is a linear function. For any distribution p, Lq(f) has a unique stationary point, and this stationary point (with β = 1/(γ + 1) > 0) corresponds to the optimal solution of Problem (1).  Proposition 1 implies that solving the proposed problem in Equations (3) and (4) provides the optimal solution of reverse KL with entropy regularization in Equation (1). In practice, we can parameterize f using a transformer and optimize the parameters with gradient ascent. We outline such a training procedure in Algorithm 1, referring to this approach as GEM, which stands for Generative and Entropy-regularized Matching of distributions. Note that the exact expectation Ey gene∼q(·|x)[·] is calculated because we assume y gene is non-sequential here (we will deal with the sequential case later). Concretely, in this case the generative distribution is categorical and has finite support, making the expectation easy to compute. This differs from the continuous optimization in Algorithm 2 GEM for Sequential Data  Input: Dataset D = {(xi, y1*, . . . , y*T )}  ![7_image_0.png](7_image_0.png)  Output: Generative model fθ GANs [18], where the generator uses inexact stochastic gradients by drawing several samples. We also note that Proposition 1 relies on β > 0, meaning that the proposed formulation cannot solve the pure reverse KL minimization problem. Intuition and Example. We provide an intuitive understanding of GEM by explaining its training mechanism on a simple model: for a fixed x ∈ X , we model fθ(y|x) = softmax(θx) with θx ∈ R K.  Consider h as the linear function described in Proposition 1. For a paired sample (y real, ygene) = (*i, j*), we have the gradient for this sample:  $$\nabla_{\theta}{\mathcal{L}}_{q}(f_{\theta})[i,j]={\left\{\begin{array}{l l}{w_{i j}e_{i j}}&{{\mathrm{if}}}\quad i\neq j}\\ {{\mathbf{0}}}&{{\mathrm{~otherwise}}}\end{array}\right.}$$  ![7_image_1.png](7_image_1.png)  Here wij = p(y real|x)q(y gene|x) lies in [0, 1], and eij is the vector with i-th element being 1 and the j-th element being −1 and 0 otherwise. Thus, the gradient of this paired data gives a direction for moving the logit θx from j-th position to i-th position, with the magnitude determined by the weight wij .  Consider a numerical example where θx = [2, 1] with K = 2, resulting in f = [0.73, 0.27].  For β = 0.7, we have q = [0.81, 0.19], which is more peaked compared with f. Given the data distribution p = [0.9, 0.1], the gradient of GEM is [0.1, −0.1], leading to a relative logit change of 0.2. In comparison, the CE's gradient in this case is [0.17, −0.11], resulting in a relative logit change of 0.28, which is 1.4 times larger. Thus, the probability change of GEM is more milder than CE, and GEM would eventually converge to a flatter distribution [0.82, 0.18] due to the induced entropy regularization.  In the above analysis, we see that the distribution q determines the weights of probability transportation. Generally, for 0 *< β <* 1, a *narrowed* distribution q, shifted from f, prioritizes the high-probability regions in f for probability transportation, while low-probabilities regions in f contributes less. This contrasts with CE, which would push probabilities of non-labeled tokens towards the labeled ones, potentially causing overfitting. We also observe that h also determines how much probability is shifted. Specifically, we have wij = p(y real|x)q(y gene|x)h ′for a general function h. For the linear function studied, h ′is always equal to 1. Another possible choice for h is the log-sigmoid function h(u) = log sigmoid(u) = u − log(1 + exp(u)), which is studied in previous research [25, 51, 43, 9]. This function provides a weighting effect. Since h ′ = sigmoid(log f(y gene|x) − log f(y real|x)) ∈ (0, 1), it results in a large weight when y real is not yet dominant in the probability distribution, and a small weight when y real has already become dominant. Later on, we will study this function in experiments.  Extensions to Sequential Data. In the above part, we have derived the algorithm for the case y is non-sequential. We note that optimization in the sequential case could be highly difficult. With a little abuse of notations, let y = (y1*, . . . , y*T ) ≜ y1:T . We can extend the formulation in Equations (3) and (4) to the following:4  $$\operatorname*{max}_{f}\quad\mathbb{E}_{x}\mathbb{E}_{y_{1:T}^{\mathrm{real}}\sim p(\cdot|x)}\mathbb{E}_{y_{1:T}^{\mathrm{seen}}\sim q(\cdot|x)}\left[h\left(\log f(y_{1:T}^{\mathrm{real}}|x)-\log f(y_{1:T}^{\mathrm{sense}}|x)\right)\right].$$ $$\mathrm{s.t.}\quad q=\operatorname*{argmax}_{\pi}\mathbb{E}_{x}\mathbb{E}_{y_{1:T}\sim\pi(\cdot|x)}\left[\log f(y_{1:T}|x)\right]+1/\beta\cdot\mathcal{H}(\pi(\cdot|x))$$  Here, we encounter a challenge: the joint distribution of y1:T , as a cascaded categorical distribution, is quite complicated. This results in the expectation Ey gene 1:T [·] cannot be easily calculated as before.  While Monte Carlo estimation—drawing samples to approximate the gradient—might seem like a viable solution, we found it does not work in experiments. We believe the main reason is that the sample space is huge,5 and the pre-trained distribution f is quite different from the data distribution p that we aim to learn.6 As a result, when we use stochastic sampling to estimate the gradient, it does not provide effective feedback.  To deal with the above challenge, we propose decomposing the multi-stage sequential optimization problem into multiple single-stage optimization problems and solve each efficiently. Concretely, we restrict the distribution matching to the case that the prefix samples up to time step t are drawn from the data distribution p and solves the optimization problem at the t-th time step as before.  Its mathematical formulation is given below:  $$\max_{f}\mathcal{L}_{q}^{\mathsf{seq}}(f)=\mathbb{E}_{x}\left\{\sum_{t=1}^{T}\mathbb{E}_{y_{1:t-1}^{\mathsf{real}}\sim\rho(\cdot|x)}\mathbb{E}_{y_{1:t-1}^{\mathsf{real}}\sim\rho(\cdot|x,y_{1:t-1}^{\mathsf{real}})}\mathbb{E}_{y_{1:t-1}^{\mathsf{real}}\sim\rho(\cdot|x,y_{1:t-1}^{\mathsf{real}})}\left[\Delta\right]\right\}\tag{5}$$  where $\Delta=\left[h\left(\log f(y_{t}^{\mathsf{real}}|x,y_{1:t-1}^{\mathsf{real}})-\log f(y_{t}^{\mathsf{mean}}|x,y_{1:t-1}^{\mathsf{real}})\right)\right],$ The main advantage of this formulation is that for each sub-problem, we still have access to the exact conditional distribution, so the gradient estimation is accurate. The same idea applies to the training of distribution q:  $$q=\operatorname*{argmax}_{\pi}\mathbb{E}_{x}\left\{\sum_{t=1}^{T}\mathbb{E}_{y_{t}^{\text{max}}-1}\circ p(\cdot|x)\mathbb{E}_{y_{t}-\pi(\cdot|x,y_{t}^{\text{max}}-1)}[\log f(y_{t}|x,y_{t-1}^{\text{max}})]+1/\beta\cdot\mathcal{H}(\pi(\cdot|x,y_{t-1}^{\text{max}}))\right\}.$$  That is, we still have the closed-form solution that q(·|*x, y*real 1:t−1 ) = softmax(1/β · log f(·|*x, y*real 1:t−1 )) 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**Proposed Algorithm: Gem**
2408.16673
 In this section, we present our experiment results for fine-tuning the Llama-3-8B model (specifically, its pre-trained version). A brief overview of our experiment setting is provided below, with further details available in Appendix D. 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**Experiments**
2408.16673
 Set-up. In this section, we develop an LLM that is capable of following instructions for various prompts. To this end, we utilize the UltraFeedback dataset [13], specifically the version filtered by the HuggingfaceH4 team7. This dataset contains prompts from instruction datasets like Evol-Instruct and UltraChat, and responses generated by models such as GPT-4 and Llama-2-7B/13B/70B-Chat.  For more information, see [13].  Each data point comprises two responses: one selected as the preferred option and the other as the rejected option, with the selection made by GPT-4. In our study, we use the preferred response for SFT, a practice commonly adopted in previous research [39, 3]. Following [66, 34, 13] we set the learning rate to 2 × 10−5, employing a cosine learning rate decay schedule, and use a macro batch size of 128. The maximum sequence length, encompassing both the prompt and response, is set to 2,048 tokens. Models are trained for three epochs.  We implement the proposed GEM method with β = 0.7. As discussed, GEM has two variations: GEM-LS (GEM with log-sigmoid), and GE-Linear, each depending on the choice of the function h. Our primary baseline is the standard CE loss. Additionally, we explore a variant incorporating a weight decay of 0.1, which has been commonly used in previous studies [39, 3]. We refer to this approach as CE + WD. We also implement a method called CE + Entropy, which adds an entropy regularization term of 0.1 to the CE loss. This method aligns with the proposed Principle 2 but not Principle 1 (see Appendix C for more discussion).  Instruction-Following. We first examine the model's learned ability in terms of instruction-following.  We follow the IFEval benchmark in [69], which includes 500 prompts from 25 types of verifiable instructions, such as writing more than 400 words. Depending on whether we use strict or loose judgment and whether we measure accuracy in the prompt space or instruction space, there are four evaluation criteria: prompt-level strict accuracy, instruction-level strict accuracy, prompt-level loose accuracy, and instruction-level loose accuracy. For all metrics, a higher value indicates better performance.  We evaluate the trained models using *greedy decoding* and present the results in Table 1. We observe that CE underperforms compared with regularization-based methods, such as weight decay and entropy regularization, suggesting that CE suffers from overfitting. On average across the four criteria, GEM-LS improves by 1.1 points (2.5% relative) and 1.4 points (3.2% relative) compared to CE. We also observe this overfitting in the evaluation perplexity: GEM-LS and GEM-Linear achieve lower perplexity (around 3.16) than CE (3.48); see Appendix E. It is important to note that this overfitting is not due to over-optimization, as performance continues to improve over three training epochs for CE (36.15 in epoch 1, 41.45 in epoch 2, and 43.70 in epoch 3).  Creative Writing. We continue to assess models' output diversity in two creative writing tasks: poem writing and story writing. For poems, we use prompts from the poetry8 dataset on the  | Instruction-Following   |                     |                 |                     |                | |-------------------------|---------------------|-----------------|---------------------|----------------| | Method                  | Strict Accuracy     | Strict Accuracy | Loose Accuracy      | Loose Accuracy | | (Prompt Level)          | (Instruction Level) | (Prompt Level)  | (Instruction Level) |                | | CE                      | 36.23               | 46.76           | 40.85               | 50.96          | | CE+WD                   | 37.89               | 47.48           | 42.88               | 52.52          | | CE+Entropy              | 36.78               | 47.60           | 40.66               | 51.08          | | GEM-Linear              | 37.34               | 48.20           | 41.96               | 52.64          | | GEM-LS                  | 37.52               | 47.60           | 42.14               | 52.04          |  Table 1: Performance of instruction-following on the benchmark IFEval [69]. For all metrics, a higher value means a better instruction following ability. The best results are shown in bold, with the second-best underlined.  Table 2: Evaluation of generation diversity in creative tasks of poem writing and story writing. For all criterion, a higher value indicates greater diversity.  | Poem Writing   | Story Writing   |           |               |        |           |               | |----------------|-----------------|-----------|---------------|--------|-----------|---------------| | Method         | N-gram          | Self-BLEU | Sentence-BERT | N-gram | Self-BLEU | Sentence-BERT | | CE             | 48.50           | 72.50     | 21.79         | 48.74  | 72.77     | 21.94         | | CE+WD          | 48.58           | 71.29     | 21.80         | 48.85  | 71.73     | 21.79         | | CE+Entropy     | 53.74           | 75.82     | 23.80         | 53.86  | 76.11     | 23.94         | | GEM-Linear     | 56.50           | 76.73     | 24.73         | 56.69  | 76.83     | 24.82         | | GEM-LS         | 56.55           | 76.31     | 24.63         | 56.82  | 76.61     | 24.68         |  Huggingface website, which includes 573 poems on themes such as love, nature, and mythology. For stories, we design 500 prompts based on the ROC story dataset [36]. In both cases, we prompt the models to write a poem or story titled "[X]" with no more than 200 words, where [X] is a title from the respective dataset. Following [26], we use three criteria to evaluate diversity: 1) N-gram diversity: the proportion of distinct n-grams in a single response (intra-diversity); 2) Self-BLEU diversity: the Self-BLEU score, treating one response as a reference among multiple generated responses (inter-diversity); 3) Sentence-BERT diversity: the cosine dissimilarity between pairs of responses in the embedding space. All criteria range from 0 to 100 (with Sentence-BERT diversity scaled by multiplying by 100), where higher values indicate greater diversity.  To calculate these metrics, we ask the trained models to generate 16 samples using the decoding configuration temperature=1, top k=50, and top p=0.9. The evaluation results are presented in Table 2. In this task, we note that weight decay does not improve generation diversity, although it has shown effectiveness in mitigating overfitting in previous examples. On the other hand, entropy regularization, implemented to support Principle 2, brings the benefit of output diversity. Chatting, Math Reasoning, and Code Generation. In this part, we show that improved generation diversity offers benefits beyond creative writing tasks. Specifically, diverse generation, when quality is ensured, is advantageous when using advanced generation methods such as Best-Of-N (BON) or Majority-Voting (MV) [59] to find better solutions. This is inline with recent advances in scaling up test-time compute [6, 50] and self-distillation [48]. Specifically, we conduct three experiments in chatting, math reasoning and code generation below to validate the superiority of GEM through its improved generation diversity. In this part, we use the configuration temperature=0.6, top k=50,  ![11_image_0.png](11_image_0.png)  and top p=0.9. A lower temperature is chosen here to enhance response quality (0.6 is the default value in Llama models). The overall performance is displayed in Figure 2 with detailed results in Appendix E. Specific settings and analyses are provided below.  For chatting, we assess the model's ability to generate human-preferred responses. We prompt the trained models to answer 805 questions from the AlpacaEval dataset [31]. For each question, the model generates 32 responses and a reward model is then used to select the best responses. We employ the reward model FsfairX-LLaMA3-RM-v0.19, which has demonstrated top performance on RewardBench [27], making it a reliable choice for this task. Since the reward value itself does not mean anything, we choose the win rate as a metric. In particular, we estimate the win rate over GPT-4's generated response by the Bradley–Terry model. From Figure 2, we observe that GEM-LS can achieve about 3 points improvement in the win rate compared with CE.  For math reasoning, we evaluate performance on the GSM8K [11] benchmark, which contains 1,319 test questions. We use chain-of-thought prompts [61] to guide LLMs to generate 32 responses for each question. We assess answer accuracy using both Majority-Voting (MV) [59] and Best-Of-N (BON) methods. Compared with CE, GEM-LS shows improvements of up to 4.8 points (7.7% relative) with MV and 2.5 points (2.8% relative) with BON. The strong performance of BON@32 indicates that while the model might know how to solve these questions, it is uncertain about these solutions in generation. This aligns with previous research [28], which found that even 7B language models demonstrate strong math reasoning abilities through sampling multiple responses.  For code generation, we consider two benchmarks: HumanEval [8] and MBPP [2]. In these scenarios, the trained models are asked to generate Python code, and the executor judges their correctness.  The common evaluation metric is the pass rate over multiple samples. We ask the trained models to generate 200 samples to estimate the pass@100. The generation configuration is the same as for the chatting task. We find that weight decay does not show significant improvement over CE, while GEM-LS can achieve up to a 7.6-point (10.7% relative) improvement over CE on HumanEval and a 6.4-point (9.0% relative) improvement on MBPP for pass@100.  tti 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**General-Purpose Fine-Tuning**
2408.16673
 In this section, we conduct experiments with domain-specific datasets. For math reasoning, we use the dataset MetaMathQA [66]. For code generation, we use the dataset Magicoder-OSS-Instruct 9https://huggingface.co/sfairXC/FsfairX-LLaMA3-RM-v0.1  ![12_image_0.png](12_image_0.png)  [62]. The experimental setup, including training details and hyperparameters, is the same as before, and the specifics are provided in the Appendix.  Math Reasoning. We present the final answer accuracy for two benchmarks: GSM8K, which has been studied previously, and MATH [21], which is more challenging10, as shown in Figure 3. Following the previous set-up, we evaluate performance using Majority Voting over 32 samples (MV@32), and Best-Of-N over 32 samples (BON@32). The greedy decoding performance is also reported. Our results show significant improvements in both majority voting and greedy decoding across both GSM8K and MATH. We observe that the weight decay regularization performs well on GSM8K but shows no clear improvement on MATH. In contrast, GEM-LS outperforms CE on GSM8K by 1.2 points (1.7% relative), 2.9 points (3.8% relative), and 2.6 points (2.9% relative) for greedy decoding, MV@32, and BON@32, respectively. On the MATH benchmark, GEM-LS shows improvements of 1.9 points (8.0% relative), 1.7 points (5.8% relative), and 1.6 points (2.7% relative) for the same methods. These improvements in greedy decoding indicate that entropy regularization methods effectively mitigate overfitting, while the enhancements in MV and BON suggest increased generation diversity.  Code Generation. Following the previous set-up, we report the pass rate over {1, 10, 100} on two key benchmarks, HumanEval and MBPP, in Figure 4. We find that the Pass@100 performance on HumanEval is reduced compared with the previous one while the Pass@100 performance on MBPP improved for all methods. We observe that weight decay (WD) does not achive consistent improvement while entropy regularization does. Notably, GEM-LS significantly enhances performance over CE: on HumanEval, it improves by 4.6 points (11.7% relative) for Pass@1, 6.5 points (11.1% relative) for Pass@10, and 9.7 points (14.7% relative) for Pass@100. On MBPP, GEM-LS achieves gains of 3.4 points (6.3% relative) for Pass@1, 6.8 points (10.2% relative) for Pass@10, and 8.0 points (11.1% relative) for Pass@100. These results suggest similar conclusions regarding overfitting and generation diversity as before.  10MATH is not studied in models fine-tuned with the UltraFeedback dataset because models have poor performance in this task.  ![13_image_0.png](13_image_0.png) 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**Domain-Specific Fine-Tuning**
2408.16673
 In this paper, we propose an alternative method for the SFT of LLMs to tackle the challenges of overfitting and limited generation diversity, which are often caused by the aggressive updates of the CE loss and limited data. We demonstrate the effectiveness of combining generative distribution matching with entropy regularization. We note that the improved diversity also boosts performance in downstream tasks when advanced generation methods, such as the best-of-n sampling, are used. Overall, our results indicate that the proposed method is well-suited for generative models.  We focus on the initial stage of post-training pipeline in this paper and recognize that the models trained with our proposed methods can be further refined in subsequent stages. Notably, the enhanced diversity achieved by our approach can be advantageous in several contexts: it supports scaling up test-time computation [6, 50], improves exploration in RL methods [46, 33], addresses the preference collapse issue [64], facilitates self-improvement through distillation with best-of-n techniques [48], and helps mitigate mode collapse in synthetic data generation [49, 5, 63]. We see significant potential for our method in these areas and plan to explore these topics in future work. 
conclusion
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
**Conclusion**
2408.16673
 We thank Yushun Zhang for reading the manuscript and providing useful feedback. 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
Acknowledgement
2408.16673
 [1] Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. *arXiv preprint arXiv:2406.11704*, 2024.  [2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. *arXiv preprint arXiv:2108.07732*, 2021.  [3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. *arXiv preprint arXiv:2204.05862*, 2022.  [4] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381–27394, 2021.  [5] Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. On the stability of iterative retraining of generative models on their own data. arXiv preprint arXiv:2310.00429, 2023.  [6] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R´e, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. *arXiv preprint arXiv:2407.21787*, 2024.  [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. *Advances in Neural Information Processing Systems 33*, pages 1877–1901, 2020.  [8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. *arXiv preprint arXiv:2107.03374*, 2021.  [9] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. *arXiv preprint arXiv:2401.01335*, 2024.  [10] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. *Journal of Machine Learning Research*, 25(70):1–53, 2024.  [11] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. *arXiv preprint arXiv:2110.14168*, 2021.  [12] Simon Colton and Geraint A Wiggins. Computational creativity: The final frontier? In *ECAI* 2012, pages 21–26. IOS Press, 2012.  [13] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. In *Forty-first International Conference on Machine Learning*, 2024.  [14] Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Maximum-entropy fine grained classification. *Advances in neural information processing systems*, 31, 2018.  [15] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. *arXiv preprint arXiv:2407.21783*, 2024.  [16] Tingchen Fu, Deng Cai, Lemao Liu, Shuming Shi, and Rui Yan. Disperse-then-merge: Pushing the limits of instruction tuning via alignment tax reduction. *arXiv preprint arXiv:2405.13432*, 2024.  [17] Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations? *arXiv preprint* arXiv:2405.05904, 2024.  [18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. *Advances in neural* information processing systems, 27, 2014.  [19] Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chlo´e Clavel. The curious decline of linguistic diversity: Training language models on synthetic text. *arXiv preprint arXiv:2311.09807*, 2023.  [20] Hyowon Gweon, Hannah Pelton, Jaclyn A Konopka, and Laura E Schulz. Sins of omission: Children selectively explore when teachers are under-informative. *Cognition*, 132(3):335–341, 2014.  [21] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.  arXiv preprint arXiv:2103.03874, 2021.  [22] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In *Advances in* Neural Information Processing Systems 29, pages 4565–4573, 2016.  [23] Edward J Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. arXiv preprint arXiv:2310.04363, 2023.  [24] Alexia Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard gan. *arXiv preprint arXiv:1807.00734*, 2018.  [25] Alexia Jolicoeur-Martineau. On relativistic f-divergences. In International Conference on Machine Learning, pages 4931–4939. PMLR, 2020.  [26] Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. *arXiv preprint arXiv:2310.06452*, 2023.  [27] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. *arXiv preprint arXiv:2403.13787*, 2024.  [28] Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706, 2024.  [29] Jiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Getting more juice out of the sft data: Reward learning from human demonstration improves sft for llm alignment. *arXiv preprint arXiv:2405.17888*, 2024.  [30] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. *arXiv preprint arXiv:1510.03055*, 2015.  [31] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023.  [32] Ziniu Li, Tian Xu, and Yang Yu. A note on target q-learning for solving finite mdps with a generative oracle. *arXiv preprint arXiv:2203.11489*, 2022.  [33] Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient method for aligning large language models. *arXiv preprint* arXiv:2310.10505, 2023.  [34] Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. *arXiv* preprint arXiv:2312.15685, 2023.  [35] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning. *Nature*, 518(7540):529–533, 2015.  [36] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In *Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 839–849, 2016.  [37] Laura O'Mahony, Leo Grinsztajn, Hailey Schoelkopf, and Stella Biderman. Attributing mode collapse in the fine-tuning of large language models. In *ICLR 2024 Workshop on Mathematical* and Empirical Understanding of Foundation Models, 2024.  [38] OpenAI. Gpt-4 technical report. *arXiv preprint arXiv:2303.08774*, 2023.  [39] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. *Advances in Neural Information Processing Systems* 35, pages 27730–27744, 2022.  [40] Vishakh Padmakumar and He He. Does writing with language models reduce content diversity?  arXiv preprint arXiv:2309.05196, 2023.  [41] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017.  [42] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners. *OpenAI blog*, 2019.  [43] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.  arXiv preprint arXiv:2305.18290, 2023.  [44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. *Journal of machine learning research*, 21(140):1–67, 2020.  [45] St´ephane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pages 627–635, 2011.  [46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. *arXiv*, 1707.06347, 2017.  [47] Laura E Schulz and Elizabeth Baraff Bonawitz. Serious fun: preschoolers engage in more exploratory play when evidence is confounded. *Developmental psychology*, 43(4):1045, 2007.  [48] Pier Giuseppe Sessa, Robert Dadashi, L´eonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ram´e, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning llms with best-of-n distillation. *arXiv preprint arXiv:2407.14622*, 2024.  [49] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. *arXiv* preprint arXiv:2305.17493, 2023.  [50] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. *arXiv preprint arXiv:2408.03314*, 2024.  [51] Ruoyu Sun, Tiantian Fang, and Alexander Schwing. Towards a better global loss landscape of gans. *Advances in Neural Information Processing Systems*, 33:10186–10198, 2020.  [52] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi`ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. *arXiv preprint arXiv:2403.08295*, 2024.  [53] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. *arXiv preprint arXiv:2307.09288*, 2023.  [54] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Cl´ementine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. *arXiv preprint arXiv:2310.16944*, 2023.  [55] Gina Turrigiano. Homeostatic synaptic plasticity: local and global mechanisms for stabilizing neuronal function. *Cold Spring Harbor perspectives in biology*, 4(1):a005736, 2012.  [56] Gina G Turrigiano. The self-tuning neuron: synaptic scaling of excitatory synapses. *Cell*, 135 (3):422–435, 2008.  [57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In *Advances in Neural Information* Processing Systems 30, pages 5998–6008, 2017.  [58] Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, R´emi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in reinforcement learning. In Advances in Neural Information Processing Systems 33, pages 12163–12174, 2020.  [59] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In *Proceedings of the 11st International Conference on Learning Representations*, 2023.  [60] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. *arXiv* preprint arXiv:2109.01652, 2021. [61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.  Advances in neural information processing systems, 35:24824–24837, 2022.  [62] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with oss-instruct. In *Forty-first International Conference on Machine Learning*, 2024.  [63] Ting Wu, Xuefeng Li, and Pengfei Liu. Progress or regress? self-improvement reversal in post-training. *arXiv preprint arXiv:2407.05013*, 2024.  [64] Jiancong Xiao, Ziniu Li, Xingyu Xie, Emily Getzen, Cong Fang, Qi Long, and Weijie J Su.  On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization. *arXiv preprint arXiv:2405.16455*, 2024.  [65] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai.  arXiv preprint arXiv:2403.04652, 2024.  [66] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. *arXiv preprint arXiv:2309.12284*, 2023.  [67] Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method. *arXiv preprint arXiv:2402.17193*, 2024. [68] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for alignment. *arXiv preprint arXiv:2305.11206*, 2023.  [69] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. *arXiv preprint* arXiv:2311.07911, 2023. 
references
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
References
2408.16673
 1 def gem_loss(logits, labels, beta=0.7, ignore_index=-100, h="linear"): 2  ```  3 shift_logits = logits[..., :-1, :].contiguous()  4 shift_labels = labels[..., 1:].contiguous()  5  6 mask = shift_labels != ignore_index  7 shift_logits = shift_logits[mask]  8 shift_labels = shift_labels[mask]  9 10 with torch.no_grad(): 11 logits_on_labels = torch.gather(  ```  ![19_image_0.png](19_image_0.png)  We have two remarks regarding the implementation above. First, we use a coefficient of 0.01 to scale the input in the log-sigmoid function. This ensures that the function behaves nearly linearly.  Second, this implementation requires almost the same GPU memory and computation time as the CE loss. 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
A **Implementation Of Gem**
2408.16673
 Proof of Proposition 1. When h is a linear function, we have that  Lq(f)  = ExEy real∼p(·|x) Ey gene∼q(·|x)  -log f(y real|x)  − log f(y gene|x)  = ExEy real∼p(·|x) Ey gene∼q(·|x)  -log f(y real|x) − ExEy real∼p(·|x) Ey gene∼q(·|x) [log f(y gene|x) ] = ExEy real∼p(·|x)  -log f(y real|x) − ExEy gene∼q(·|x) [log f(y gene|x) ] For any x ∈ X , we have that ∂L ∂f = p − q f(6)  To calculate the stationary point of L, we require that p = q. Since q = softmax(1/β · log f) , the above equality requires that f = softmax(β · log p) . As analyzed in Proposition 2, for β = 1/(γ + 1) , this corresponds to the the optimal solution of minimizing reverse KL with entropy regularization. Proposition 2. *For the entropy-regularized KL minimization problem in Equation* (1) *, in the* function space, we have the optimal solution:  $$f^{\star}(y|x) =\frac{1}{Z_{x}}p(y|x) ^{1/(\gamma+1) },$$  _where $Z_{x}$ is a normalization constant $\sum_{y^{\prime}}p(y^{\prime}|x) ^{1/(\gamma+1) }$._ (6)  $$\binom{6}{4}$$. 
body
Entropic Distribution Matching In Supervised Fine-Tuning Of Llms: Less Overfitting And Better Diversity
B **Proof**