<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

<head>  <title>[2408.16293] Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
  <link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
  <link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20240822" />
  <link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
  <script language="javascript" src="/static/browse/0.3.4/js/accordion.js" /></script>
  
    <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/slider.css?v=1.14" />
    <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="/static/browse/0.3.4/js/donate.js?v=1.11"></script><link rel="canonical" href="/abs/2408.16293"/><meta property="og:type" content="website" />
<meta property="og:site_name" content="arXiv.org" />
<meta property="og:title" content="Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems" />
<meta property="og:url" content="https://arxiv.org/abs/2408.16293v1" />
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="700" />
<meta property="og:image:alt" content="arXiv logo"/>
<meta property="og:description" content="Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to &#34;self-correct&#34; their mistakes via multi-round prompting. In this paper, we follow this line of work but focus on understanding the usefulness of incorporating &#34;error-correction&#34; data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. Using a synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount of error-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed on the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others."/>
<meta name="twitter:site" content="@arxiv"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Physics of Language Models: Part 2.2, How to Learn From Mistakes..."/>
<meta name="twitter:description" content="Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active..."/>
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
<meta name="twitter:image:alt" content="arXiv logo"/>
  <link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/>  <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script>  <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
  <script src="/static/browse/0.3.4/js/toggle-labs.js?20230622" type="text/javascript"></script>
  <script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems" /><meta name="citation_author" content="Ye, Tian" /><meta name="citation_author" content="Xu, Zicheng" /><meta name="citation_author" content="Li, Yuanzhi" /><meta name="citation_author" content="Allen-Zhu, Zeyuan" /><meta name="citation_date" content="2024/08/29" /><meta name="citation_online_date" content="2024/08/29" /><meta name="citation_pdf_url" content="http://arxiv.org/pdf/2408.16293" /><meta name="citation_arxiv_id" content="2408.16293" /><meta name="citation_abstract" content="Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to &#34;self-correct&#34; their mistakes via multi-round prompting. In this paper, we follow this line of work but focus on understanding the usefulness of incorporating &#34;error-correction&#34; data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. Using a synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount of error-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed on the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others." />
</head>

<body  class="with-cu-identity">
    <aside class="slider-wrapper bps-banner forum green">
    <a class="close-slider bps-banner" href="#"><img src="/static/browse/0.3.4/images/icons/close-slider.png" alt="close this message"></a>
    <div class="columns">
      <img role="presentation" class="bps-banner-image" src="/static/browse/0.3.4/images/icons/smileybones-infinity-large.png" alt="arXiv Accessibility Forum 2024">
      <div class="copy-donation bps-banner">
        <h2>The countdown begins!</h2>
        <p>The <a href="https://accessibility2024.arxiv.org/" target="_blank">Accessibility Forum</a> starts next week! <a href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y" target="_blank">Sign up</a> and join us in September.</p>
      </div>
      <div class="amount-donation bps-banner">
        <div class="donate-cta"><a class="banner_link banner-btn-grad" target="_blank" href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y"><b>Sign Up</b></a></div>
      </div>
    </div>
  </aside>
  <div class="flex-wrap-footer">
    <header>
      <a href="#content" class="is-sr-only">Skip to main content</a>
      <!-- start desktop header -->
      <div class="columns is-vcentered is-hidden-mobile" id="cu-identity">
        <div class="column" id="cu-logo">
          <a href="https://www.cornell.edu/"><img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University" /></a>
        </div><div class="column banner-minimal forum">
            <!-- /from July 7 at 1:00 AM to Sept 2 at 1:00 AM --><p>arXiv's Accessibility Forum starts next month!</p>
            <a href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y" target="_blank">Sign Up</a>

            <!-- /from Sept 2 at 1:00 AM to Sept 13 at 11:30PM --></div><div class="column" id="support-ack">
          <span id="support-ack-url">We gratefully acknowledge support from the Simons Foundation, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and all contributors.</span>
          <a href="https://info.arxiv.org/about/donate.html" class="btn-header-donate">Donate</a>
        </div>
      </div>

      <div id="header" class="is-hidden-mobile">
<a aria-hidden="true" href="{url_path('ignore_me')}"></a>
  <div class="header-breadcrumbs is-hidden-mobile">
    <a href="/"><img src="/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg" alt="arxiv logo" style="height:40px;"/></a> <span>&gt;</span> <a href="/list/cs/recent">cs</a> <span>&gt;</span> arXiv:2408.16293
  </div>
          <div class="search-block level-right">
    <form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
      <div class="field has-addons">
        <div class="control">
          <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
          <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
        </div>
        <div class="control">
          <div class="select is-small">
            <select name="searchtype" aria-label="Field to search">
              <option value="all" selected="selected">All fields</option>
              <option value="title">Title</option>
              <option value="author">Author</option>
              <option value="abstract">Abstract</option>
              <option value="comments">Comments</option>
              <option value="journal_ref">Journal reference</option>
              <option value="acm_class">ACM classification</option>
              <option value="msc_class">MSC classification</option>
              <option value="report_num">Report number</option>
              <option value="paper_id">arXiv identifier</option>
              <option value="doi">DOI</option>
              <option value="orcid">ORCID</option>
              <option value="author_id">arXiv author ID</option>
              <option value="help">Help pages</option>
              <option value="full_text">Full text</option>
            </select>
          </div>
        </div>
        <input type="hidden" name="source" value="header">
        <button class="button is-small is-cul-darker">Search</button>
      </div>
    </form>
  </div>
     </div><!-- /end desktop header -->

      <div class="mobile-header">
        <div class="columns is-mobile">
          <div class="column logo-arxiv"><a href="https://arxiv.org/"><img src="/static/browse/0.3.4/images/arxiv-logomark-small-white.svg" alt="arXiv logo" style="height:60px;" /></a></div>
          <div class="column logo-cornell"><a href="https://www.cornell.edu/">
            <picture>
              <source media="(min-width: 501px)"
                srcset="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg  400w"
                sizes="400w" />
              <source srcset="/static/browse/0.3.4/images/icons/cu/cornell_seal_simple_black.svg 2x" />
              <img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University Logo" />
            </picture>
          </a></div>
          <div class="column nav" id="toggle-container" role="menubar">
            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-white"><title>open search</title><path d="M505 442.7L405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9 0 208 0S0 93.1 0 208s93.1 208 208 208c48.3 0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9 0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7 0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7 0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
                <div class="field has-addons">
                  <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
                  <input type="hidden" name="source" value="header">
                  <input type="hidden" name="searchtype" value="all">
                  <button class="button">GO</button>
                </div>
              </form>
            </div>

            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-white" role="menu"><title>open navigation menu</title><path d="M16 132h416c8.837 0 16-7.163 16-16V76c0-8.837-7.163-16-16-16H16C7.163 60 0 67.163 0 76v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16z"/ ></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <nav class="mobile-menu" aria-labelledby="mobilemenulabel">
                <h2 id="mobilemenulabel">quick links</h2>
                <ul>
                    <li><a href="https://arxiv.org/login">Login</a></li>
                    <li><a href="https://info.arxiv.org/help">Help Pages</a></li>
                    <li><a href="https://info.arxiv.org/about">About</a></li>
                </ul>
              </nav>
            </div>
          </div>
        </div>
      </div><!-- /end mobile-header -->
    </header>

    <main>
      <div id="content">
<!--
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
    <rdf:Description
        rdf:about="/abs/2408.16293"
        dc:identifier="/abs/2408.16293"
        dc:title="Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems"
        trackback:ping="/trackback/2408.16293" />
    </rdf:RDF>
--><div id="abs-outer">

  <div class="leftcolumn">
    <div class="subheader">
      <h1>Computer Science > Computation and Language</h1>
    </div>

    <div class="header-breadcrumbs-mobile">
      <strong>arXiv:2408.16293</strong> (cs)
    </div>
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
<div id="content-inner">
  <div id="abs">
    <div class="dateline">
  [Submitted on 29 Aug 2024]</div>
    <h1 class="title mathjax"><span class="descriptor">Title:</span>Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems</h1>
    <div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ye,+T">Tian Ye</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Xu,+Z">Zicheng Xu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Li,+Y">Yuanzhi Li</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Allen-Zhu,+Z">Zeyuan Allen-Zhu</a></div>            <div id="download-button-info" hidden>View a PDF of the paper titled Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems, by Tian Ye and 3 other authors</div>
    <a class="mobile-submission-download" href="/pdf/2408.16293">View PDF</a>
    <a class="mobile-submission-download" href="https://arxiv.org/html/2408.16293v1">HTML (experimental)</a>



    <blockquote class="abstract mathjax">
            <span class="descriptor">Abstract:</span>Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to &#34;self-correct&#34; their mistakes via multi-round prompting. In this paper, we follow this line of work but focus on understanding the usefulness of incorporating &#34;error-correction&#34; data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. Using a synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount of error-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed on the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others.
    </blockquote>

    <!--CONTEXT-->
    <div class="metatable">
      <table summary="Additional metadata">        <tr>
          <td class="tablecell label">Comments:</td>
          <td class="tablecell comments mathjax">arXiv admin note: text overlap with <a href="https://arxiv.org/abs/2407.20311" data-arxiv-id="2407.20311" class="link-https">arXiv:2407.20311</a></td>
        </tr>
<tr>
          <td class="tablecell label">Subjects:</td>
          <td class="tablecell subjects">
            <span class="primary-subject">Computation and Language (cs.CL)</span>; Artificial Intelligence (cs.AI); Machine Learning (cs.LG)</td>
        </tr><tr>
          <td class="tablecell label">Cite as:</td>
          <td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2408.16293">arXiv:2408.16293</a> [cs.CL]</span></td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxividv">(or <span class="arxivid">
              <a href="https://arxiv.org/abs/2408.16293v1">arXiv:2408.16293v1</a> [cs.CL]</span> for this version)
          </td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxivdoi">              <a href="https://doi.org/10.48550/arXiv.2408.16293" id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2408.16293</a><div class="button-and-tooltip">
              <button class="more-info" aria-describedby="more-info-desc-1">
                <svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
                <span class="visually-hidden">Focus to learn more</span>
              </button>
              <!-- tooltip description -->
              <div role="tooltip" id="more-info-desc-1">
                <span class="left-corner"></span>                  arXiv-issued DOI via DataCite (pending registration)</div>
            </div>
          </td>
        </tr></table>
    </div>
  </div>
</div>
    <div class="submission-history">
      <h2>Submission history</h2> From: Zeyuan Allen-Zhu [<a href="/show-email/81e0381a/2408.16293">view email</a>]      <br/>    <strong>[v1]</strong>
        Thu, 29 Aug 2024 06:49:20 UTC (448 KB)<br/>
</div>
  </div>
  <!--end leftcolumn-->
<div class="extra-services">    <div class="full-text">
      <a name="other"></a>
      <span class="descriptor">Full-text links:</span>
      <h2>Access Paper:</h2>
      <ul>
  <div id="download-button-info" hidden>
View a PDF of the paper titled Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems, by Tian Ye and 3 other authors</div><li><a href="/pdf/2408.16293" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2408.16293v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2408.16293" class="abs-button download-eprint">TeX Source</a></li><li><a href="/format/2408.16293" class="abs-button download-format">Other Formats</a></li></ul>
      <div class="abs-license"><a href="http://arxiv.org/licenses/nonexclusive-distrib/1.0/" title="Rights to this article">view license</a></div>
    </div>
    <!--end full-text-->    <div class="browse">
    Current browse context: <div class="current">cs.CL</div>

  <div class="prevnext">
<span class="arrow">
      <a class="abs-button prev-url" href="/prevnext?id=2408.16293&amp;function=prev&amp;context=cs.CL"
         accesskey="p" title="previous in cs.CL (accesskey p)">&lt;&nbsp;prev</a>
    </span>
    <span class="is-hidden-mobile">&nbsp; | &nbsp;</span>    <span class="arrow">
      <a class="abs-button next-url" href="/prevnext?id=2408.16293&amp;function=next&amp;context=cs.CL" accesskey="n"
         title="next in cs.CL (accesskey n)">next&nbsp;&gt;</a>
    </span><br/>
  </div><div class="list">
    <a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.CL/new">new</a>
    <span class="is-hidden-mobile"> | </span>
    <a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.CL/recent">recent</a>
    <span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.CL/2024-08">2024-08</a>
  </div><div class="abs-switch-cat">
    Change to browse by:
    <div class="switch context-change">
        <a href="/abs/2408.16293?context=cs">cs</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.16293?context=cs.AI">cs.AI</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.16293?context=cs.LG">cs.LG</a><br class="is-hidden-mobile">
    </div>
  </div>

    </div>

    <div class="extra-ref-cite">
      <h3>References &amp; Citations</h3>
      <ul>
        <li><a  class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2408.16293">NASA ADS</a></li><li><a  class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2408.16293" target="_blank" rel="noopener">Google Scholar</a></li>
        <li><a  class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2408.16293" target="_blank" rel="noopener">Semantic Scholar</a></li>
      </ul>
      <div style="clear:both;"></div>
    </div>

<div class='extra-ref-cite'>
    <a id='bib-cite-css' hidden='true' href='/static/browse/0.3.4/css/cite.css'>a</a>

    <span id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</span>
    <span id='bib-cite-loading' hidden='true'>Loading...</span>
</div>

<div id='bib-cite-modal' class='bib-modal' hidden='true'>
    <div class='bib-modal-content'>
        <div class='bib-modal-title'>
            <h2>BibTeX formatted citation</h2>
            <span class='bib-modal-close' >&times;</span>
        </div>
        <div>
            <textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
        </div>
        <div>
            <span>Data provided by: </span>
            <a id='bib-cite-source-api'></a>
        </div>
    </div>
</div><div class="bookmarks">
  <div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2408.16293&amp;description=Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems"
     title="Bookmark on BibSonomy">
    <img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
         alt="BibSonomy logo"/>
  </a>
  <a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2408.16293&amp;title=Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems"
     title="Bookmark on Reddit">
    <img src="/static/browse/0.3.4/images/icons/social/reddit.png"
         alt="Reddit logo"/>
  </a>
</div>  </div>
  <!--end extra-services-->
<!-- LABS AREA -->
<div id="labstabs">
  <div class="labstabs"><input type="radio" name="tabs" id="tabone"checked="checked">
    <label for="tabone">Bibliographic Tools</label>
    <div class="tab labs-display-bib">
      <h1>Bibliographic and Citation Tools</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input id="bibex-toggle" type="checkbox" class="lab-toggle">
              <span class="slider"></span>
              <span class="is-sr-only">Bibliographic Explorer Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
          </div>
        </div>
<div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="litmaps-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
                aria-labelledby="label-for-litmaps">
              <span class="slider"></span>
              <span class="is-sr-only">Litmaps Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
          </div>
        </div>
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="scite-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
                aria-labelledby="label-for-scite">
              <span class="slider"></span>
              <span class="is-sr-only">scite.ai Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
          </div>
        </div>
      </div>
        <div class="labs-content-placeholder labs-display" style="display: none;"></div>
        <div style="min-height: 15px" id="litmaps-open-in"></div>
        <div style="min-height: 15px" id="scite-open-in"></div>
    </div>


    <input type="radio" name="tabs" id="tabtwo">
    <label for="tabtwo">Code, Data, Media</label>
    <div class="tab">
      <h1>Code, Data and Media Associated with this Article</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input        
                id="catalyzex-toggle"
                data-script-url="/static/browse/0.3.4/js/catalyzex.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="dagshub-toggle"
                data-script-url="/static/browse/0.3.4/js/dagshub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
              <span class="slider"></span>
              <span class="is-sr-only">DagsHub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
          </div>
        </div>
  
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="gotitpub-toggle"
                data-script-url="/static/browse/0.3.4/js/gotitpub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
              <span class="slider"></span>
              <span class="is-sr-only">GotitPub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="paperwithcode-toggle"
                data-script-url="/static/browse/0.3.4/js/paperswithcode.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-pwc">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-pwc">Papers with Code</span> <em>(<a href="https://paperswithcode.com/" target="_blank">What is Papers with Code?</a>)</em>
          </div>
        </div>


        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="sciencecast-toggle"
                data-script-url="/static/browse/0.3.4/js/sciencecast.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
              <span class="slider"></span>
              <span class="is-sr-only">ScienceCast Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
          </div>
        </div>
      </div>

      <div id="catalyzex-output" style="display:none"></div>
      <div id="dagshub-output" style="display:none"></div>
      <div id="gotitpub-output" style="display:none"></div>
      <div id="pwc-output" style="display:none"></div>
      <div id="pwc-data-output" style="display:none"></div>
      <div id="sciencecast-output" style="display:none"></div>
    </div>


      <input type="radio" name="tabs" id="labstabs-demos-input">
      <label for="labstabs-demos-input" id="labstabs-demos-label">Demos</label>
      <div class="tab">
        <h1>Demos</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="replicate-toggle"
                  data-script-url="/static/browse/0.3.4/js/replicate.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
                <span class="slider"></span>
                <span class="is-sr-only">Replicate Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="spaces-toggle"
                  data-script-url="/static/browse/0.3.4/js/spaces.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="txyz-toggle"
                  data-script-url="/static/browse/0.3.4/js/txyz.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
            </div>
          </div>
        </div>
        <div id="replicate-output"></div>
        <div id="spaces-output"></div>
        <div id="txyz-output"></div>
      </div>
      <input type="radio" name="tabs" id="tabfour">
      <label for="tabfour">Related Papers</label>
      <div class="tab">
        <h1>Recommenders and Search Tools</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="influenceflower-toggle"
                data-script-url="/static/browse/0.3.4/js/influenceflower.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
                <span class="slider"></span>
                <span class="is-sr-only">Link to Influence Flower</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="connectedpapers-toggle" type="checkbox" class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
                       aria-labelledby="label-for-connected-papers" >
                <span class="slider"></span>
                <span class="is-sr-only">Connected Papers Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
                <span class="slider"></span>
                <span class="is-sr-only">Core recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
            </div>
          </div></div>
        <div id="influenceflower-output"></div>
        <div id="influenceflower-output-graph" style="display:none">
          <ul class="flower-tabs">
            <li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
          </ul>
          <div class="flower-tab-content">
            <div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
            <div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
            <div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
            <div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
          </div>
        </div>
        <div style="min-height: 15px" id="connectedpapers-output"></div>
        <div id="coreRecommenderOutput"></div>
        <div id="iarxivOutput"></div>
      </div>

      <input type="radio" name="tabs" id="tabfive">
      <label for="tabfive">
        About arXivLabs
      </label>
      <div class="tab">
        <div class="columns">
          <div class="column">
            <h1>arXivLabs: experimental projects with community collaborators</h1>
            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
            <p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
          </div>
          <div class="column is-narrow is-full-mobile">
            <p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
          </div>
        </div>
      </div>

    </div>
</div>
<!-- END LABS AREA -->
  <div class="endorsers">
    <a href="/auth/show-endorsers/2408.16293" class="endorser-who">Which authors of this paper are endorsers?</a> |
    <a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
    <span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
  </div>
  <script type="text/javascript" language="javascript">mathjaxToggle();</script>
</div>
      </div>
    </main>

    <footer style="clear: both;">
      <div class="columns is-desktop" role="navigation" aria-label="Secondary" style="margin: -0.75em -0.75em 0.75em -0.75em">
        <!-- Macro-Column 1 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/about">About</a></li>
                <li><a href="https://info.arxiv.org/help">Help</a></li>
              </ul>
            </div>
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>contact arXiv</title><desc>Click here to contact arXiv</desc><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>
                  <a href="https://info.arxiv.org/help/contact.html"> Contact</a>
                </li>
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>subscribe to arXiv mailings</title><desc>Click here to subscribe</desc><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
                  <a href="https://info.arxiv.org/help/subscribe"> Subscribe</a>
                </li>
              </ul>
            </div>
          </div>
        </div>
        <!-- End Macro-Column 1 -->
        <!-- Macro-Column 2 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/license/index.html">Copyright</a></li>
                <li><a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy Policy</a></li>
              </ul>
            </div>
            <div class="column sorry-app-links">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/web_accessibility.html">Web Accessibility Assistance</a></li>
                <li>
                  <p class="help">
                    <a class="a11y-main-link" href="https://status.arxiv.org" target="_blank">arXiv Operational Status <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 512" class="icon filter-dark_grey" role="presentation"><path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"/></svg></a><br>
                    Get status notifications via
                    <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/email/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>email</a>
                    or <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/slack/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-black" role="presentation"><path d="M94.12 315.1c0 25.9-21.16 47.06-47.06 47.06S0 341 0 315.1c0-25.9 21.16-47.06 47.06-47.06h47.06v47.06zm23.72 0c0-25.9 21.16-47.06 47.06-47.06s47.06 21.16 47.06 47.06v117.84c0 25.9-21.16 47.06-47.06 47.06s-47.06-21.16-47.06-47.06V315.1zm47.06-188.98c-25.9 0-47.06-21.16-47.06-47.06S139 32 164.9 32s47.06 21.16 47.06 47.06v47.06H164.9zm0 23.72c25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06H47.06C21.16 243.96 0 222.8 0 196.9s21.16-47.06 47.06-47.06H164.9zm188.98 47.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06h-47.06V196.9zm-23.72 0c0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06V79.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06V196.9zM283.1 385.88c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06v-47.06h47.06zm0-23.72c-25.9 0-47.06-21.16-47.06-47.06 0-25.9 21.16-47.06 47.06-47.06h117.84c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06H283.1z"/></svg>slack</a>
                  </p>
                </li>
              </ul>
            </div>
          </div>
        </div> <!-- end MetaColumn 2 -->
        <!-- End Macro-Column 2 -->
      </div>
    </footer>
  </div>

  <script src="/static/base/1.0.1/js/member_acknowledgement.js"></script>

</body>

</html>
2408.16293
 Language models have demonstrated remarkable performance in solving reasoning tasks; however, even the strongest models still occasionally make reasoning mistakes. Recently, there has been active research aimed at improving reasoning accuracy, particularly by using pretrained language models to "self-correct" their mistakes via multi-round prompting. In this paper, we follow this line of work but focus on understanding the usefulness of incorporating "errorcorrection" data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. Using a synthetic math dataset, we show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy directly (i.e., through simple auto-regression, without multi-round prompting) compared to pretraining on the same amount of error-free data. We also delve into many details, such as (1) how this approach differs from beam search, (2) how such data can be prepared, (3) whether masking is needed on the erroneous tokens, (4) the amount of error required, (5) whether such data can be deferred to the fine-tuning stage, and many others. 
abstract
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Abstract
2408.16293
 Language models have achieved near-human-level performance in various tasks, including math solving, coding, and natural language understanding [1, 2, 18, 21, 31]. However, their problemsolving skills are still imperfect, sometimes resulting in logical errors. Recently, there have been numerous attempts to improve the reasoning accuracy of language models.  One promising approach is to use a verifier to check the correctness of the language model's output [9, 12, 23, 28, 30]. Interestingly, some studies show that language models can "selfverify" [15, 27]: They can be prompted to verify the correctness of their own generation, thereby improving overall accuracy. An illustrative example is shown in Figure 1.  This leads to the following fundamental questions: If a language model can correct its own mistakes after generation, (1) Why does it make those mistakes to begin with? (2) Why doesn't it correct the mistakes immediately during generation, instead of waiting until after?  There are many works that attempt to understand question (1). Notably, studies such as [6, 17, 22] have shown that many mistakes arise due to "distribution shift" - the training distribution of the language model differs from the prompts used during testing. Thus, even if the training data is error-free, language models can still make mistakes during generation.  Much less work focuses on question (2). While correcting mistakes after generation is a valid approach to improve a language model's accuracy, it is more desirable to correct mistakes immediately as they occur, such as "A ⇒ B, oh I made a mistake, actually A ⇒ C." Doing so during generation can save inference tokens (the model does not need to continue generating based  ![1_image_0.png](1_image_0.png)  on an erroneous step) and simplify the inference framework (eliminating the need to call the model multiple times to verify and correct the answer), see Figure 1. Thus, can we train language models to directly perform (2)?  A natural conjecture for why current language models cannot perform (2) is an alignment issue: There might be little to no training data in the language model's corpus to teach it to correct errors immediately. After all, people are unlikely to write an erroneous sentence followed by immediate correction in internet articles.1 However, even if we are given enough such "retry" training data (i.e., error + immediate correction such as "A ⇒ B, oh I made a mistake, actually A ⇒ C"), is it clear that the language model can improve its overall reasoning correctness?  - Potential harm of next-token prediction on data with mistakes: Training on data with mistakes can be dangerous. By minimizing the next-token prediction objective, the model might learn to generate mistakes (such as "A ⇒ B") instead of writing the correct steps (i.e., A ⇒ C). One could try to mitigate this by masking out the erroneous steps. However, unlike the "error correction after entire generation" type of training data where we can simply mask out the entire generation,2 here, the error and correction steps are intertwined, making it much harder to mask out the error only. - Unclear benefits over training on "perfectly error-free" data: Intuitively, if our final goal is to have model output correct reasoning steps to improve the reasoning accuracy, then why would training on "A ⇒ B, oh I made a mistake, actually A ⇒ C" be better than training directly on the correct step "A ⇒ C"? Introducing errors is a distribution shift compared to what we want the model to generate during test time (i.e. a solution without error). Moreover, if we mask out the error part [A ⇒ B, oh I made a mistake, actually], would it just reduce to training on perfectly correct data, with the model simply learning A ⇒ C?  In this paper, we provide initial results towards understanding the usefulness of including training data in the language model's corpus that teaches it to correct errors immediately. Specifically, we ask the following question: Can training on retry data (errors and immediate corrections) successfully teach the language model to perform error correction? Can a language model achieve higher reasoning accuracy compared to training on the same amount of perfectly error-free data?  discuss the dataset and provide examples in Section 2 to make this paper self-contained. Due to the generation procedure, we can also easily construct erroneous steps (simulating the reasoning mistakes that GPT-4 can make on such problems) and make 100% correct corrections to those steps. We choose this setting because mathematical reasoning errors are among the most widely observed errors made by language models. We outline our results in the following section. Section 3: retry upon regret. We begin with a warmup result. After pretraining the model on perfectly *error-free* math data, one can finetune it with a small number of trainable parameters (e.g., a rank-8 update on its embedding layer) to detect errors in its solution steps. This finetuning process is remarkably lightweight and essentially indicates that the model "already knows" it has made a mistake (i.e., its internal states exhibit regretful behavior).4 Next, we consider a "retry upon regret" generation process. During the inference phase, if the model detects an error in its generated solution step, it will "regenerate" from the end of its previous sentence. We demonstrate that this process leads to improved accuracy on this math dataset, surpassing what can be achieved using beam search. Remark 1.1. This should not be confused with the "self-correction" works [11, 15, 19], where the model evaluates its own output through careful prompting. Here, we finetune the pretrained model for error detection (> 99% accuracy), and this error detection accuracy is significantly higher than that achieved through prompting.  While this provides strong evidence that language models "regret" their mistakes, improving final accuracy relies solely on randomness (for re-generation) to correct errors - similar to beam search.5 This can take many rounds to re-generate, which is inefficient and requires the error detector to be highly accurate. It also alters the generation process, which might not align with the "general intelligence" framework, where the goal is typically for one model to perform all tasks using the same (autoregressive) decoding algorithm.  Section 4: pretrain with retry data. We now turn our attention to *retry data*. If the pretraining data includes errors and their immediate corrections (which we call "retry data"), the model can learn to not only detect errors but also correct them, achieving much higher accuracy compared to "retry upon regret" in Section 3. This differs from the "self-correction" line of work: the model learns to *automatically retry* after detecting an error, without requiring additional prompting or multi-stage generation.  Furthermore, we discover that even when the model is pretrained entirely on retry data with high error rate (e.g., for the iGSM data, p = 20% or even p = 50%, meaning roughly half of the solution steps have an inserted error), it does not tend to produce erroneous steps during generation. The model still strives to generate "perfectly correct" solutions most of the time and only corrects itself on the rare occasions when it makes a mistake, leading to an overall improvement in accuracy. In fact, within a reasonable range, the higher the p value in the pretraining data, the better the model performs in reasoning accuracy.6 We also observe that it is not necessary to perform label masking on the errors, so the vanilla autoregressive (causal masking) training simply works. Section 5: finetune with retry data. In contrast, if a model is already pretrained with error-free data, even using sufficiently many (additional) retry data, with a wide range of LoRA finetuning configurations, label masking or not, the model's accuracy does not significantly improve.  This indicates that the skill of error correction can be very different from the original error-free reasoning, and thus requires major weight changes, potentially beyond what parameter-efficient fine-tuning (PEFT) can handle. For comparison, we show that full finetuning is effective when sufficiently many retry data is available, though this resembles continued pretraining.  Thus, unlike error detection (see Section 3, where even a rank-8 adaptation on the embedding layer suffices), error correction is not a skill that can be easily adapted from a model pretrained with only error-free data. This implies that retry data should be included in the pretraining stage for practical LLM training, rather than in the finetuning (alignment) stage.  Section 6: prepare fake retry data. Retry data can be difficult to obtain, so we explore practical methods to automatically augment correct math solutions with "fake" mistakes, ideally without needing to semantically parse or understand the correct solution. The most effective method we found is to introduce a random future step B as a "fake error" at each step A in the solution, followed by A as its "correction." This approach encourages the model not to skip steps, even though some solution steps may be interchangeable in order so these may not be true errors. In the synthetic iGSM setting, this method is nearly as effective as retry data with perfectly designed errors and corrections, and it could potentially be adapted to real-life math problems. Conclusion. The full conclusion is deferred to Section 7. By utilizing fully-controllable synthetic data (e.g., controlling error rates or label masking), conducting controlled experiments (e.g., beam search vs. retry vs. error-free; pretrain vs. finetune), and performing fair comparisons (e.g., same number of training tokens), the goal of this paper is to try to predict the needs of future LLMs. We do not claim that the synthetic data used here can directly aid in building future LLMs. However, given that commercial LLMs already employ synthetic data [16, 25] and future LLMs are rumored to use Q⋆, it is perhaps crucial to understand how to best prepare and use such data effectively to teach models to learn from the mistakes. 
introduction
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Introduction
2408.16293
 Ye et al. [29] introduced a family of controllable, synthetic datasets of math problems with stepby-step solutions. These data simulate GSM8K [9], while removing arithmetic difficulties (by restricting computations to integers modulo 23) and common sense knowledge (e.g., a candle burns and its length decreases). What remains is the "logic reasoning" part. The dataset has much larger diversity (over 90 trillion solution templates), and the solutions are fully verifiable. We briefly summarize it to make the paper self-contained, emphasizing some important aspects.  An example from their dataset is in Figure 2. The structure graph describes the set of *instance* parameters, such as "the number of school daypacks in each film studio." They also allow for abstract parameters, such as "the (total) number of backpacks in central high," which requires hierarchical computation.7 The exact data construction is not important for this paper. What matters is that the parameters form a *dependency graph*, as shown in Figure 2, where a parameter can be computed only when its predecessors have all been computed. To remove arithmetic difficulty, the computations are broken into binary operations - such as 12 + 13 + 7 is broken into (12 + 13) + 7— ensuring that failure to solve the problems is not due to arithmetic difficulties. They use op to denote the number of operations needed in the solution and prepared four families of data: 7In this example, it equals A×(B1 +B2) for A = central high's number of film studios, B1, B2 = each film studio's number of school daypacks / messenger backpacks.  (Problem) The number of each Riverview High's Film Studio equals 5 times as much as the sum of each Film Studio's Backpack and each Dance Studio's School Daypack. The number of each Film Studio's School Daypack equals 12 more than the sum of each Film Studio's Messenger Backpack and each Central High's Film Studio. The number of each Central High's Film Studio equals the sum of each Dance Studio's School Daypack and each Film Studio's Messenger Backpack. The number of each Riverview High's Dance Studio equals the sum of each Film Studio's Backpack, each Film Studio's Messenger Backpack, each Film Studio's School Daypack and each Central High's Backpack. The number of each Dance Studio's School Daypack equals 17. The number of each Film Studio's Messenger Backpack equals 13. How many Backpack does Central High have?  (Solution) Define Dance Studio's School Daypack as p; so p = 17. Define Film Studio's Messenger Backpack as W; so W = 13. Define Central High's Film Studio as B; so B = p + W = 17 + 13 = 7. Define Film Studio's School Daypack as g; R = W + B = 13 + 7 = 20; so g = 12 + R = 12 + 20 = 9. Define Film Studio's Backpack as w; so w = g + W = 9 + 13 = 22.  Define Central High's Backpack as c; so c = B * w = 7 * 22 = 16. *Answer: 16.*  ![5_image_0.png](5_image_0.png)  - iGSM-med*pq/qp* uses op ≤ 15 for train; op ∈ {20, 21, 22, 23} for OOD (out-of-distribution) test.  - iGSM-hard*pq/qp* uses op ≤ 21 for training and op ∈ {28, 29, 30, 31, 32} for OOD testing.  Here, pq denotes the problem description comes before the question, and qp otherwise. They also introduce reask data for evaluation purposes: for instance, iGSM-medop=20,reask pq is a dataset constructed by first creating iGSM-medop=20 pq and then re-sampling a parameter to query - this greatly changes their math data distribution, making it a good candidate for OOD evaluation.  (After reask, the problem's op value may change.) They showed that GPT-4/GPT-4o cannot solve such problems for op > 10 (even with few-shot learning and their best efforts to remove English and arithmetic difficulties), indicating that the datasets are of some non-trivial difficulty. For completeness, we include in Figure 9 an example with op = 21 to illustrate that these problems require non-trivial reasoning even for humans. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Synthetic Math Data From Prior Work
2408.16293
 Generative models solve math problems step by step in a chain-of-thought (CoT) manner. Each step in our math problems is a single sentence formatted as "Define [param] as X; so...", as shown in Figure 2. How do generative models make mistakes in this CoT process?  The most *common* reasoning mistake8 occurs when the model generates a [param] that is not yet ready for computation (i.e., the model has not determined the values of all the parameters that [param] depends on, also known as "skipping steps", a frequent error even in GPT-4 [8]).  For example, in Figure 2, if the model generates "Define Central High's Film Studio" as the first few words in its solution, it cannot go back and erase this sentence, leading to a mistake. Ye et al. [29] confirmed that not only do language models pretrained using the iGSM datasets make mistakes in this manner, but even GPT-4/GPT-4o make such mistakes (when using few-shot). It's worth noting that the failure example in Figure 1 is also in this spirit. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Result 0-1: Language Models Can Retry Upon Regret
2408.16293
 Result 0 (corollary of [29]). For models pretrained on iGSM *(with correct solutions only!), during* their solution generation process, after writing "Define [param] as" for a wrong [param], they often "realize" such a mistake, showing a *regretful pattern in their internal states*.  To see this, one can apply their probing technique (illustrated in Figure 3(a)) to extract information from the model's last hidden layer after "Define [param A] as" to see if the model knows A can truly be computed next. This probing task is denoted as can next(A) ∈ {true, false}. They found:  - When A ranges over all possible parameters, the probing 99% accurately predicts can next(A), meaning the model knows if A can be computed next, even for the hardest op = 32 problems.  - When the model makes a mistake, the first sentence with a mistake usually has can next(A) = false. Probing shows the model has ∼ 60% chance of knowing can next(A) = false, indicating it often knows it has made a mistake, *right after* stating the parameter name in full.9 The statistical difference between the two cases signifies that the model's internal states do exhibit a "regretful" pattern, which can be detected via probing such as can next, which is almost just a linear classifier on top of its hidden states.10 In other words, error detection is easy and is a skill almost already embedded within the model's internal states, even when pretrained on correct math problems only. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Result 0: Models Can Be "Regretful" After Making Mistakes Interestingly, Their Same Paper Also Implies The Following:
2408.16293
 If a model knows it is a mistake, why does it generate the wrong [param A] in the first place?  The issue lies in the generation process. *Before explicitly stating* "Define [param A] as", the model might falsely think A is ready to compute among *all the parameters* it can focus on. After stating it, the model shifts its focus to the actual computation of A, and this is the moment it can better realize that A is not ready for computation (using its attention mechanism).11 Now that we know the model exhibits some "regret" towards the mistake, can we use this to improve accuracy?  Predict can_next(A) ∈ **{True, False}** [BOS] The number of each Riverview High ... *How many Backpack …?* [SOL] Define … Define Central High's Film Studio as B; so B = p + W = 17 + 13 = 7. Define **Film Studio's School Daypack** as decoder layer (attention + MLP) …decoder layer (attention + MLP) problem *question* a prefix of solution some **parameter A** (a) can next probing [29]. After pretraining, V-probing technique can detect if the model's internal states exhibit a regretful pattern: right after "Define param A as," the model no longer thinks A is ready for compute.  ![7_image_0.png](7_image_0.png)  (b) Retry upon regret vs. original accuracies. Version1/2 uses the can next probing to detect regret (see Section 3.2, they can detect errors to 99% accuracy), and versionP uses a perfect error detector to decide when to retry.  Figure 3: The can next probing and using it to assist the model's generation process.  call this method "retry upon regret", and we carefully compare a few different versions (details are in Appendix D.2).  - The original can next probing [29] freezes the pretrained model and fine-tunes a rank-8 update on the embedding layer - we call this "version1", and it can 99% accurately detect errors.13 - We additionally consider a different can next probing that fine-tunes more parameters but with only 0.25% data - we call this "version2", and it can also 99% accurately detect errors.  - For comparison, we also consider the perfect, ground-truth can next error detector and call this "versionP"; and we use "versionP50" to indicate using a maximum of 50 total retries. The goal of *version1* is to check if we can use extremely small parameter changes from the pretrained model for error detection and use this towards higher accuracies. The goal of *version2* is to check if we can use a small number of fine-tuning samples for error detection. The goal of versionP/P50 is for comparison purposes; after all, allowing perfect error detection can enable the model to correct nearly all mistakes if retried infinitely.  We compared this with the original generation (with beam=1, 4, 16, and 32); and we used beam=1 or 4 for all versions of "retry upon regret." The results give rise to the following: Result 1 (Figure 3(b)). The idea of "retry upon regret" can improve reasoning accuracy, to an extent better than beam search. However, the accuracy gain can be limited unless the error detector is highly accurate. This also alters the inference process and increases the inference complexity.  We explain the three sentences in detail below: (Solution - retry rate 0.5) Define Dance Studio's School Daypack as p; so p = 17. Define Film Studio's School Daypack as [BACK]. Define Film Studio's Messenger Backpack as W; so W = 13. Define Central High's Classroom as [BACK]. Define Central High's Backpack as [BACK]. Define Central High's Film Studio as B; so B = p + W = 17 + 13 = 7. Define Film Studio's School Daypack as g; R = W + B = 13 + 7 = 20; so g = 12 + R = 12 + 20 = 9. Define Riverview High's Dance Studio as [BACK]. Define Film Studio's Backpack as w; so w = g + W = 9 + 13 = 22. Define Riverview High's Dance Studio as [BACK]. Define Central High's Backpack as c; so c = B * w = 7 * 22 = 16.  (a) A solution example identical to Figure 2 but with retry rate = 0.5. The strikethrough like "Define Central High's Backpack as" is for illustration purpose, and the actual data is normal English text without strikethrough symbols.  ![8_image_0.png](8_image_0.png)  Figure 4: Pretrain language models on error-free vs retry data. Observation: especially on the hardest tasks (op = 23 or 32), models pretrained from retry data exhibit the greatest improvements for larger retry rate.  - Comparing version1/2 with beam32, we see "retry upon regret" improves upon beam search.  However, even though error detection is 99% accurate, this improvement is still marginal: about 2% for the op = 23 (resp. op = 32) case for iGSM-med (resp. iGSM-hard).  - Comparing version1/2 with versionP, we see that the success of "retry upon regret" largely depends on an extremely accurate error detector - increasing the error detection success rate from 99% to 100% can significantly improve the final reasoning accuracy, but this is too ideal.14 - The idea of "retry upon regret" increases the inference complexity because one needs to keep an error detector model alongside and keep checking the correctness of the generated solution steps. In the event of an error, the model needs to regenerate using randomness (possibly multiple times) until it passes the error detector. Ideally, one wishes to have just a single model to achieve "general intelligence" using the simplest autoregressive decoding algorithm, without multi-round error corrections. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Result 1: Let Models Retry Upon Regret
2408.16293
 In this section, we prepare pretrain data to teach the model to directly correct mistakes. Math data with retry. Since we use a controllable, synthetic math dataset, we can, at the beginning of each solution sentence, with probability retry rate ∈ [0, 1), insert a wrong parameter that cannot be computed next, followed by a special token [BACK].15 We repeat this process, so with probability (retry rate) 2, it may generate another wrong parameter at the same location, and so on. We provide an extreme example with retry rate = 0.5 in Figure 4(a), and a more complex  ![9_image_0.png](9_image_0.png)  example with op = 21 and retry rate = 0.2 in Figure 9. We call this the retry data (with error and correction), to distinguish that from the original (error-free) data.  - *Can a language model be pretrained using retry data to improve its reasoning accuracy compared* with language models pretrained on the error-free data?  We conduct experiments on the iGSM-med*pq/qp* and iGSM-hard*pq/qp* datasets for retry rate ranging from 0.01 to 0.5. For a controlled comparison, we compare models when they are pretrained using the *same number of tokens* (along with other parameters).16 This means, when pretrained with retry data, the model sees *fewer* math problems compared to the original training; when pretrained with larger retry rate, the model also sees fewer math problems.  Another important question when pretraining with retry data is that: - *Is it necessary to introduce label masks to prevent the model from learning the mistakes?* Adding masking might make intuitive sense because we do not want the model to learn the mistakes - we only want it to learn to correct errors if needed. To answer this, we also perform controlled experiment to compare (1) standard auto-regressive pretraining and (2) pretraining with masking: adding label masks to ignore the auto-regressive loss on the wrong parameter tokens (i.e., ignoring text with strikethrough in Figure 4(a)).  Result 2-3. *Our results in Figure 4(b) strongly support that:*  - Within a reasonable range,17 the more mistakes the better. *Especially on hard problems,* such as on *iGSM-med*op=23 qp *, the accuracy jumps from 78% to 94% by using* retry rate = 0.5.  - Masking mistakes is unnecessary. We observe that it is generally not needed to introduce label masking on the error data even for large retry rate = 0.5.  ![10_image_0.png](10_image_0.png)  data is full of mistakes (even with retry rate = 0.5), doesn't this interfere with the model's learning and encourage it to make lots of mistakes? To address this, we compute the statistics on how many times the pretrained model performs retries (i.e., how many [BACK] tokens it uses) during evaluation. We discover that: Result 4 (Figure 5). *Models pretrained on retry data hardly retry (unless* retry rate *is very high).* For instance, Figure 5(a) shows if retry rate = 0.2, even when pretrained without label masking, the model retries an average of < 0.3 times even for math problems with large op. This is because at each solution step, the retry data still has a higher 1 − 0.2 = 0.8 chance to produce an error-free step. Thus, a model pretrained from such data is still incentivized to generate correct steps (e.g., using a low temperature).18 If retry rate = 0.5, this average retry count becomes 2 ∼ 4, but can be mitigated using label masking. Only on those wrong solutions does the model try its best to retry and fail in the end, see Figure 5(b).  Another concern with the retry data is whether the model can still output shortest solutions.  Ye et al. [29] discovered that language models can learn to produce shortest solutions without computing any unnecessary parameters.19 Does the introduction of math data with errors increase the solution length? We discover that: Result 5 (Figure 6). *Models pretrained on retry data can still output shortest math solutions.* Combining Results 2-5, we see that it is safe to include retry data of this format as pretrain data to improve the model's reasoning accuracy. There is no change to the pretrain or inference process, no need to increase training time, and the solutions still satisfy many desirable properties.  Finally, it is tempting to compare retry data with beam search, which lets the model generate multiple solutions at each step and pick the most likely continuation based on the log-likelihood. As long as the model is pretrained on error-free data, even with 16 or 32 beams, its accuracy does not noticeable improve in all the cases (recall Figure 3(b)); while in contrast, if the model is pretrained with retry data, the accuracy easily goes up in Figure 4(b) (and this holds even for beam=1).20 Similarly, "retry upon regret" lets the model re-generate another solution step if error is detected, and this for instance gives only accuracy gain 78% ⇒ 80% in the iGSM-medop=23 pq case (recall Figure 3(b)), while pretrain with retry data can give 78% ⇒ 95%. This is strong signal that: Result 6. Error correction is a skill that can be fundamentally different *from beam search or* retry based on the model's randomness.  Therefore, to truly improve the model's reasoning capability, it is crucial to modify the training data to include mistakes and their corrections. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Result 2-6: Pretrain With Retry Data
2408.16293
 In this section, we consider a model *pretrained* using only error-free math data but *finetuned* with the retry data from Section 4. This simulates a real-world scenario where one is given an opensource pretrained model and wants to finetune it for better reasoning/math accuracies. Our goal is to determine if this approach works as well as pretraining directly on the retry data.  We focus on parameter-efficient fine-tuning (PEFT) methods such as LoRA [10], which are widely adopted in practice. LoRA fine-tunes a small number of trainable parameters (i.e., low-rank matrix updates) on top of the original, frozen pretrained weights.  Interestingly, despite using the same high-quality retry data as in Section 4 (with or without label masking), and our best efforts in selecting LoRA fine-tune parameters (adopting various rank choices), and ensuring sufficient training (comparable training steps/samples to pretraining), we find that LoRA finetuning falls short of pretraining directly with the retry data. When the LoRA rank is small, it even underperforms compared to pretraining with error-free data, see Figure 7. From this, we conclude that: Result 7 (Figure 7). *Error correction is a skill that can be very different from the original (errorfree) reasoning and cannot be acquired during a LoRA finetune stage from language models pretrained only using error-free data.* (In contrast, such a skill can be learned using full finetuning with sufficiently many retry training samples, see Figure 7; but the finetune cost of this process is no less than pretraining with retry data, and is essentially continued pretraining.21) One may compare this to error detection (see Section 3.1): error detection is an easier skill that even models pretrained from error-free data can acquire almost for free (such as via probing, not to mention LoRA finetuning). However, error correction cannot be easily achieved via LoRA finetuning from a language model pretrained on error-free data.  We conjecture this is because, when a mistake is made, the model needs to revise its internal computations to find alternative solutions. Such revision may not be straightforward; otherwise, simply re-generating a few times from the previous sentence (i.e., "retry upon regret") should have already achieved higher reasoning accuracies. We do not analyze such internal computations in this paper, but refer interested readers to [29], which explains language models' internal computations (via probing) on the error-free iGSM data.  ![12_image_0.png](12_image_0.png)  Figure 7: Pretrain vs. finetune on retry data. Details and more results (for retry rate ̸= 0.2) are in Appendix D.3.  The original and retry (pretrain) rows are pretraining with error-free / retry data (same as Figure 4(b)).  The retry (finetune, lora) rows correspond to LoRA finetuning using the retry data on a model pretrained with error-free data. qv stands for the LoRA rank on the query/value matrices, and e stands for the LoRA rank on the embedding matrix. Observation: No significant improvement over the original model; for small LoRA ranks, finetuning even hurts and label masking becomes important.  The retry (continued pretrain) rows correspond to full finetuning using the retry data on a model pretrained with error-free data. Note: this is for illustration only. Our full finetuning (+ pretrain) uses twice the training tokens compared to retry (pretrain), so it is not surprising that accuracies can be higher. We compare this to directly pretraining with twice the retry data tokens in Figure 10.  Conclusion: Error correction is a skill very different from the original (error-free) reasoning and may not be acquired during a LoRA finetune stage, even with a sufficient number of finetune (retry) samples. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Result 7: Finetune With Retry Data
2408.16293
 While it is possible to prepare "perfect" retry data on synthetic iGSM datasets, obtaining math data with mistakes and corrections can be challenging in real life. For this reason, we explore the possibility of using more realistic approaches to augment math problems (for which we only have correct solutions) with "fake" mistakes and retries. Ideally, this process should not require any semantic parsing or understanding of the problem and the solution.  We explore two approaches and compare them with the perfect retry data. We still use retry rate, but instead of selecting a wrong parameter to retry, we simplify the process: - In the first approach, we randomly select a sentence that appears *later* in the step-by-step solution to retry. For instance, in the example of Figure 2: (Solution - retry **weak**) Define Dance Studio's School Daypack as p; so p = 17. Define Film Studio's Messenger Backpack as W; so W = 13. **Define** ♢♢♢ **as [BACK].** Define Central High's Film Studio as B; so B = p + W = 17 + 13 = 7. Define Film Studio's School Daypack as g; R = W + B = 13 + 7 = 20; so g = 12 + R = 12 + 20 = 9.  Define Film Studio's Backpack as w; so w = g + W = 9 + 13 = 22. Define Central High's Backpack as c; so c = B * w = 7 * 22 = 16. *Answer: 16.*(6.1) At the ♢♢♢ position, we randomly select one of the three underlined parameters and add it with [BACK] for retry.22 At the end of each sentence, we add at least one such retry parameter with retry rate probability, at least two with (retry rate) 2 probability, and so on.  This approach introduces *fake* mistakes because a parameter in a later sentence might still be 22Alternatively, one can also insert the entire sentence (as opposed to just the parameter name) for retry.  ![13_image_0.png](13_image_0.png)  computable at the current position. However, it is very easy to implement: one simply needs to select a future sentence and append it with a [BACK] symbol. Intuitively, it encourages the model *not to skip steps*. We call this retry **weak** data.  - In the second approach, we randomly select a parameter that appears in the problem statement and has not yet appeared in the solution. For instance in (6.1), at location ♢♢♢, one can add not only one of the underlined parameters but also any other parameter from the problem that hasn't appeared before (such as Riverview High's Film Studio). We call this retry **miss** data. This type of data is harder to obtain than retry weak but easier than the perfect retry data. Intuitively, it encourages the model not only "not to skip steps" but also not to compute unnecessary parameters. Our results are in Figure 8 (cf. Figure 4(b) for the perfect retry data); to summarize: Result 8. The realistic, simple-to-obtain retry weak data significantly improve the model's accuracy; yet, the slightly more complex retry miss *data does not improve accuracy by much.* (Additionally, Figure 11(a) in Appendix C shows that using retry weak/retry miss data, the model has a higher retry rate at inference time compared to Figure 5(a); Figure 11(b) shows in both cases, the model still learns to find shortest solutions, similar to Figure 6 on perfect retry data.) Please note, while our experiments are on synthetic data, we aim to use controlled experiments to predict what could be the important data changes that can help improve LLMs' reasoning capabilities in real life. While future LLMs may not be trained directly on such retry weak data, our results suggest that it can be beneficial to, for instance, use auxiliary models to rewrite math data to include fake mistakes of this type. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Result 8: Pretrain With Fake Mistakes
2408.16293
 In this paper, we investigate whether language models can benefit from pretraining on data containing mistakes, followed by immediate error correction. Using a fully controllable synthetic setting, we demonstrate that models trained on such data outperform those trained on the same amount of error-free data.  In addition to the accuracy gain, Section 4 shows that using retry data is very safe: the model rarely makes mistakes even after pretraining with high error-rate retry data, and it is unnecessary to change the training process (simply autoregressive, no need to label-mask the errors). Retry data teaches models how to correct errors if needed, rather than encouraging mistakes.  It is important to note that such *error correction skill* does not come easily. A model pretrained with only error-free data cannot use (1) beam search or (2) retry based on error detection ("retry upon regret") to achieve comparable performance, see Section 3, unless the error detection is nearly perfect. This error correction skill is also very different from the original error-free reasoning and thus cannot be learned during parameter-efficient fine-tuning (PEFT) such as LoRA, see Section 5.  This implies the necessity of adding retry data to the pretrain data for language models to truly learn the capability to correct errors.  While grade-school level math problems have many other difficulties (including arithmetic or common sense), following [29], we have focused on the (logic-following) reasoning aspect, which is one of the weakest aspects of GPT-4.23 While it is unlikely that iGSM retry data will be directly used for pretraining future commerciallevel LLMs, this paper aims to find guiding principles for necessary ingredients. We strongly discourage using fine-tuning to teach a model to correct errors (or using beam search, or letting the model regenerate upon encountering a mistake) as these are not effective. We advocate for adding mistakes and corrections at the pretrain level. While commercial LLMs use synthetic data [16, 25] and future LLMs are rumored to use Q⋆, it remains a question of how to prepare such synthetic data for the model to best learn error correction. Our Section 6 suggests that it is critical to teach a model not to skip steps. This can be done either through naively creating retry weak data like ours or using more advanced prompting methods to encourage an auxiliary model to rewrite math data into such a format. We cannot explore such follow-up directions due to GPU resource limitations.  Finally, Part 2 of this work series focuses on how language models solve grade-school math level reasoning problems (including Part 2.1 [29]). We also cover how language models learn language structures in Part 1 [2] and learn world knowledge in Part 3 [3–5]. 
conclusion
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Conclusion
2408.16293
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
Appendix
2408.16293
 (Problem) The number of each Jungle Jim's International Market's Cheese equals the sum of each Parmesan Cheese's Pear and each The Fresh Market's Ice Cream. The number of each Ice Cream's Pineapple equals 2 more than each Goat Cheese's Grape. The number of each New Seasons Market's Goat Cheese equals the sum of each Residential College District's Jungle Jim's International Market, each Jungle Jim's International Market's Parmesan Cheese and each Residential College District's Supermarket. The number of each Arts Campus's New Seasons Market equals each Cheese's Pineapple. The number of each Goat Cheese's Banana equals each Vocational School District's Product. The number of each Residential College District's Jungle Jim's International Market equals 5 more than each Ice Cream's Grape. The number of each Parmesan Cheese's Pineapple equals each Parmesan Cheese's Pear. The number of each Residential College District's The Fresh Market equals each Arts Campus's Trader Joe's. The number of each Arts Campus's Trader Joe's equals each Parmesan Cheese's Ingredient. The number of each Goat Cheese's Grape equals 0. The number of each The Fresh Market's Ice Cream equals 13 more than the difference of each Residential College District's The Fresh Market and each Parmesan Cheese's Grape. The number of each Goat Cheese's Pineapple equals each New Seasons Market's Product. The number of each Vocational School District's The Fresh Market equals the sum of each Trader Joe's's Cheese and each The Fresh Market's Cheese. The number of each Trader Joe's's Cheese equals 6. The number of each The Fresh Market's Cheese equals 3. The number of each Jungle Jim's International Market's Ice Cream equals the difference of each Ice Cream's Banana and each Goat Cheese's Grape. The number of each Jungle Jim's International Market's Parmesan Cheese equals each Ice Cream's Pineapple. The number of each Parmesan Cheese's Pear equals the difference of each Goat Cheese's Grape and each Ice Cream's Grape. The number of each Parmesan Cheese's Grape equals 12 times as much as each Residential College District's Jungle Jim's International Market. The number of each The Fresh Market's Parmesan Cheese equals each The Fresh Market's Cheese. The number of each Ice Cream's Banana equals the sum of each Parmesan Cheese's Pineapple and each Ice Cream's Pineapple. The number of each School District's Jungle Jim's International Market equals each The Fresh Market's Ice Cream. The number of each Cheese's Pineapple equals 20 more than the sum of each Trader Joe's's Cheese and each The Fresh Market's Cheese. The number of each Trader Joe's's Parmesan Cheese equals 16. The number of each Ice Cream's Pear equals 8. The number of each Ice Cream's Grape equals each Goat Cheese's Grape. *How many Product does School District have?* (Solution) Define Goat Cheese's Grape as u; so u = 0. Define Ice Cream's Grape as x; so x = u = 0. Define Residential College District's Jungle Jim's International Market as N; so N = 5 + x = 5 + 0 = 5. Define Parmesan Cheese's Pear as G; so G = u - x = 0 - 0 = 0. Define Parmesan Cheese's Grape as f; so f = 12 * N = 12 * 5 = 14. Define Parmesan Cheese's Pineapple as C; so C = G = 0. Define Parmesan Cheese's Ingredient as Z; e = f + C = 14 + 0 = 14; so Z = e + G = 14 + 0 = 14. Define Arts Campus's Trader Joe's as q; so q = Z = 14. Define Residential College District's The Fresh Market as j; so j = q = 14. Define Ice Cream's Pineapple as X; so X = 2 + u = 2 + 0 = 2. Define Ice Cream's Banana as K; so K = C + X = 0 + 2 = 2. Define The Fresh Market's Ice Cream as P; i = j - f = 14 - 14 = 0; so P = 13 + i = 13 + 0 = 13. Define Jungle Jim's International Market's Ice Cream as R; so R = K - u = 2 - 0 = 2. Define School District's Jungle Jim's International Market as V; so V = P = 13. Define Jungle Jim's International Market's Cheese as v; so v = G + P = 0 + 13 = 13. Define Jungle Jim's International Market's Parmesan Cheese as S; so S = X = 2. Define Jungle Jim's International Market's Product as y; U = S + R = 2 + 2 = 4; so y = U + v = 4 + 13 = 17. Define School District's Product as J; so J = V * y = 13 * 17 = 14. *Answer: 14.* (Solution with retry rate = 0.2) Define Arts Campus's Ingredient as [BACK]. Define Vocational School District's Supermarket as [BACK]. Define Jungle Jim's International Market's Cheese as [BACK]. Define Goat Cheese's Grape as u; so u = 0. Define Ice Cream's Grape as x; so x = u = 0.  Define Residential College District's Jungle Jim's International Market as N; so N = 5 + x = 5 + 0 = 5. Define New Seasons Market's Product as [BACK]. Define Parmesan Cheese's Pear as G; so G = u - x = 0 - 0 = 0. Define Parmesan Cheese's Grape as f; so f = 12 * N = 12 * 5 = 14. Define Parmesan Cheese's Pineapple as C; so C = G = 0. Define Parmesan Cheese's Ingredient as Z; e = f + C = 14 + 0 = 14; so Z = e + G = 14 + 0 = 14. Define Arts Campus's Trader Joe's as q; so q = Z = 14. Define Residential College District's The Fresh Market as j; so j = q = 14. Define Jungle Jim's International Market's Product as [BACK]. Define Ice Cream's Pineapple as X; so X = 2 + u = 2 + 0 = 2. Define Ice Cream's Banana as K; so K = C + X = 0 + 2 = 2. Define The Fresh Market's Ice Cream as P; i = j - f = 14 - 14 = 0; so P = 13 + i = 13 + 0 = 13. Define Jungle Jim's International Market's Ice Cream as R; so R = K - u = 2 - 0 = 2. Define Vocational School District's Supermarket as [BACK]. Define School District's Jungle Jim's International Market as V; so V = P = 13. Define New Seasons Market's Ingredient as [BACK].  Define Jungle Jim's International Market's Cheese as v; so v = G + P = 0 + 13 = 13. Define Jungle Jim's International Market's Parmesan Cheese as S; so S = X = 2. Define Jungle Jim's International Market's Product as y; U = S + R = 2 + 2 = 4; so y = U + v = 4 + 13 = 17. Define School District's Product as J; so J = V * y = 13 * 17 = 14.  ![15_image_0.png](15_image_0.png) 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
A A Harder Example With Retry
2408.16293
 We have included the finetune results only for retry rate = 0.2 in Figure 7 in Section 5 (Result 7).  In Figure 10 below, we include the additional results for retry rate ̸= 0.2.  ![16_image_0.png](16_image_0.png)   Figure 10: Pretrain vs. finetune on retry data. This figure complements Figure 7 by including additional results for retry rate ̸= 0.2. All conclusions under Figure 7 remain the same.  Additional observation 1: although rarely, retry (finetune, lora) can outperform the original model especially when retry rate = 0.5, and label masking is on, and a very high-rank LoRA finetune is used.  Additional observation 2: although retry (continued pretrain) improves upon retry (pretrain), it uses twice the number of training tokens (T tokens from retry plus T tokens from error-free data); if comparing this against retry (pretrain, double-time) which is to directly pretrain using 2T tokens from retry data, there is no significant accuracy improvement. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
B More Experiments For Finetune (Result 7)
2408.16293
 We have included the accuracy results for the retry weak or retry miss data in Figure 8 in Section 6 (Result 8). Below in Figure 11, we additionally showcase the number of retries and that the models can still generate shortest solutions most of the time.  ![17_image_0.png](17_image_0.png)  (a) Model's average number of retries per test problem (among generated correct solutions); this figure complements Figure 5(a) by including also retry weak and retry miss.  ![17_image_1.png](17_image_1.png)   (b) Model's average number of unnecessary operations or parameters per test problem (among correct solutions); this figure complements Figure 6 by including also retry weak and retry miss.  Figure 11: Average number of retries and unnecessary operations per generated (correct) solution. Details see Appendix D.1.  Observations. When pretrained with retry weak or retry miss data, the model tends to retry more frequently than the perfect retry data, especially for retry rate ≥ 0.1 (top table); yet the model does not tend to compute more unnecessary parameters than the perfect retry data (bottom table). 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
C More Experiments For Fake Retry Data (Result 8)
2408.16293
 Model. We use the GPT2 architecture [20], replacing its absolute positional embedding with modern rotary positional embedding [7, 24], still referred to as GPT2 for short. We also played with the Llama architecture (especially with gated MLP layers) and did not see any benefit of using it. This GPT2 performs comparably to Llama/Mistral at least for knowledge tasks [5].  Let GPT2-ℓ-h denote an ℓ-layer, h-head, 64h-dim GPT2 model. We primarily use GPT2-12-12 (a.k.a. GPT2-small) throughout this paper. We use the default GPT2Tokenizer, and a context length of 768/1024 for language model pretraining on iGSM-med/iGSM-hard and a context length of 2048 for evaluation.  Data Size. For both pretraining and finetuning, we did not limit the amount of training data; we generated new data on-the-fly. We do not explore sample complexity in this paper, such as the number of math problems needed to achieve a certain level of accuracy, as it would complicate the main message of this paper. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
D Experiment Details And Parameters
2408.16293
 Pretrain parameters. Throughout this paper, when we pretrained a model (except for "pretrain double-time"), we followed the same pretrain parameter choices of [29]. That is, we used the AdamW optimizer with mixed-precision fp16, β = (0.9, 0.98), cosine learning rate decay (down to 0.01x in the end + 1000 steps of linear ramp-up). We used a mixture of V100/A100 GPUs, but the GPU specifications are not relevant here.24 For all the experiments (with original data, or retry data, or retry weak, or retry miss):  - For pretraining on the iGSM-med datasets, we used a learning rate of 0.002, weight decay of 0.05, batch size of 512, context length of 768, and trained for 100, 000 steps.  - For pretraining on the iGSM-hard datasets, we used a learning rate of 0.002, weight decay of 0.03, batch size of 256, context length of 1024, and trained for 200, 000 steps. Our pretrain data is constructed by randomly generating math problems (and solutions), concatenating them together, and truncating them (in the right) to fit within the 768 or 1024-sized context window. If a problem (with solution) is longer than the context window size, we discard it.  Test-time parameters. When evaluating on test data, we use context length 2048 for both iGSM-med and iGSM-hard.  25 We use either beam=1 and dosample=false (greedy) or beam=4 and dosample=true (beam-search multinomial sampling) to present test accuracies. (Except for the original no-retry training, we also tried beam=16/32 with dosample=true.) We discover it is better to keep dosample=false while beam=1 and dosample=true while beam > 1.  Our accuracies *are not simply* from comparing the answer integers (between 0 and 22); instead we have written a parser to make sure the model's intermediate solution steps are fully-correct.  In Figure 4(b) and Figure 8, since we care about the (relatively small) accuracy differences across models, we pretrain using two different random seeds, and evaluate with both beam=1/4; we then present the best accuracies in each cell with respect to the 2 seeds and 2 beam choices.  For the retry count statistics in Figure 5(a), Figure 5(b), Figure 11(a) and the unnecessary parameter count statistics in Figure 6 and Figure 11(b), we also tested each model with 4096 math problems in each case. We then presented the statistics among the model's correct solutions or wrong solutions in these figures.  Pretrain (double-time) parameters. In Figure 10 of this appendix, we have also included experiments with pretraining for twice the number of retry data tokens. In that experiment, we have followed the same pretraining parameters, except that:  - For pretraining (double-time) on the iGSM-med datasets, we decreased weight decay to 0.03, and trained for 200, 000 steps (twice than before).  - For pretraining (double-time) on the iGSM-hard datasets, we decreased weight decay to 0.02, and trained for 400, 000 steps (twice than before). 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
D.1 Pretrain Experiment Details
2408.16293
 The V-probing for can next(A) was introduced in Ye et al. [29]. It is a *fine-tuning* process upon the pretrained language model, with an additional linear head on the output layer, and a small rank-r update on the input (embedding) layer. The pretrained model is freezed, and only this linear head and the rank-r update are trainable parameters during the fine-tuning stage (for the probing task).  In this paper, our "can next probing version1" followed the exact same parameters as [29].  That is, we use a small r = 8 so if probing accuracy is high, it mostly comes from the pretrained model and not the additional trainable parameters. We used learning rate 0.002 (with no ramp-up, linear decay down to 0), weight decay of 0.01, and a batch size of 256 for iGSM-med (or 128 for iGSM-hard) and trained for 100, 000 steps. One can calculate that in version1, this is 50% (or 25% for iGSM-hard) of the training tokens comparing to the pretrain process. That is a lot of finetune samples.26 For such reason, we also implemented a "can next probing version2", which uses more trainable parameters but significantly fewer training samples. Specifically, we choose r = 32 and additionally allow for a rank-2 update on the query/value matrices in the GPT2 model (this is 768 × 4 × 2 × 12 trainable parameters for all the 12 layers). We used batch size 8, best learning rate between {0.001, 0.0005} (same linear scheduling), weight decay 0, and finetuned for 16, 000 steps. This is only 0.25% of the training tokens comparing to the pretrain stage.  We implemented "retry upon regret" for both versions and their accuracies were presented in Figure 3(b) - once again, averaged over 4096 problems in each cell. 
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
D.2 V-Probing Experiment Details
2408.16293
 LoRA finetune. In Section 5, we applied LoRA finetuning [10] on a pretrained model using the new "error + correction" data. LoRA involves freezing the pretrained model and adding trainable low-rank updates to the weight matrices. It is recommended to apply low-rank updates to the query/value matrices [10] and the embedding matrix [4].
body
Physics Of Language Models: Part 2.2, How To Learn From Mistakes On Grade-School Math Problems
D.3 Finetune Experiment Details