<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

<head>  <title>[2408.13359] Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
  <link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
  <link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20240822" />
  <link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
  <script language="javascript" src="/static/browse/0.3.4/js/accordion.js" /></script>
  
    <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/slider.css?v=1.14" />
    <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="/static/browse/0.3.4/js/donate.js?v=1.11"></script><link rel="canonical" href="/abs/2408.13359"/><meta property="og:type" content="website" />
<meta property="og:site_name" content="arXiv.org" />
<meta property="og:title" content="Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler" />
<meta property="og:url" content="https://arxiv.org/abs/2408.13359v1" />
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="700" />
<meta property="og:image:alt" content="arXiv logo"/>
<meta property="og:description" content="Finding the optimal learning rate for language model pretraining is a challenging task. This is not only because there is a complicated correlation between learning rate, batch size, number of training tokens, model size, and other hyperparameters but also because it is prohibitively expensive to perform a hyperparameter search for large language models with Billions or Trillions of parameters. Recent studies propose using small proxy models and small corpus to perform hyperparameter searches and transposing the optimal parameters to large models and large corpus. While the zero-shot transferability is theoretically and empirically proven for model size related hyperparameters, like depth and width, the zero-shot transfer from small corpus to large corpus is underexplored. In this paper, we study the correlation between optimal learning rate, batch size, and number of training tokens for the recently proposed WSD scheduler. After thousands of small experiments, we found a power-law relationship between variables and demonstrated its transferability across model sizes. Based on the observation, we propose a new learning rate scheduler, Power scheduler, that is agnostic about the number of training tokens and batch size. The experiment shows that combining the Power scheduler with Maximum Update Parameterization (muP) can consistently achieve impressive performance with one set of hyperparameters regardless of the number of training tokens, batch size, model size, and even model architecture. Our 3B dense and MoE models trained with the Power scheduler achieve comparable performance as state-of-the-art small language models. We open-source these pretrained models at https://ibm.biz/BdKhLa."/>
<meta name="twitter:site" content="@arxiv"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Power Scheduler: A Batch Size and Token Number Agnostic Learning..."/>
<meta name="twitter:description" content="Finding the optimal learning rate for language model pretraining is a challenging task. This is not only because there is a complicated correlation between learning rate, batch size, number of..."/>
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
<meta name="twitter:image:alt" content="arXiv logo"/>
  <link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/>  <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script>  <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
  <script src="/static/browse/0.3.4/js/toggle-labs.js?20230622" type="text/javascript"></script>
  <script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler" /><meta name="citation_author" content="Shen, Yikang" /><meta name="citation_author" content="Stallone, Matthew" /><meta name="citation_author" content="Mishra, Mayank" /><meta name="citation_author" content="Zhang, Gaoyuan" /><meta name="citation_author" content="Tan, Shawn" /><meta name="citation_author" content="Prasad, Aditya" /><meta name="citation_author" content="Soria, Adriana Meza" /><meta name="citation_author" content="Cox, David D." /><meta name="citation_author" content="Panda, Rameswar" /><meta name="citation_date" content="2024/08/23" /><meta name="citation_online_date" content="2024/08/23" /><meta name="citation_pdf_url" content="http://arxiv.org/pdf/2408.13359" /><meta name="citation_arxiv_id" content="2408.13359" /><meta name="citation_abstract" content="Finding the optimal learning rate for language model pretraining is a challenging task. This is not only because there is a complicated correlation between learning rate, batch size, number of training tokens, model size, and other hyperparameters but also because it is prohibitively expensive to perform a hyperparameter search for large language models with Billions or Trillions of parameters. Recent studies propose using small proxy models and small corpus to perform hyperparameter searches and transposing the optimal parameters to large models and large corpus. While the zero-shot transferability is theoretically and empirically proven for model size related hyperparameters, like depth and width, the zero-shot transfer from small corpus to large corpus is underexplored. In this paper, we study the correlation between optimal learning rate, batch size, and number of training tokens for the recently proposed WSD scheduler. After thousands of small experiments, we found a power-law relationship between variables and demonstrated its transferability across model sizes. Based on the observation, we propose a new learning rate scheduler, Power scheduler, that is agnostic about the number of training tokens and batch size. The experiment shows that combining the Power scheduler with Maximum Update Parameterization (muP) can consistently achieve impressive performance with one set of hyperparameters regardless of the number of training tokens, batch size, model size, and even model architecture. Our 3B dense and MoE models trained with the Power scheduler achieve comparable performance as state-of-the-art small language models. We open-source these pretrained models at https://ibm.biz/BdKhLa." />
</head>

<body  class="with-cu-identity">
    <aside class="slider-wrapper bps-banner forum green">
    <a class="close-slider bps-banner" href="#"><img src="/static/browse/0.3.4/images/icons/close-slider.png" alt="close this message"></a>
    <div class="columns">
      <img role="presentation" class="bps-banner-image" src="/static/browse/0.3.4/images/icons/smileybones-infinity-large.png" alt="arXiv Accessibility Forum 2024">
      <div class="copy-donation bps-banner">
        <h2>The countdown begins!</h2>
        <p>The <a href="https://accessibility2024.arxiv.org/" target="_blank">Accessibility Forum</a> starts next week! <a href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y" target="_blank">Sign up</a> and join us in September.</p>
      </div>
      <div class="amount-donation bps-banner">
        <div class="donate-cta"><a class="banner_link banner-btn-grad" target="_blank" href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y"><b>Sign Up</b></a></div>
      </div>
    </div>
  </aside>
  <div class="flex-wrap-footer">
    <header>
      <a href="#content" class="is-sr-only">Skip to main content</a>
      <!-- start desktop header -->
      <div class="columns is-vcentered is-hidden-mobile" id="cu-identity">
        <div class="column" id="cu-logo">
          <a href="https://www.cornell.edu/"><img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University" /></a>
        </div><div class="column banner-minimal forum">
            <!-- /from July 7 at 1:00 AM to Sept 2 at 1:00 AM --><p>arXiv's Accessibility Forum starts next month!</p>
            <a href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y" target="_blank">Sign Up</a>

            <!-- /from Sept 2 at 1:00 AM to Sept 13 at 11:30PM --></div><div class="column" id="support-ack">
          <span id="support-ack-url">We gratefully acknowledge support from the Simons Foundation, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and all contributors.</span>
          <a href="https://info.arxiv.org/about/donate.html" class="btn-header-donate">Donate</a>
        </div>
      </div>

      <div id="header" class="is-hidden-mobile">
<a aria-hidden="true" href="{url_path('ignore_me')}"></a>
  <div class="header-breadcrumbs is-hidden-mobile">
    <a href="/"><img src="/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg" alt="arxiv logo" style="height:40px;"/></a> <span>&gt;</span> <a href="/list/cs/recent">cs</a> <span>&gt;</span> arXiv:2408.13359
  </div>
          <div class="search-block level-right">
    <form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
      <div class="field has-addons">
        <div class="control">
          <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
          <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
        </div>
        <div class="control">
          <div class="select is-small">
            <select name="searchtype" aria-label="Field to search">
              <option value="all" selected="selected">All fields</option>
              <option value="title">Title</option>
              <option value="author">Author</option>
              <option value="abstract">Abstract</option>
              <option value="comments">Comments</option>
              <option value="journal_ref">Journal reference</option>
              <option value="acm_class">ACM classification</option>
              <option value="msc_class">MSC classification</option>
              <option value="report_num">Report number</option>
              <option value="paper_id">arXiv identifier</option>
              <option value="doi">DOI</option>
              <option value="orcid">ORCID</option>
              <option value="author_id">arXiv author ID</option>
              <option value="help">Help pages</option>
              <option value="full_text">Full text</option>
            </select>
          </div>
        </div>
        <input type="hidden" name="source" value="header">
        <button class="button is-small is-cul-darker">Search</button>
      </div>
    </form>
  </div>
     </div><!-- /end desktop header -->

      <div class="mobile-header">
        <div class="columns is-mobile">
          <div class="column logo-arxiv"><a href="https://arxiv.org/"><img src="/static/browse/0.3.4/images/arxiv-logomark-small-white.svg" alt="arXiv logo" style="height:60px;" /></a></div>
          <div class="column logo-cornell"><a href="https://www.cornell.edu/">
            <picture>
              <source media="(min-width: 501px)"
                srcset="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg  400w"
                sizes="400w" />
              <source srcset="/static/browse/0.3.4/images/icons/cu/cornell_seal_simple_black.svg 2x" />
              <img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University Logo" />
            </picture>
          </a></div>
          <div class="column nav" id="toggle-container" role="menubar">
            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-white"><title>open search</title><path d="M505 442.7L405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9 0 208 0S0 93.1 0 208s93.1 208 208 208c48.3 0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9 0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7 0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7 0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
                <div class="field has-addons">
                  <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
                  <input type="hidden" name="source" value="header">
                  <input type="hidden" name="searchtype" value="all">
                  <button class="button">GO</button>
                </div>
              </form>
            </div>

            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-white" role="menu"><title>open navigation menu</title><path d="M16 132h416c8.837 0 16-7.163 16-16V76c0-8.837-7.163-16-16-16H16C7.163 60 0 67.163 0 76v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16z"/ ></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <nav class="mobile-menu" aria-labelledby="mobilemenulabel">
                <h2 id="mobilemenulabel">quick links</h2>
                <ul>
                    <li><a href="https://arxiv.org/login">Login</a></li>
                    <li><a href="https://info.arxiv.org/help">Help Pages</a></li>
                    <li><a href="https://info.arxiv.org/about">About</a></li>
                </ul>
              </nav>
            </div>
          </div>
        </div>
      </div><!-- /end mobile-header -->
    </header>

    <main>
      <div id="content">
<!--
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
    <rdf:Description
        rdf:about="/abs/2408.13359"
        dc:identifier="/abs/2408.13359"
        dc:title="Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler"
        trackback:ping="/trackback/2408.13359" />
    </rdf:RDF>
--><div id="abs-outer">

  <div class="leftcolumn">
    <div class="subheader">
      <h1>Computer Science > Computation and Language</h1>
    </div>

    <div class="header-breadcrumbs-mobile">
      <strong>arXiv:2408.13359</strong> (cs)
    </div>
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
<div id="content-inner">
  <div id="abs">
    <div class="dateline">
  [Submitted on 23 Aug 2024]</div>
    <h1 class="title mathjax"><span class="descriptor">Title:</span>Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler</h1>
    <div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Shen,+Y">Yikang Shen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Stallone,+M">Matthew Stallone</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Mishra,+M">Mayank Mishra</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Zhang,+G">Gaoyuan Zhang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Tan,+S">Shawn Tan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Prasad,+A">Aditya Prasad</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Soria,+A+M">Adriana Meza Soria</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Cox,+D+D">David D. Cox</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Panda,+R">Rameswar Panda</a></div>            <div id="download-button-info" hidden>View a PDF of the paper titled Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler, by Yikang Shen and 8 other authors</div>
    <a class="mobile-submission-download" href="/pdf/2408.13359">View PDF</a>
    <a class="mobile-submission-download" href="https://arxiv.org/html/2408.13359v1">HTML (experimental)</a>



    <blockquote class="abstract mathjax">
            <span class="descriptor">Abstract:</span>Finding the optimal learning rate for language model pretraining is a challenging task. This is not only because there is a complicated correlation between learning rate, batch size, number of training tokens, model size, and other hyperparameters but also because it is prohibitively expensive to perform a hyperparameter search for large language models with Billions or Trillions of parameters. Recent studies propose using small proxy models and small corpus to perform hyperparameter searches and transposing the optimal parameters to large models and large corpus. While the zero-shot transferability is theoretically and empirically proven for model size related hyperparameters, like depth and width, the zero-shot transfer from small corpus to large corpus is underexplored. In this paper, we study the correlation between optimal learning rate, batch size, and number of training tokens for the recently proposed WSD scheduler. After thousands of small experiments, we found a power-law relationship between variables and demonstrated its transferability across model sizes. Based on the observation, we propose a new learning rate scheduler, Power scheduler, that is agnostic about the number of training tokens and batch size. The experiment shows that combining the Power scheduler with Maximum Update Parameterization (muP) can consistently achieve impressive performance with one set of hyperparameters regardless of the number of training tokens, batch size, model size, and even model architecture. Our 3B dense and MoE models trained with the Power scheduler achieve comparable performance as state-of-the-art small language models. We open-source these pretrained models at <a href="https://ibm.biz/BdKhLa" rel="external noopener nofollow" class="link-external link-https">this https URL</a>.
    </blockquote>

    <!--CONTEXT-->
    <div class="metatable">
      <table summary="Additional metadata"><tr>
          <td class="tablecell label">Subjects:</td>
          <td class="tablecell subjects">
            <span class="primary-subject">Computation and Language (cs.CL)</span>; Artificial Intelligence (cs.AI); Machine Learning (cs.LG)</td>
        </tr><tr>
          <td class="tablecell label">Cite as:</td>
          <td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2408.13359">arXiv:2408.13359</a> [cs.CL]</span></td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxividv">(or <span class="arxivid">
              <a href="https://arxiv.org/abs/2408.13359v1">arXiv:2408.13359v1</a> [cs.CL]</span> for this version)
          </td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxivdoi">              <a href="https://doi.org/10.48550/arXiv.2408.13359"  id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2408.13359</a><div class="button-and-tooltip">
              <button class="more-info" aria-describedby="more-info-desc-1">
                <svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
                <span class="visually-hidden">Focus to learn more</span>
              </button>
              <!-- tooltip description -->
              <div role="tooltip" id="more-info-desc-1">
                <span class="left-corner"></span>                  arXiv-issued DOI via DataCite</div>
            </div>
          </td>
        </tr></table>
    </div>
  </div>
</div>
    <div class="submission-history">
      <h2>Submission history</h2> From: Yikang Shen Dr. [<a href="/show-email/446c41b3/2408.13359">view email</a>]      <br/>    <strong>[v1]</strong>
        Fri, 23 Aug 2024 20:22:20 UTC (483 KB)<br/>
</div>
  </div>
  <!--end leftcolumn-->
<div class="extra-services">    <div class="full-text">
      <a name="other"></a>
      <span class="descriptor">Full-text links:</span>
      <h2>Access Paper:</h2>
      <ul>
  <div id="download-button-info" hidden>
View a PDF of the paper titled Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler, by Yikang Shen and 8 other authors</div><li><a href="/pdf/2408.13359" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2408.13359v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2408.13359" class="abs-button download-eprint">TeX Source</a></li><li><a href="/format/2408.13359" class="abs-button download-format">Other Formats</a></li></ul>
      <div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
          <img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
          <span>view license</span>
        </a></div>
    </div>
    <!--end full-text-->    <div class="browse">
    Current browse context: <div class="current">cs.CL</div>

  <div class="prevnext">
<span class="arrow">
      <a class="abs-button prev-url" href="/prevnext?id=2408.13359&amp;function=prev&amp;context=cs.CL"
         accesskey="p" title="previous in cs.CL (accesskey p)">&lt;&nbsp;prev</a>
    </span>
    <span class="is-hidden-mobile">&nbsp; | &nbsp;</span>    <span class="arrow">
      <a class="abs-button next-url" href="/prevnext?id=2408.13359&amp;function=next&amp;context=cs.CL" accesskey="n"
         title="next in cs.CL (accesskey n)">next&nbsp;&gt;</a>
    </span><br/>
  </div><div class="list">
    <a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.CL/new">new</a>
    <span class="is-hidden-mobile"> | </span>
    <a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.CL/recent">recent</a>
    <span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.CL/2024-08">2024-08</a>
  </div><div class="abs-switch-cat">
    Change to browse by:
    <div class="switch context-change">
        <a href="/abs/2408.13359?context=cs">cs</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.13359?context=cs.AI">cs.AI</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.13359?context=cs.LG">cs.LG</a><br class="is-hidden-mobile">
    </div>
  </div>

    </div>

    <div class="extra-ref-cite">
      <h3>References &amp; Citations</h3>
      <ul>
        <li><a  class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2408.13359">NASA ADS</a></li><li><a  class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2408.13359" target="_blank" rel="noopener">Google Scholar</a></li>
        <li><a  class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2408.13359" target="_blank" rel="noopener">Semantic Scholar</a></li>
      </ul>
      <div style="clear:both;"></div>
    </div>

<div class='extra-ref-cite'>
    <a id='bib-cite-css' hidden='true' href='/static/browse/0.3.4/css/cite.css'>a</a>

    <span id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</span>
    <span id='bib-cite-loading' hidden='true'>Loading...</span>
</div>

<div id='bib-cite-modal' class='bib-modal' hidden='true'>
    <div class='bib-modal-content'>
        <div class='bib-modal-title'>
            <h2>BibTeX formatted citation</h2>
            <span class='bib-modal-close' >&times;</span>
        </div>
        <div>
            <textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
        </div>
        <div>
            <span>Data provided by: </span>
            <a id='bib-cite-source-api'></a>
        </div>
    </div>
</div><div class="bookmarks">
  <div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2408.13359&amp;description=Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler"
     title="Bookmark on BibSonomy">
    <img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
         alt="BibSonomy logo"/>
  </a>
  <a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2408.13359&amp;title=Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler"
     title="Bookmark on Reddit">
    <img src="/static/browse/0.3.4/images/icons/social/reddit.png"
         alt="Reddit logo"/>
  </a>
</div>  </div>
  <!--end extra-services-->
<!-- LABS AREA -->
<div id="labstabs">
  <div class="labstabs"><input type="radio" name="tabs" id="tabone"checked="checked">
    <label for="tabone">Bibliographic Tools</label>
    <div class="tab labs-display-bib">
      <h1>Bibliographic and Citation Tools</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input id="bibex-toggle" type="checkbox" class="lab-toggle">
              <span class="slider"></span>
              <span class="is-sr-only">Bibliographic Explorer Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
          </div>
        </div>
<div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="litmaps-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
                aria-labelledby="label-for-litmaps">
              <span class="slider"></span>
              <span class="is-sr-only">Litmaps Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
          </div>
        </div>
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="scite-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
                aria-labelledby="label-for-scite">
              <span class="slider"></span>
              <span class="is-sr-only">scite.ai Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
          </div>
        </div>
      </div>
        <div class="labs-content-placeholder labs-display" style="display: none;"></div>
        <div style="min-height: 15px" id="litmaps-open-in"></div>
        <div style="min-height: 15px" id="scite-open-in"></div>
    </div>


    <input type="radio" name="tabs" id="tabtwo">
    <label for="tabtwo">Code, Data, Media</label>
    <div class="tab">
      <h1>Code, Data and Media Associated with this Article</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input        
                id="catalyzex-toggle"
                data-script-url="/static/browse/0.3.4/js/catalyzex.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="dagshub-toggle"
                data-script-url="/static/browse/0.3.4/js/dagshub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
              <span class="slider"></span>
              <span class="is-sr-only">DagsHub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
          </div>
        </div>
  
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="gotitpub-toggle"
                data-script-url="/static/browse/0.3.4/js/gotitpub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
              <span class="slider"></span>
              <span class="is-sr-only">GotitPub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="paperwithcode-toggle"
                data-script-url="/static/browse/0.3.4/js/paperswithcode.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-pwc">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-pwc">Papers with Code</span> <em>(<a href="https://paperswithcode.com/" target="_blank">What is Papers with Code?</a>)</em>
          </div>
        </div>


        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="sciencecast-toggle"
                data-script-url="/static/browse/0.3.4/js/sciencecast.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
              <span class="slider"></span>
              <span class="is-sr-only">ScienceCast Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
          </div>
        </div>
      </div>

      <div id="catalyzex-output" style="display:none"></div>
      <div id="dagshub-output" style="display:none"></div>
      <div id="gotitpub-output" style="display:none"></div>
      <div id="pwc-output" style="display:none"></div>
      <div id="pwc-data-output" style="display:none"></div>
      <div id="sciencecast-output" style="display:none"></div>
    </div>


      <input type="radio" name="tabs" id="labstabs-demos-input">
      <label for="labstabs-demos-input" id="labstabs-demos-label">Demos</label>
      <div class="tab">
        <h1>Demos</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="replicate-toggle"
                  data-script-url="/static/browse/0.3.4/js/replicate.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
                <span class="slider"></span>
                <span class="is-sr-only">Replicate Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="spaces-toggle"
                  data-script-url="/static/browse/0.3.4/js/spaces.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="txyz-toggle"
                  data-script-url="/static/browse/0.3.4/js/txyz.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
            </div>
          </div>
        </div>
        <div id="replicate-output"></div>
        <div id="spaces-output"></div>
        <div id="txyz-output"></div>
      </div>
      <input type="radio" name="tabs" id="tabfour">
      <label for="tabfour">Related Papers</label>
      <div class="tab">
        <h1>Recommenders and Search Tools</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="influenceflower-toggle"
                data-script-url="/static/browse/0.3.4/js/influenceflower.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
                <span class="slider"></span>
                <span class="is-sr-only">Link to Influence Flower</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="connectedpapers-toggle" type="checkbox" class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
                       aria-labelledby="label-for-connected-papers" >
                <span class="slider"></span>
                <span class="is-sr-only">Connected Papers Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
                <span class="slider"></span>
                <span class="is-sr-only">Core recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
            </div>
          </div></div>
        <div id="influenceflower-output"></div>
        <div id="influenceflower-output-graph" style="display:none">
          <ul class="flower-tabs">
            <li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
          </ul>
          <div class="flower-tab-content">
            <div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
            <div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
            <div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
            <div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
          </div>
        </div>
        <div style="min-height: 15px" id="connectedpapers-output"></div>
        <div id="coreRecommenderOutput"></div>
        <div id="iarxivOutput"></div>
      </div>

      <input type="radio" name="tabs" id="tabfive">
      <label for="tabfive">
        About arXivLabs
      </label>
      <div class="tab">
        <div class="columns">
          <div class="column">
            <h1>arXivLabs: experimental projects with community collaborators</h1>
            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
            <p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
          </div>
          <div class="column is-narrow is-full-mobile">
            <p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
          </div>
        </div>
      </div>

    </div>
</div>
<!-- END LABS AREA -->
  <div class="endorsers">
    <a href="/auth/show-endorsers/2408.13359" class="endorser-who">Which authors of this paper are endorsers?</a> |
    <a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
    <span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
  </div>
  <script type="text/javascript" language="javascript">mathjaxToggle();</script>
</div>
      </div>
    </main>

    <footer style="clear: both;">
      <div class="columns is-desktop" role="navigation" aria-label="Secondary" style="margin: -0.75em -0.75em 0.75em -0.75em">
        <!-- Macro-Column 1 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/about">About</a></li>
                <li><a href="https://info.arxiv.org/help">Help</a></li>
              </ul>
            </div>
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>contact arXiv</title><desc>Click here to contact arXiv</desc><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>
                  <a href="https://info.arxiv.org/help/contact.html"> Contact</a>
                </li>
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>subscribe to arXiv mailings</title><desc>Click here to subscribe</desc><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
                  <a href="https://info.arxiv.org/help/subscribe"> Subscribe</a>
                </li>
              </ul>
            </div>
          </div>
        </div>
        <!-- End Macro-Column 1 -->
        <!-- Macro-Column 2 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/license/index.html">Copyright</a></li>
                <li><a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy Policy</a></li>
              </ul>
            </div>
            <div class="column sorry-app-links">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/web_accessibility.html">Web Accessibility Assistance</a></li>
                <li>
                  <p class="help">
                    <a class="a11y-main-link" href="https://status.arxiv.org" target="_blank">arXiv Operational Status <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 512" class="icon filter-dark_grey" role="presentation"><path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"/></svg></a><br>
                    Get status notifications via
                    <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/email/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>email</a>
                    or <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/slack/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-black" role="presentation"><path d="M94.12 315.1c0 25.9-21.16 47.06-47.06 47.06S0 341 0 315.1c0-25.9 21.16-47.06 47.06-47.06h47.06v47.06zm23.72 0c0-25.9 21.16-47.06 47.06-47.06s47.06 21.16 47.06 47.06v117.84c0 25.9-21.16 47.06-47.06 47.06s-47.06-21.16-47.06-47.06V315.1zm47.06-188.98c-25.9 0-47.06-21.16-47.06-47.06S139 32 164.9 32s47.06 21.16 47.06 47.06v47.06H164.9zm0 23.72c25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06H47.06C21.16 243.96 0 222.8 0 196.9s21.16-47.06 47.06-47.06H164.9zm188.98 47.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06h-47.06V196.9zm-23.72 0c0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06V79.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06V196.9zM283.1 385.88c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06v-47.06h47.06zm0-23.72c-25.9 0-47.06-21.16-47.06-47.06 0-25.9 21.16-47.06 47.06-47.06h117.84c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06H283.1z"/></svg>slack</a>
                  </p>
                </li>
              </ul>
            </div>
          </div>
        </div> <!-- end MetaColumn 2 -->
        <!-- End Macro-Column 2 -->
      </div>
    </footer>
  </div>

  <script src="/static/base/1.0.1/js/member_acknowledgement.js"></script>

</body>

</html>
2408.13359
 Finding the optimal learning rate for language model pretraining is a challenging task. This is not only because there is a complicated correlation between learning rate, batch size, number of training tokens, model size, and other hyperparameters but also because it is prohibitively expensive to perform a hyperparameter search for large language models with Billions or Trillions of parameters. Recent studies propose using small proxy models and small corpus to perform hyperparameter searches and transposing the optimal parameters to large models and large corpus.  While the zero-shot transferability is theoretically and empirically proven for model size related hyperparameters, like depth and width, the zero-shot transfer from small corpus to large corpus is underexplored. In this paper, we study the correlation between optimal learning rate, batch size, and number of training tokens for the recently proposed WSD scheduler. After thousands of small experiments, we found a power-law relationship between variables and demonstrated its transferability across model sizes. Based on the observation, we propose a new learning rate scheduler, *Power scheduler*, that is agnostic about the number of training tokens and batch size. The experiment shows that combining the Power scheduler with Maximum Update Parameterization (µP) can consistently achieve impressive performance with one set of hyperparameters regardless of the number of training tokens, batch size, model size, and even model architecture. Our 3B dense and MoE models trained with the Power scheduler achieve comparable performance as state-of-the-art small language models. We open source these pretrained models at https://ibm.biz/BdKhLa. 
abstract
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
Abstract
2408.13359
 Learning rate is a critical hyperparameter for deep neural network training. In the context of Large Language Models (LLMs), the cosine learning rate scheduler is the most commonly used strategy. It has been shown to be effective across multiple state-of-the-art models, including Llama 3 (Dubey et al., 2024), Gopher (Rae et al., 2021), etc. However, the cosine scheduler requires pre-defined training step counts to achieve the optimal loss. This results in two main drawbacks: 1) the intermediate training checkpoints are suboptimal, 2) continual training of an existing language model becomes complicated.  MiniCPM (Hu et al., 2024) proposes the Warmup-Stable-Decay (WSD) learning rate scheduler (illustrated in Figure 1) to address these issues. The WSD learning rate schedule is divided into three phases: 1) warmup phase, linearly increase the learning rate from 0 to peak; 2) stable phase, maintain the learning rate at peak value and training the model for most of the time; 3) decay phase, annealing the learning rate to 0 in a relatively short period. The main advantage of this schedule is that specifying the number of training steps in advance is not required. This is particularly convenient for large runs, as the decay can be applied at any time to observe model performance and decide whether to stop. It also allows for continual learning by default, as training can be resumed from a stable phase checkpoint. Moreover, the data mixture can be changed during the decay phase to increase the ratio of high-quality data. This data curriculum is shown to be effective in several recent language models (Team et al., 2023; Hu et al., 2024; Dubey et al., 2024; Shen et al., 2024).  ∗Corresponding Author: rpanda@ibm.com  ![1_image_0.png](1_image_0.png)  However, is WSD really agnostic to token count? In our experiments, we found that although the WSD scheduler could, in theory, continue in the stable phase forever, the optimal learning rates are different for different amounts of training tokens. In other words, the optimal learning rate scheduler is tied to the number of training tokens. Thus, the WSD scheduler still faces the same issue as the cosine scheduler: the intermediate and continual training checkpoints are suboptimal if the number of tokens is too different from the original plan. Furthermore, deciding the optimal learning rate is still challenging for large-scaling pretraining. As our experiments will show, there is a complicated correlation between hyperparameters, including learning rate, model size, batch size, and number of training steps. The size and training cost of modern LLMs make it impossible to do a hyperparameter search on the target model size and training flops. Researchers have proposed using small proxy models to run hyperparameter searches and predict the optimal hyperparameters of large models from search results (Dey et al., 2023; Hu et al., 2024; Yang et al., 2022). Among these methods, µTransfer (Yang et al., 2022; 2023) is proposed to facilitate zero-shot hyperparameter transfer between different model sizes. µTransfer has been successfully applied to several language models, including Cerebras-GPT (Dey et al., 2023), miniCPM (Hu et al., 2024), and AFM (Gunter et al., 2024).  In this paper, we first combine the WSD scheduler and µTransfer to study the learning rate transfer between proxy and large models. Our extensive experiments show that µTransfer does not provide direct zero-shot learning rate transferability across the numbers of tokens and batch sizes for the WSD optimizer. Instead, the optimal learning rate ηopt satisfies a power-law relation with respect to batch size β and number of tokens T: ηopt = β · aTb(1) where a and b are power-low coefficients. Furthermore, our experiment confirms the zero-shot transferability across model sizes, different model sizes share very similar coefficients. Inspired by this observation, we propose a new learning rate scheduler, PowerLR, that is agnostic to batch size and token number. It allows direct transfer of the optimal learning rate scheduling across batch size, token numbers, and model size. Thus, the expensive pretraining runs can be trained without specifying the number of training tokens, enabling early stop and continual pretraining without sacrificing convergence. 
introduction
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
Introduction
2408.13359
 Maximal Update Parameterization (µP) (Yang & Hu, 2020; Yang et al., 2022; 2023) controls initialization, layer-wise learning rates, and activation magnitudes to ensure analytically stable training, independent of a model's width and depth. In addition to improving training stability, µP improves the transferability of training hyperparameters from small proxy models to large models, a technique called µTransfer. The hyperparameter transferability of µP is theoretically justified for width (Yang et al., 2022) and depth (Yang et al., 2023). Previous work also show empirical evidence for transferability across batch size, training steps, and training sequence length.  | Name                    | Function                                                                                                            | |-------------------------|---------------------------------------------------------------------------------------------------------------------| | Embedding multiplier    | Multiply the embedding output with memb                                                                             | | Residual Multiplier     | Multiply the output of each attention and MLP layer with mres before adding to residual connection                  | | Initialization std      | Initialize internal weight matrices (excluding input and output √mwidth embedding) with standard deviation initstd/ | | Learning rate scaling   | Set learning rate of internal weight matrices to η/mwidth                                                           | | Attention logit scaling | Divide attention logits by dhead                                                                                    |  In this paper, we follow the µP config used in CerebrasGPT (Dey et al., 2023) to study the transferability of batch size and learning rate across different numbers of training tokens and model sizes.  Table 1 lists the µP changes we applied to model initialization, learning rate, and multipliers. 
body
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
Background 2.1 Maximum Update Parametrization (Μp)
2408.13359
 MiniCPM (Hu et al., 2024) proposes the WSD learning rate scheduler to divide pretraining into three stages: the warmup stage, the stable training stage, and the remaining decay stage. The WSD scheduler is defined as:  $$\text{WSD}\left(n\right)=\begin{cases}\frac{n}{N_{\text{\tiny{\sc warmup}}}}\cdot\eta&\text{if}n<N_{\text{\tiny{\sc warmup}}}\\ \eta&\text{if}N_{\text{\tiny{\sc warmup}}}<n\leq N-N_{\text{\tiny{\sc decay}}}\,\\ f(n,N,N_{\text{\tiny{\sc decay}}})\cdot\eta&\text{if}n>N-N_{\text{\tiny{\sc decay}}}\end{cases}\tag{2}$$  where n is the current number of steps, η is the stable learning rate, N is the total number of steps, Nwarmup is the number of warmup steps, Ndecay is the number of decay steps, f(*n, N, N*decay) is the learning rate decay function. The main advantage of WSD scheduler over other schedulers (e.g.  cosine and linear) is that specifying the number of training steps in advance is not required. Since the learning rate in the stable training stage does not depend on the number of training steps, the WSD scheduler does not need to specify the number of training steps in advance. This is particularly convenient for large runs, as the cooldown can be initiated at any time to observe model behavior and decide whether to stop (Hagele et al., 2024). It also facilitates extended pre-training, which can be ¨ applied to the last checkpoint in the stable training phase. Moreover, the data mixture can be changed during the cooldown phase to incorporate more high-quality data toward the end of the training. This data mixture strategy has been proven effective in many recent language models (Hu et al., 2024; Dubey et al., 2024; Team et al., 2024). 
body
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
Warmup-Stable-Decay (Wsd) Scheduler
2408.13359
 In this section, we focus on finding the optimal learning rate η for a small proxy model. Due to our use of µP, we expect the optimal learning rate to be invariant across different model sizes, batch sizes, and training steps. To verify this, we conducted extensive experiments to find the optimal learning rate for different model sizes and numbers of training tokens. Table 2 shows our overall hyperparameter search configurations. We use the WSD scheduler with 10% tokens in the decay phase following previous works (Hu et al., 2024; Hagele et al., 2024). All models in this section and ¨ the next section are trained on the RedPajama (Computer, 2023) corpus and tested on a holdout test set. 
body
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
Search Optimal Learning Rate For Wsd Scheduler With Μp
2408.13359
 We conduct two controlled experiments to verify the optimal learning rate's transferability. First, we fix the batch size β = 128 and swept across all combinations of learning rates η and numbers of training tokens T. Figure 2(a) shows that the optimal learning rate, ηopt, does not transfer across a  | across different numbers of training tokens. Model size 12M   | 36M                                                            | 121M   |      | |---------------------------------------------------------------|----------------------------------------------------------------|--------|------| | Fixed hyperparameters Attention head size                     | 64                                                             | 64     | 64   | | Number of layers                                              | 32                                                             | 32     | 32   | | Number of attention heads                                     | 2                                                              | 4      | 8    | | Embedding size                                                | 128                                                            | 256    | 512  | | MLP hidden size                                               | 320                                                            | 640    | 1280 | | Initialization std                                            | 0.02                                                           | 0.02   | 0.02 | | Sequence Length                                               | 4096                                                           | 4096   | 4096 | | Warmup tokens                                                 | 1B                                                             | 1B     | 1B   | | mwidth                                                        | 0.5                                                            | 1      | 2    | | memb                                                          | 1                                                              | 1      | 1    | | mresidual                                                     | 1                                                              | 1      | 1    | | Variable hyperparameters Training tokens T                    | 2B, 4B, 8B, 16B, 32B, 64B, 128B, 256B                          |        |      | | Batch size β                                                  | 16, 32, 64, 128, 256, 512                                      |        |      | | Learning rate η                                               | 0.0002, 0.0004, 0.0008, 0.0016, 0.0032, 0.0064, 0.0128, 0.0256 |        |      |  ![3_image_0.png](3_image_0.png)  wide range of T. ηopt tends to decrease with respect to the number of training tokens. For example, ηopt is 0.0128 for 2B training tokens and 0.0008 for 256B. Second, we fix the number of training tokens T to 128 billion tokens and swept across all combinations of learning rates η and batch sizes. Figure 2(b) shows that the optimal learning rates do not transfer across a wide range of β, but tends to increase with respect to the batch size. The optimal learning rate ηopt is 0.0002 for batch size 16 and 0.0064 for batch size 512.  3.2 WHAT IS THE RELATIONSHIP BETWEEN ηopt, β AND T ?  To better understand the relationship between ηopt, β, and T, we swept across all possible combinations of these three variables. Figure 3 shows the optimal learning rate for each batch size and training token combination. Like our previous observation, the optimal learning rate consistently decreases with respect to the number of training tokens across different batch sizes. Furthermore, we 2B 4B 8B 16B 32B 64B 128B  ![4_image_0.png](4_image_0.png)  also notice that the ratio between the optimal learning rate ηopt and batch size β is relatively stable for each number of training tokens. Based on this observation, we make our first hypothesis: Hypothesis 1 *The optimal learning rate* ηopt for the WSD scheduler and a given pair of (*T, β*) is proportional to the training batch size β.  Thus, we define γ as the ratio between ηopt and β:  $$\gamma={\frac{\eta_{\mathrm{opt}}}{\beta}}$$ $$({\mathfrak{I}})$$  β(3) To verify Hypothesis 1, we conducted an extensive hyperparameter search for three model sizes: 12M, 36M, and 121M. After finding the optimal learning rate ηopt for every combination of (*T, β,* model size), we only keep the three best batch sizes to focus on the optimal scenario. The γ of the three best batch sizes for each T and model size are plotted in in Figure 4(a), Figure 4(b), and Figure 4(c). These results show that, given a fixed number of training tokens T, γ falls in a relatively small region.  Furthermore, we notice that γ approximately follows a power-law relation with respect to the number of training tokens. Thus, we make a second hypothesis: Hypothesis 2 The Learning rate to batch size ratio γ *has a power-law correlation with* T:  $\tau\epsilon\mathfrak{f}_{\mu}$ γ = aTb(4) when using µP*, the correlation can be transferred across model sizes.* To verify the Hypothesis 2, we compute the average γ of the best batch sizes to estimate the real γ for each number of tokens β. Figure 4(d) shows that all three model sizes share a similar power-law correlation. After aggregating the results from three sizes, we get the following correlation: γ = 4.6T −0.51 (5) where x is the number of training tokens.  In other words, the relation between optimal learning rate ηopt for the WSD scheduler, batch size β, and the number of training tokens T can be approximately modeled with the following equations: ηopt = β · aTb(6) Equation 6 provides an easy way to predict the optimal learning rate given the number of training tokens and training batch size for the WSD scheduler. As in prior work, a and b can be easily estimated through hyperparameter search on a small proxy model.  ![5_image_0.png](5_image_0.png)  However, this relationship between the number of training tokens and the optimal learning rate constrains the number of training steps to be specified before training. We also noticed that the equation tends to provide a small learning rate for a large-scale training corpus. For example, given a 10 trillion token corpus and a batch size 1024, the predicted optimal learning rate is 0.0011. If we consider that µP will divide the learning rate by a factor of mwidth = dmodel/dbase for matrices inside the model, the actual learning rate will be very small for large models. While a small learning rate is good for final convergence, it is also likely to cause insufficient exploration at the beginning of training. To solve these issues, we propose a novel power learning rate scheduler in the next section. 
body
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
Does Optimal Learning Rate Transfer?
2408.13359
 Inspired by previous observations, we propose a new power learning rate based on the observation from the previous section: ηpower (n) = min ηmax, β · anb(7) where β is the batch size, n is the number of tokens already trained, a is the amplitude of the learning rate, b is a power-law exponent for decaying the learning with respect to the number of trained tokens, and ηmax is the learning rate upper bound that rejects very large learning. Like the constant learning rate, the power learning rate also does not require specifying the number of training steps or total training tokens beforehand, since the learning rate only depends on the current number of training tokens.  Like the WSD scheduler, we can combine the power learning rate with warmup and decay to get the final Power scheduler:   Power $\left(n\right)=$.   $\frac{n}{N_{\text{warmup}}}\cdot\eta_{\text{power}}\left(N_{\text{warmup}}\right)$  $\eta_{\text{power}}\left(n\right)$  $f(n,N,N_{\text{decay}})\cdot\eta_{\text{power}}\left(N-N_{\text{decay}}\right)$ · ηpower (Nwarmup) if *n < N*warmup ηpower (n) if Nwarmup < n ≤ N − Ndecay f(*n, N, N*decay) · ηpower (N − Ndecay) if *n > N* − Ndecay if $n<N_{\rm warmup}$  if $N_{\rm warmup}<n\leq N-N_{\rm decay}$ (8)  if $n>N-N_{\rm decay}$  Table 3: Hyperparameter search config for Power scheduler. The search range for a and b is decided  based on our observation in Section 3.  Model size 36M  Fixed hyperparameters  ![6_image_0.png](6_image_0.png)  Number of attention heads 4  Max learning rate ηmax 0.02  Variable hyperparameters  b [−0.6, −0.4]  The Power scheduler requires three separate hyperparameters (ηmax*, a, b*), instead of only one hyperparameter required by the cosine and WSD schedulers. However, we expect these hyperparameters to transfer across different model sizes, number of training tokens, and batch sizes. Additionally, since Figure 1 shows that the ηmax has a very limited impact on the overall learning rate curve, we can simply set it to a large enough value. In this paper, we set ηmax to 0.02. Then, we conduct another hyperparameter search for a and b to find the optimal learning rate hyperparameter. Table 3 shows the configuration used in the Power scheduler hyperparameter search.  ![6_image_1.png](6_image_1.png)  ![7_image_0.png](7_image_0.png)  Figure 5 shows the hyperparameter search results for 32 billion training tokens. We observe that the test perplexity is not sensitive to different choices of a within our search range of [3, 5]. In contrast, the test perplexity is more sensitive to the choices of b. The optimal b is between −0.52 and −0.51.  A similar observation has been made across different numbers of training tokens. Figure 6 shows that the optimal b for different β and T consistently falls within the range of [−0.49, −0.53] with few exceptions. Combining this and previous observations, we select a = 4 and b = −0.51 for the rest of this paper. 
body
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
Power Scheduler
2408.13359
 This section compares the Power scheduler with the WSD and Cosine scheduler across different scenarios. In the first part, we conduct a controlled experiment to train 1B transformer models and 1B mixture-of-experts (MoE) models with different learning rate schedulers to show that the Power scheduler is comparable to or better than other schedulers. In the second part, we take a more realistic setting, training a 3B transformer language model and a 3B MoE language model with high-quality data in the decay phase to compare with strong open-source language models. For all the Power scheduler experiments in this section, we will use the hyperparameters from Section 4, a = 4, b = −0.51, and ηmax = 0.02. 
body
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
Pre-Training Experiments
2408.13359
 We train a series of 1B parameter dense and Mixture-of-Experts (MoE) transformer models using WSD, cosine, and Power schedulers. All models are trained with 1T tokens. We use the optimal hyperparameters proposed in MiniCPM (Hu et al., 2024) and adapt them to our 1B setting with µTransfer. All models are trained with batch size 1024. For WSD and cosine scheduler, we set η = 0.01, following the optimal learning rate from MiniCPM. For the Power and WSD scheduler, we exponentially decay the learning rate to 0 for the last 100B tokens. The MoE models are implemented with ScatterMoE (Tan et al., 2024). More model details can be found in Table 4.  We evaluate all 1B models on language model tasks and multiple-choice tasks from LM evaluation Harness (Gao et al., 2024). The multiple-choice tasks include grade-school science questions (ARC, Clark et al. (2018)), yes/no questions (BoolQ, Clark et al. (2019)), common sense reasoning (Hellaswag, Zellers et al. (2019)), open book question answering (OpenBookQA, Mihaylov et al.  (2018)), physical questions (PIQA, Bisk et al. (2020)), and Winograd schema task (Winogrande, Sakaguchi et al. (2021)). Table 5 shows the performance. The Power scheduler provides consistently better or comparable performance for both language modeling and downstream tasks. Surprisingly,  | Table 4: Hyperparameter for 1B and 3B models. Model 1B Dense 1B MoE PowerLM-3B PowerMoE-3B   |        |        |        |        | |----------------------------------------------------------------------------------------------|--------|--------|--------|--------| | Embedding size                                                                               | 1536   | 1024   | 2304   | 1536   | | Number of layers                                                                             | 40     | 24     | 40     | 32     | | Attention head size                                                                          | 64     | 64     | 64     | 64     | | Number of attention heads                                                                    | 24     | 16     | 36     | 24     | | Number of KV heads                                                                           | 24     | 8      | 36     | 8      | | MLP hidden size                                                                              | 4096   | 512    | 9216   | 512    | | MLP activation                                                                               | swiglu | swiglu | swiglu | swiglu | | Number of Experts                                                                            | -      | 32     | -      | 40     | | MoE TopK                                                                                     | -      | 8      | -      | 8      | | Initialization std                                                                           | 0.1    | 0.1    | 0.1    | 0.1    | | Sequence Length                                                                              | 4096   | 4096   | 4096   | 4096   | | mwidth                                                                                       | 6      | 4      | 8      | 6      | | memb                                                                                         | 12     | 12     | 12     | 12     | | mresidual                                                                                    | 0.22   | 0.28   | 0.22   | 0.22   | | #Parameters                                                                                  | 1.2B   | 1.3B   | 3.5B   | 3.3B   | | #Active Parameters                                                                           | 1.2B   | 377M   | 3.5B   | 800M   | | #Training tokens                                                                             | 1T     | 1.1T   | 1.25T  | 2.75T  |  Table 5: Language Modeling and Zero-shot performance of 1B models. accn means accuracy with  average log probability.  Task Wiki ARC BoolQ Hellaswag OBQA PIQA WinoGrande **Average**  Metric ppl accn acc accn accn accn acc  Dense  Cosine 14.5 **44.6** 63.5 63.6 37.6 75.3 61.3 57.7  WSD 13.9 43.0 62.8 **64.8 38.0** 74.5 **62.1** 57.6  Power **13.8** 44.3 **65.6** 64.6 37.0 **76.0** 61.8 **58.2**  MoE  Cosine 15.7 40.6 **63.6** 59.7 35.2 74.0 57.2 55.1  WSD 14.8 43.9 59.7 61.0 34.4 **76.0 59.3** 55.7  Power 14.6 **44.0** 60.4 **61.6 36.0** 74.8 58.3 **55.9**  even though we only performed a hyperparameter search on a small dense model, the hyperparameters still performed well on the MoE model. 
body
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
1B Controlled Experiment
2408.13359
 To compare performance against strong open-source language models, we pretrain two 3B language models: 1) PowerLM-3B, a 3B dense language model, and 2) PowerMoE-3B, a 3B MoE language model1. We pretrain these two models using the two-stage training schema in Hu et al. (2024) and Power scheduler. Stage 1 linearly warms up the learning rate and then applies the power decay. The training corpus is a mix of large-scale, medium-quality open-source datasets with permissive licenses.  PowerLM-3B is trained on 1T tokens, and PowerMoE-3B on 2.5T tokens. Stage 2 exponentially decays the learning rate to zero. The training corpus has 250B tokens, mixed from stage 1 data and a small amount of high-quality open-source/synthetic corpora with permissive licenses. The training batch size is 1024.  Table 6 shows the multi-choices performance of our model and state-of-the-art models. Table 7 shows MMLU and generative performance in the math and code domain. The results show that, despite being trained with relatively fewer tokens, our PowerLM-3B still achieves comparable performance with state-of-the-art 2B to 4B language models. Furthermore, our PowerMoE-3B uses only 800M active parameters but performs similarly to state-of-the-art 1B to 2B dense models.  9  | Table 6: Zero-shot performance on multiple-choice tasks.   |      |       |           |      |      |            |         | |------------------------------------------------------------|------|-------|-----------|------|------|------------|---------| | Task                                                       | ARC  | BoolQ | Hellaswag | OBQA | PIQA | WinoGrande | Average | | Metric                                                     | accn | acc   | accn      | accn | accn | acc        |         | | Qwen1.5-4B                                                 | 50.6 | 77.7  | 71.5      | 39.8 | 77.0 | 64.4       | 63.5    | | Gemma2-2B                                                  | 65.0 | 72.8  | 73.0      | 41.4 | 79.2 | 68.9       | 66.7    | | PowerLM-3B                                                 | 57.2 | 75.0  | 74.2      | 41.2 | 79.9 | 66.3       | 65.6    | | Qwen1.5-1.8B                                               | 47.1 | 66.3  | 60.9      | 34.2 | 74.0 | 60.8       | 57.2    | | Qwen2-1.5B                                                 | 48.1 | 72.5  | 65.4      | 36.4 | 75.3 | 66.2       | 60.7    | | SmolLM-1.7B                                                | 59.9 | 66.1  | 65.6      | 42.0 | 75.7 | 60.5       | 61.7    | | PowerMoE-3B                                                | 54.8 | 67.1  | 70.3      | 38.4 | 77.5 | 64.6       | 62.2    |  | Task         | GSM8k   | MATH       | HumanEval   | MBPP   | MMLU   |        |        | |--------------|---------|------------|-------------|--------|--------|--------|--------| | Prompt       | 5-shot  | 8-shot,cot | 5-shot      | 4-shot | 0-shot | 0-shot | 0-shot | | Metric       | exact   | exact      | exact       | sympy  | pass@1 | pass@1 | acc    | | Qwen1.5-4B   | 53.1    | 54.8       | 9.76        | 12.8   | 7.3    | 30.6   | 54.1   | | Gemma2-2B    | 23.9    | 25.5       | 9.80        | 15.4   | 19.5   | 27.8   | 49.4   | | PowerLM-3B   | 35.9    | 43.1       | 8.28        | 14.0   | 21.9   | 28.0   | 44.3   | | SmolLM-1.7B  | 6.6     | 6.8        | 4.86        | 3.0    | 20.7   | 30.6   | 27.5   | | Qwen1.5-1.8B | 34.7    | 37.3       | 5.66        | 9.6    | 4.3    | 21.6   | 45.0   | | Qwen2-1.5B   | 58.3    | 58.6       | 4.38        | 23.3   | 16.5   | 31.0   | 55.1   | | PowerMoE-3B  | 27.2    | 32.1       | 6.70        | 10.0   | 15.2   | 24.0   | 33.4   | 
body
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
3B Realistic Experiment
2408.13359
 In this paper, we systematically study the relationship between optimal learning rate, batch size, and number of training tokens. We observed a power-law relation between these variables while using the WSD learning rate scheduler. Inspired by this observation, we propose a new learning rate scheduler, the Power scheduler, that is invariant with respect to the number of tokens and batch size.  The experiment shows that combining the Power scheduler with Maximum Update Parameterization (µP) can consistently achieve impressive performance with one set of hyperparameters regardless of the number of training tokens, batch size, model size, and even model architecture. 
conclusion
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
Conclusion
2408.13359
 Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In *Proceedings of the AAAI conference on artificial intelligence*, volume 34, pp. 7432–7439, 2020.  Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.  Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.  arXiv preprint arXiv:1803.05457, 2018.  Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023.  URL https://github.com/togethercomputer/RedPajama-Data.  Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness, et al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. *arXiv preprint arXiv:2304.03208*, 2023.  Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.  arXiv preprint arXiv:2407.21783, 2024.  Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo.org/records/12608602.  Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, et al. Apple intelligence foundation language models. *arXiv preprint arXiv:2407.21075*, 2024.  Alexander Hagele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin ¨ Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. *arXiv preprint* arXiv:2405.18392, 2024.  Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. *NeurIPS*, 2021.  Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. *arXiv preprint arXiv:2404.06395*, 2024.  Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. *Advances in Neural Information Processing Systems*, 35:3843–3857, 2022.  Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. *arXiv preprint arXiv:1809.02789*, 2018.  Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. *arXiv preprint arXiv:2112.11446*, 2021.  Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. *Communications of the ACM*, 64(9) :99–106, 2021. Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching llama2 performance with 0.1 m dollars. *arXiv preprint arXiv:2404.07413*, 2024.  Shawn Tan, Yikang Shen, Rameswar Panda, and Aaron Courville. Scattered mixture-of-experts implementation. *arXiv preprint arXiv:2403.08245*, 2024.  Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. *arXiv preprint arXiv:2312.11805*, 2023.  Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´ e, et al. ´ Gemma 2: Improving open language models at a practical size. *arXiv preprint arXiv:2408.00118*, 2024. Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. *arXiv preprint* arXiv:2011.14522, 2020.  Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. *arXiv preprint arXiv:2203.03466*, 2022. Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks. *arXiv preprint arXiv:2310.02244*, 2023.  Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? *arXiv preprint arXiv:1905.07830*, 2019.  12
references
Power Scheduler: A Batch Size And Token Num-Ber Agnostic Learning Rate Scheduler
References