<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

<head>  <title>[2408.12857] Memory-Efficient LLM Training with Online Subspace Descent</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
  <link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
  <link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20240822" />
  <link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
  <script language="javascript" src="/static/browse/0.3.4/js/accordion.js" /></script>
  
    <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/slider.css?v=1.14" />
    <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="/static/browse/0.3.4/js/donate.js?v=1.11"></script><link rel="canonical" href="/abs/2408.12857"/><meta property="og:type" content="website" />
<meta property="og:site_name" content="arXiv.org" />
<meta property="og:title" content="Memory-Efficient LLM Training with Online Subspace Descent" />
<meta property="og:url" content="https://arxiv.org/abs/2408.12857v1" />
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="700" />
<meta property="og:image:alt" content="arXiv logo"/>
<meta property="og:description" content="Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimizer states into a subspace using projection matrix found by singular value decomposition (SVD). However, convergence of these algorithms is highly dependent on the update rules of their projection matrix. In this work, we provide the \emph{first} convergence guarantee for arbitrary update rules of projection matrix. This guarantee is generally applicable to optimizers that can be analyzed with Hamiltonian Descent, including most common ones, such as LION, Adam. Inspired by our theoretical understanding, we propose Online Subspace Descent, a new family of subspace descent optimizer without SVD. Instead of updating the projection matrix with eigenvectors, Online Subspace Descent updates the projection matrix with online PCA. Online Subspace Descent is flexible and introduces only minimum overhead to training. We show that for the task of pretraining LLaMA models ranging from 60M to 7B parameters on the C4 dataset, Online Subspace Descent achieves lower perplexity and better downstream tasks performance than state-of-the-art low-rank training methods across different settings and narrows the gap with full-rank baselines."/>
<meta name="twitter:site" content="@arxiv"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Memory-Efficient LLM Training with Online Subspace Descent"/>
<meta name="twitter:description" content="Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimizer states into a..."/>
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
<meta name="twitter:image:alt" content="arXiv logo"/>
  <link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/>  <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script>  <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
  <script src="/static/browse/0.3.4/js/toggle-labs.js?20230622" type="text/javascript"></script>
  <script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Memory-Efficient LLM Training with Online Subspace Descent" /><meta name="citation_author" content="Liang, Kaizhao" /><meta name="citation_author" content="Liu, Bo" /><meta name="citation_author" content="Chen, Lizhang" /><meta name="citation_author" content="Liu, Qiang" /><meta name="citation_date" content="2024/08/23" /><meta name="citation_online_date" content="2024/08/23" /><meta name="citation_pdf_url" content="http://arxiv.org/pdf/2408.12857" /><meta name="citation_arxiv_id" content="2408.12857" /><meta name="citation_abstract" content="Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimizer states into a subspace using projection matrix found by singular value decomposition (SVD). However, convergence of these algorithms is highly dependent on the update rules of their projection matrix. In this work, we provide the \emph{first} convergence guarantee for arbitrary update rules of projection matrix. This guarantee is generally applicable to optimizers that can be analyzed with Hamiltonian Descent, including most common ones, such as LION, Adam. Inspired by our theoretical understanding, we propose Online Subspace Descent, a new family of subspace descent optimizer without SVD. Instead of updating the projection matrix with eigenvectors, Online Subspace Descent updates the projection matrix with online PCA. Online Subspace Descent is flexible and introduces only minimum overhead to training. We show that for the task of pretraining LLaMA models ranging from 60M to 7B parameters on the C4 dataset, Online Subspace Descent achieves lower perplexity and better downstream tasks performance than state-of-the-art low-rank training methods across different settings and narrows the gap with full-rank baselines." />
</head>

<body  class="with-cu-identity">
  
  <aside class="slider-wrapper bps-banner forum grey">
    <a class="close-slider bps-banner" href="#"><img src="/static/browse/0.3.4/images/icons/close-slider.png" alt="close this message"></a>
  	<div class="columns">
      <img role="presentation" class="bps-banner-image" src="/static/browse/0.3.4/images/events/arxiv-forum-logotype-title.png" alt="arXiv Accessibility Forum 2024">
      <div class="copy-donation bps-banner">
        <h2 id="forum-session-title"></h2>
        <p id="forum-session-link-small"></p>
      </div>
      <div class="amount-donation bps-banner">
          <div id="forum-session-link-box" class="donate-cta"></div>
      </div>
    </div>
  </aside>
  <script src="/static/browse/0.3.4/js/rotate_forum_info.js" type="text/javascript"></script>
  <div class="flex-wrap-footer">
    <header>
      <a href="#content" class="is-sr-only">Skip to main content</a>
      <!-- start desktop header -->
      <div class="columns is-vcentered is-hidden-mobile" id="cu-identity">
        <div class="column" id="cu-logo">
          <a href="https://www.cornell.edu/"><img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University" /></a>
        </div><div class="column banner-minimal forum">
            <!-- /from July 7 at 1:00 AM to Sept 2 at 1:00 AM --><p>This week: the arXiv Accessibility Forum</p>
            <a href="https://accessibility2024.arxiv.org/schedule" target="_blank">Forum Schedule</a></div><div class="column" id="support-ack">
          <span id="support-ack-url">We gratefully acknowledge support from the Simons Foundation, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and all contributors.</span>
          <a href="https://info.arxiv.org/about/donate.html" class="btn-header-donate">Donate</a>
        </div>
      </div>

      <div id="header" class="is-hidden-mobile">
<a aria-hidden="true" href="{url_path('ignore_me')}"></a>
  <div class="header-breadcrumbs is-hidden-mobile">
    <a href="/"><img src="/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg" alt="arxiv logo" style="height:40px;"/></a> <span>&gt;</span> <a href="/list/cs/recent">cs</a> <span>&gt;</span> arXiv:2408.12857
  </div>
          <div class="search-block level-right">
    <form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
      <div class="field has-addons">
        <div class="control">
          <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
          <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
        </div>
        <div class="control">
          <div class="select is-small">
            <select name="searchtype" aria-label="Field to search">
              <option value="all" selected="selected">All fields</option>
              <option value="title">Title</option>
              <option value="author">Author</option>
              <option value="abstract">Abstract</option>
              <option value="comments">Comments</option>
              <option value="journal_ref">Journal reference</option>
              <option value="acm_class">ACM classification</option>
              <option value="msc_class">MSC classification</option>
              <option value="report_num">Report number</option>
              <option value="paper_id">arXiv identifier</option>
              <option value="doi">DOI</option>
              <option value="orcid">ORCID</option>
              <option value="author_id">arXiv author ID</option>
              <option value="help">Help pages</option>
              <option value="full_text">Full text</option>
            </select>
          </div>
        </div>
        <input type="hidden" name="source" value="header">
        <button class="button is-small is-cul-darker">Search</button>
      </div>
    </form>
  </div>
     </div><!-- /end desktop header -->

      <div class="mobile-header">
        <div class="columns is-mobile">
          <div class="column logo-arxiv"><a href="https://arxiv.org/"><img src="/static/browse/0.3.4/images/arxiv-logomark-small-white.svg" alt="arXiv logo" style="height:60px;" /></a></div>
          <div class="column logo-cornell"><a href="https://www.cornell.edu/">
            <picture>
              <source media="(min-width: 501px)"
                srcset="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg  400w"
                sizes="400w" />
              <source srcset="/static/browse/0.3.4/images/icons/cu/cornell_seal_simple_black.svg 2x" />
              <img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University Logo" />
            </picture>
          </a></div>
          <div class="column nav" id="toggle-container" role="menubar">
            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-white"><title>open search</title><path d="M505 442.7L405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9 0 208 0S0 93.1 0 208s93.1 208 208 208c48.3 0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9 0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7 0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7 0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
                <div class="field has-addons">
                  <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
                  <input type="hidden" name="source" value="header">
                  <input type="hidden" name="searchtype" value="all">
                  <button class="button">GO</button>
                </div>
              </form>
            </div>

            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-white" role="menu"><title>open navigation menu</title><path d="M16 132h416c8.837 0 16-7.163 16-16V76c0-8.837-7.163-16-16-16H16C7.163 60 0 67.163 0 76v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16z"/ ></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <nav class="mobile-menu" aria-labelledby="mobilemenulabel">
                <h2 id="mobilemenulabel">quick links</h2>
                <ul>
                    <li><a href="https://arxiv.org/login">Login</a></li>
                    <li><a href="https://info.arxiv.org/help">Help Pages</a></li>
                    <li><a href="https://info.arxiv.org/about">About</a></li>
                </ul>
              </nav>
            </div>
          </div>
        </div>
      </div><!-- /end mobile-header -->
    </header>

    <main>
      <div id="content">
<!--
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
    <rdf:Description
        rdf:about="/abs/2408.12857"
        dc:identifier="/abs/2408.12857"
        dc:title="Memory-Efficient LLM Training with Online Subspace Descent"
        trackback:ping="/trackback/2408.12857" />
    </rdf:RDF>
--><div id="abs-outer">

  <div class="leftcolumn">
    <div class="subheader">
      <h1>Computer Science > Machine Learning</h1>
    </div>

    <div class="header-breadcrumbs-mobile">
      <strong>arXiv:2408.12857</strong> (cs)
    </div>
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
<div id="content-inner">
  <div id="abs">
    <div class="dateline">
  [Submitted on 23 Aug 2024]</div>
    <h1 class="title mathjax"><span class="descriptor">Title:</span>Memory-Efficient LLM Training with Online Subspace Descent</h1>
    <div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Liang,+K">Kaizhao Liang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Liu,+B">Bo Liu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Chen,+L">Lizhang Chen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Liu,+Q">Qiang Liu</a></div>            <div id="download-button-info" hidden>View a PDF of the paper titled Memory-Efficient LLM Training with Online Subspace Descent, by Kaizhao Liang and 3 other authors</div>
    <a class="mobile-submission-download" href="/pdf/2408.12857">View PDF</a>
    <a class="mobile-submission-download" href="https://arxiv.org/html/2408.12857v1">HTML (experimental)</a>



    <blockquote class="abstract mathjax">
            <span class="descriptor">Abstract:</span>Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimizer states into a subspace using projection matrix found by singular value decomposition (SVD). However, convergence of these algorithms is highly dependent on the update rules of their projection matrix. In this work, we provide the \emph{first} convergence guarantee for arbitrary update rules of projection matrix. This guarantee is generally applicable to optimizers that can be analyzed with Hamiltonian Descent, including most common ones, such as LION, Adam. Inspired by our theoretical understanding, we propose Online Subspace Descent, a new family of subspace descent optimizer without SVD. Instead of updating the projection matrix with eigenvectors, Online Subspace Descent updates the projection matrix with online PCA. Online Subspace Descent is flexible and introduces only minimum overhead to training. We show that for the task of pretraining LLaMA models ranging from 60M to 7B parameters on the C4 dataset, Online Subspace Descent achieves lower perplexity and better downstream tasks performance than state-of-the-art low-rank training methods across different settings and narrows the gap with full-rank baselines.
    </blockquote>

    <!--CONTEXT-->
    <div class="metatable">
      <table summary="Additional metadata">        <tr>
          <td class="tablecell label">Comments:</td>
          <td class="tablecell comments mathjax">Code is available at <a href="https://github.com/kyleliang919/Online-Subspace-Descent" rel="external noopener nofollow" class="link-external link-https">this https URL</a></td>
        </tr>
<tr>
          <td class="tablecell label">Subjects:</td>
          <td class="tablecell subjects">
            <span class="primary-subject">Machine Learning (cs.LG)</span>; Artificial Intelligence (cs.AI); Computation and Language (cs.CL)</td>
        </tr><tr>
          <td class="tablecell label">Cite as:</td>
          <td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2408.12857">arXiv:2408.12857</a> [cs.LG]</span></td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxividv">(or <span class="arxivid">
              <a href="https://arxiv.org/abs/2408.12857v1">arXiv:2408.12857v1</a> [cs.LG]</span> for this version)
          </td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxivdoi">              <a href="https://doi.org/10.48550/arXiv.2408.12857"  id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2408.12857</a><div class="button-and-tooltip">
              <button class="more-info" aria-describedby="more-info-desc-1">
                <svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
                <span class="visually-hidden">Focus to learn more</span>
              </button>
              <!-- tooltip description -->
              <div role="tooltip" id="more-info-desc-1">
                <span class="left-corner"></span>                  arXiv-issued DOI via DataCite</div>
            </div>
          </td>
        </tr></table>
    </div>
  </div>
</div>
    <div class="submission-history">
      <h2>Submission history</h2> From: Kaizhao Liang [<a href="/show-email/0d22742d/2408.12857">view email</a>]      <br/>    <strong>[v1]</strong>
        Fri, 23 Aug 2024 05:54:53 UTC (669 KB)<br/>
</div>
  </div>
  <!--end leftcolumn-->
<div class="extra-services">    <div class="full-text">
      <a name="other"></a>
      <span class="descriptor">Full-text links:</span>
      <h2>Access Paper:</h2>
      <ul>
  <div id="download-button-info" hidden>
View a PDF of the paper titled Memory-Efficient LLM Training with Online Subspace Descent, by Kaizhao Liang and 3 other authors</div><li><a href="/pdf/2408.12857" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2408.12857v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2408.12857" class="abs-button download-eprint">TeX Source</a></li><li><a href="/format/2408.12857" class="abs-button download-format">Other Formats</a></li></ul>
      <div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
          <img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
          <span>view license</span>
        </a></div>
    </div>
    <!--end full-text-->    <div class="browse">
    Current browse context: <div class="current">cs.LG</div>

  <div class="prevnext">
<span class="arrow">
      <a class="abs-button prev-url" href="/prevnext?id=2408.12857&amp;function=prev&amp;context=cs.LG"
         accesskey="p" title="previous in cs.LG (accesskey p)">&lt;&nbsp;prev</a>
    </span>
    <span class="is-hidden-mobile">&nbsp; | &nbsp;</span>    <span class="arrow">
      <a class="abs-button next-url" href="/prevnext?id=2408.12857&amp;function=next&amp;context=cs.LG" accesskey="n"
         title="next in cs.LG (accesskey n)">next&nbsp;&gt;</a>
    </span><br/>
  </div><div class="list">
    <a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.LG/new">new</a>
    <span class="is-hidden-mobile"> | </span>
    <a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.LG/recent">recent</a>
    <span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.LG/2024-08">2024-08</a>
  </div><div class="abs-switch-cat">
    Change to browse by:
    <div class="switch context-change">
        <a href="/abs/2408.12857?context=cs">cs</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.12857?context=cs.AI">cs.AI</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.12857?context=cs.CL">cs.CL</a><br class="is-hidden-mobile">
    </div>
  </div>

    </div>

    <div class="extra-ref-cite">
      <h3>References &amp; Citations</h3>
      <ul>
        <li><a  class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2408.12857">NASA ADS</a></li><li><a  class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2408.12857" target="_blank" rel="noopener">Google Scholar</a></li>
        <li><a  class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2408.12857" target="_blank" rel="noopener">Semantic Scholar</a></li>
      </ul>
      <div style="clear:both;"></div>
    </div>

<div class='extra-ref-cite'>
    <a id='bib-cite-css' hidden='true' href='/static/browse/0.3.4/css/cite.css'>a</a>

    <span id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</span>
    <span id='bib-cite-loading' hidden='true'>Loading...</span>
</div>

<div id='bib-cite-modal' class='bib-modal' hidden='true'>
    <div class='bib-modal-content'>
        <div class='bib-modal-title'>
            <h2>BibTeX formatted citation</h2>
            <span class='bib-modal-close' >&times;</span>
        </div>
        <div>
            <textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
        </div>
        <div>
            <span>Data provided by: </span>
            <a id='bib-cite-source-api'></a>
        </div>
    </div>
</div><div class="bookmarks">
  <div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2408.12857&amp;description=Memory-Efficient LLM Training with Online Subspace Descent"
     title="Bookmark on BibSonomy">
    <img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
         alt="BibSonomy logo"/>
  </a>
  <a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2408.12857&amp;title=Memory-Efficient LLM Training with Online Subspace Descent"
     title="Bookmark on Reddit">
    <img src="/static/browse/0.3.4/images/icons/social/reddit.png"
         alt="Reddit logo"/>
  </a>
</div>  </div>
  <!--end extra-services-->
<!-- LABS AREA -->
<div id="labstabs">
  <div class="labstabs"><input type="radio" name="tabs" id="tabone"checked="checked">
    <label for="tabone">Bibliographic Tools</label>
    <div class="tab labs-display-bib">
      <h1>Bibliographic and Citation Tools</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input id="bibex-toggle" type="checkbox" class="lab-toggle">
              <span class="slider"></span>
              <span class="is-sr-only">Bibliographic Explorer Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
          </div>
        </div>
<div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="litmaps-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
                aria-labelledby="label-for-litmaps">
              <span class="slider"></span>
              <span class="is-sr-only">Litmaps Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
          </div>
        </div>
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="scite-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
                aria-labelledby="label-for-scite">
              <span class="slider"></span>
              <span class="is-sr-only">scite.ai Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
          </div>
        </div>
      </div>
        <div class="labs-content-placeholder labs-display" style="display: none;"></div>
        <div style="min-height: 15px" id="litmaps-open-in"></div>
        <div style="min-height: 15px" id="scite-open-in"></div>
    </div>


    <input type="radio" name="tabs" id="tabtwo">
    <label for="tabtwo">Code, Data, Media</label>
    <div class="tab">
      <h1>Code, Data and Media Associated with this Article</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input        
                id="catalyzex-toggle"
                data-script-url="/static/browse/0.3.4/js/catalyzex.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="dagshub-toggle"
                data-script-url="/static/browse/0.3.4/js/dagshub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
              <span class="slider"></span>
              <span class="is-sr-only">DagsHub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
          </div>
        </div>
  
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="gotitpub-toggle"
                data-script-url="/static/browse/0.3.4/js/gotitpub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
              <span class="slider"></span>
              <span class="is-sr-only">GotitPub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="paperwithcode-toggle"
                data-script-url="/static/browse/0.3.4/js/paperswithcode.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-pwc">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-pwc">Papers with Code</span> <em>(<a href="https://paperswithcode.com/" target="_blank">What is Papers with Code?</a>)</em>
          </div>
        </div>


        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="sciencecast-toggle"
                data-script-url="/static/browse/0.3.4/js/sciencecast.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
              <span class="slider"></span>
              <span class="is-sr-only">ScienceCast Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
          </div>
        </div>
      </div>

      <div id="catalyzex-output" style="display:none"></div>
      <div id="dagshub-output" style="display:none"></div>
      <div id="gotitpub-output" style="display:none"></div>
      <div id="pwc-output" style="display:none"></div>
      <div id="pwc-data-output" style="display:none"></div>
      <div id="sciencecast-output" style="display:none"></div>
    </div>


      <input type="radio" name="tabs" id="labstabs-demos-input">
      <label for="labstabs-demos-input" id="labstabs-demos-label">Demos</label>
      <div class="tab">
        <h1>Demos</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="replicate-toggle"
                  data-script-url="/static/browse/0.3.4/js/replicate.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
                <span class="slider"></span>
                <span class="is-sr-only">Replicate Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="spaces-toggle"
                  data-script-url="/static/browse/0.3.4/js/spaces.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="txyz-toggle"
                  data-script-url="/static/browse/0.3.4/js/txyz.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
            </div>
          </div>
        </div>
        <div id="replicate-output"></div>
        <div id="spaces-output"></div>
        <div id="txyz-output"></div>
      </div>
      <input type="radio" name="tabs" id="tabfour">
      <label for="tabfour">Related Papers</label>
      <div class="tab">
        <h1>Recommenders and Search Tools</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="influenceflower-toggle"
                data-script-url="/static/browse/0.3.4/js/influenceflower.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
                <span class="slider"></span>
                <span class="is-sr-only">Link to Influence Flower</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="connectedpapers-toggle" type="checkbox" class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
                       aria-labelledby="label-for-connected-papers" >
                <span class="slider"></span>
                <span class="is-sr-only">Connected Papers Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
                <span class="slider"></span>
                <span class="is-sr-only">Core recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
            </div>
          </div>          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="iarxiv-toggle"
                       type="checkbox"
                       class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/iarxiv.js"
                       aria-labelledby="label-for-iarxiv">
                <span class="slider"></span>
                <span class="is-sr-only">IArxiv recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-iarxiv">IArxiv Recommender</span>
              <em>(<a href="https://iarxiv.org/about">What is IArxiv?</a>)</em>
            </div>
          </div>
</div>
        <div id="influenceflower-output"></div>
        <div id="influenceflower-output-graph" style="display:none">
          <ul class="flower-tabs">
            <li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
          </ul>
          <div class="flower-tab-content">
            <div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
            <div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
            <div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
            <div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
          </div>
        </div>
        <div style="min-height: 15px" id="connectedpapers-output"></div>
        <div id="coreRecommenderOutput"></div>
        <div id="iarxivOutput"></div>
      </div>

      <input type="radio" name="tabs" id="tabfive">
      <label for="tabfive">
        About arXivLabs
      </label>
      <div class="tab">
        <div class="columns">
          <div class="column">
            <h1>arXivLabs: experimental projects with community collaborators</h1>
            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
            <p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
          </div>
          <div class="column is-narrow is-full-mobile">
            <p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
          </div>
        </div>
      </div>

    </div>
</div>
<!-- END LABS AREA -->
  <div class="endorsers">
    <a href="/auth/show-endorsers/2408.12857" class="endorser-who">Which authors of this paper are endorsers?</a> |
    <a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
    <span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
  </div>
  <script type="text/javascript" language="javascript">mathjaxToggle();</script>
</div>
      </div>
    </main>

    <footer style="clear: both;">
      <div class="columns is-desktop" role="navigation" aria-label="Secondary" style="margin: -0.75em -0.75em 0.75em -0.75em">
        <!-- Macro-Column 1 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/about">About</a></li>
                <li><a href="https://info.arxiv.org/help">Help</a></li>
              </ul>
            </div>
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>contact arXiv</title><desc>Click here to contact arXiv</desc><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>
                  <a href="https://info.arxiv.org/help/contact.html"> Contact</a>
                </li>
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>subscribe to arXiv mailings</title><desc>Click here to subscribe</desc><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
                  <a href="https://info.arxiv.org/help/subscribe"> Subscribe</a>
                </li>
              </ul>
            </div>
          </div>
        </div>
        <!-- End Macro-Column 1 -->
        <!-- Macro-Column 2 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/license/index.html">Copyright</a></li>
                <li><a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy Policy</a></li>
              </ul>
            </div>
            <div class="column sorry-app-links">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/web_accessibility.html">Web Accessibility Assistance</a></li>
                <li>
                  <p class="help">
                    <a class="a11y-main-link" href="https://status.arxiv.org" target="_blank">arXiv Operational Status <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 512" class="icon filter-dark_grey" role="presentation"><path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"/></svg></a><br>
                    Get status notifications via
                    <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/email/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>email</a>
                    or <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/slack/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-black" role="presentation"><path d="M94.12 315.1c0 25.9-21.16 47.06-47.06 47.06S0 341 0 315.1c0-25.9 21.16-47.06 47.06-47.06h47.06v47.06zm23.72 0c0-25.9 21.16-47.06 47.06-47.06s47.06 21.16 47.06 47.06v117.84c0 25.9-21.16 47.06-47.06 47.06s-47.06-21.16-47.06-47.06V315.1zm47.06-188.98c-25.9 0-47.06-21.16-47.06-47.06S139 32 164.9 32s47.06 21.16 47.06 47.06v47.06H164.9zm0 23.72c25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06H47.06C21.16 243.96 0 222.8 0 196.9s21.16-47.06 47.06-47.06H164.9zm188.98 47.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06h-47.06V196.9zm-23.72 0c0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06V79.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06V196.9zM283.1 385.88c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06v-47.06h47.06zm0-23.72c-25.9 0-47.06-21.16-47.06-47.06 0-25.9 21.16-47.06 47.06-47.06h117.84c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06H283.1z"/></svg>slack</a>
                  </p>
                </li>
              </ul>
            </div>
          </div>
        </div> <!-- end MetaColumn 2 -->
        <!-- End Macro-Column 2 -->
      </div>
    </footer>
  </div>

  <script src="/static/base/1.0.1/js/member_acknowledgement.js"></script>

</body>

</html>
2408.12857
 Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimizer states into a subspace using projection matrix found by singular value decomposition (SVD). However, convergence of these algorithms is highly dependent on the update rules of their projection matrix. In this work, we provide the *first* convergence guarantee for arbitrary update rules of projection matrix. This guarantee is generally applicable to optimizers that can be analyzed with Hamiltonian Descent, including most common ones, such as LION, Adam. Inspired by our theoretical understanding, we propose Online Subspace Descent, a new family of subspace descent optimizer without SVD. Instead of updating the projection matrix with eigenvectors, Online Subspace Descent updates the projection matrix with online PCA. Online Subspace Descent is flexible and introduces only minimum overhead to training. We show that for the task of pretraining LLaMA models ranging from 60M to 7B parameters on the C4 dataset, Online Subspace Descent achieves lower perplexity and better downstream tasks performance than state-ofthe-art low-rank training methods across different settings and narrows the gap with full-rank baselines.1 
abstract
Memory-Efficient Llm Training With Online Subspace Descent
Abstract
2408.12857
 The continual advancement in training large language models (LLMs) presents a compelling challenge in balancing computational efficiency with model performance. As the scope and complexity of these models grow, so does the necessity for innovative strategies that optimize memory usage without compromising the learning capabilities of the model. Recent approaches in low-rank adaptation strategies, including Stochastic Subspace Descent [13], LoRA [11], ReLoRA [15], Gradient LowRank Projection (GaLore) [25] and Sketchy [9] , have paved the way for memory-efficient training by utilizing a periodically updated low-rank projection matrix to manage parameter updates. In particular, GaLore and Sketchy both utilize expensive singular value decomposition to determine the projection matrix, whereas stochastic subspace descent suggests using random matrices as projection matrices and provides convergence analysis on convex functions and objectives. However, to the best of our knowledge, no one has offered any guarantee of convergence for this class of methods on non-convex functions and objectives.  In this work, we provide the first convergence guarantee for arbitrary update rules of the projection matrix. This guarantee is significant because it is broadly applicable to a wide range of optimizers that can be analyzed within the Hamiltonian descent framework [18]. By establishing this convergence guarantee, we demonstrate that our approach is not limited to specific or narrowly defined update rules, but can be extended to include many commonly used optimizers in the field. In particular, this Algorithm 1 Online Subspace Descent 1: Required: Optimizer OptimizerW, learning rate ϵW t , weight decay λW for model weights Wt; and {OptimizerP, ϵ P t , λ P } for the projection matrix P t. Proper initialization.  2: for iteration t do 3: Calculate gradient Gt = ∇L(Wt); Update model weights Wt by (∆ˆ t, Sˆt) = OptimizerW(P ⊤ t Gt,Sˆt−1), Wt+1 = Wt + ϵ W t (P t∆ˆ t+1 − λ WWt) 4: Calculate GP t = ∇LGt (P t) for LGt (·) in Eq (6); Update the projection P t by (∆P t ,S P t ) = OptimizerP(GP t , S P t−1 ), P t+1 = P t + ϵ P t (∆P t − λ P P t) 5: **end for** 6: Remark: We added weight decay as a common heuristic. We recommend using Adam for both optimizers, and set ϵ P t = αϵW t with a constant α (e.g., α = 5), and λW = λ P .  2 includes popular algorithms such as LION [4] and Adam [12], which are widely used in various machine learning and optimization tasks. Our results therefore offer a robust theoretical foundation for understanding and analyzing the behavior of these optimizers, ensuring their effectiveness and reliability in diverse applications.  Inspired by our theoretical understanding, we introduce a novel family of memory-efficient optimizers named Online Subspace Descent, which incorporates a dynamically changing projection matrix, replacing the conventional periodic updating approach (SVD) with online PCA. By allowing the projection matrix to evolve in response to the changing gradient landscape, Online Subspace Descent enhances the model's ability to navigate the parameter space more effectively. This dynamic adaptation aligns more closely with the natural progression of learning in deep neural networks, which is characterized by changes in the importance of different characteristics and interactions as training progresses. Through extensive experiments and comparative analysis, we demonstrate that our approach presents lower perplexity in pretraining LLaMA models (ranging from 60M to 1B parameters) on the C4 dataset compared to state-of-the-art low-rank training methods, closing the perplexity gap with full-rank baselines on language model pretraining. 
introduction
Memory-Efficient Llm Training With Online Subspace Descent
Introduction
2408.12857
 The training of deep learning models reduces to an optimization problem min W L(W), where W is the set of weight matrices of the model. For simplicity, we assume W ∈ R n×m is a single matrix of size (*n, m*) without loss of generality. For notation, we write ⟨A, B⟩ = tr(A ⊤B) for inner products of matrices, and ∥A∥ 2 = tr(A ⊤A) the Frobenius norm. We use A ⊙ B to denote the elementwise product, and A⊙2 = A ⊙ A.  Example 2.1. Update rules of common optimizers: Gradient Descent : Wt+1 = Wt − ϵt∇L(Wt), Momentum : Wt+1 = Wt − ϵtMt, Mt = (1 − β)∇L(Wt) + βMt−1, Lion-K [4] : Wt+1 = Wt − ϵt∇K(Nt), Nt = (1 − β1)∇L(Wt) + β1Mt Mt = (1 − β2)∇L(Wt) + β2Mt−1, Adam [12] : Wt+1 = Wt − ϵtMt √V t + e , Mt = (1 − β1t)∇L(Wt) + β1tMt−1, V t = (1 − β2t)∇L(Wt) ⊙2 + β2tV t−1, where ϵt are step sizes, and Mt,V t are the first and second order momentum, and β, β1, β2 are momentum coefficients in (0, 1)*, with* βit = βi−β t+1 i 1−β t+1 i , i = 1, 2 for β1, β2 ∈ (0, 1) *in Adam, and* K is any convex function with ∇K(0) = 0 for Lion-K *[4], and Lion [5] uses* K(X) = ∥X∥1,1 and ∇K(X) = sign(X).  These optimizers can be unifiedly viewed as updating Wt together with an optimizer state St: Wt+1 = Wt + ϕt(St), St = ψt(St−1, ∇L(Wt)), (1) with some mapping ϕt, ψt. We have St = Mt for momentum and St = {Mt,V t} for Adam.  Note that both Mt,V t are of the same size as the model weights Wt, resulting in high memory consumption for large models. This issue is particularly pronounced for Adam, which typically yields the best performance for large language models (LLMs) but incurs the highest memory cost due to the need to maintain both Mt and V t. One key challenge is to retain the high performance of Adam while enhancing its memory efficiency.  Hamiltonian+Descent One powerful approach to studying the dynamic properties of optimizers is to examine their continuous-time ODE forms in the limit of infinitesimal step size. The continuoustime forms provide clearer insights into the asymptotic convergence of the algorithm, abstracting away the choices of step size, discretization, and stochastic errors. The underlying logic is that a "sound" optimizer should be guaranteed to converge to local optima of the loss, at least when using sufficiently small step sizes.  Inspired by [4, 18], we observe that the continuous-time form of many common optimizers yields a Hamiltian+Descent structure,  $${\mathrm{(2)}}$$ $$\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{W}_{t}=\partial_{\mathbf{S}}H(\mathbf{W}_{t},\mathbf{S}_{t})-\Phi(\partial_{\mathbf{W}}H(\mathbf{W}_{t},\mathbf{S}_{t}))$$ $$\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{S}_{t}=-\partial_{\mathbf{W}}H(\mathbf{W}_{t},\mathbf{S}_{t})-\Psi(\partial_{\mathbf{S}}H(\mathbf{W}_{t},\mathbf{S}_{t})),$$  where $H(\mathbf{W},\mathbf{S})$ is a Hamiltonian (or Lyapunov) function that satisfies  $$\min_{\mathbf{S}}H(\mathbf{W},\mathbf{S})=L(\mathbf{W}),\quad\forall\mathbf{W},$$ 3 so that minimizing L(W) reduces to minimizing H(W,S); and Φ(·), Ψ(·) are two monotonic mappings satisfying ∥X∥ 2 Φ := ⟨X, Φ(X)⟩ ≥ 0, ∥X∥ 2 Ψ := ⟨X, Ψ(X)⟩ ≥ 0, ∀X.  With Φ(X) = Ψ(X) = 0, the system in (2) reduces to the standard Hamiltonian system that keeps H(Wt,St) = *const* along the trajectory. When adding the descending components with Φ and Ψ, the system then keeps H(W,S) monotonically non-decreasing:  d dt H(Wt,St) = ∂W Ht, d dt Wt + ∂SHt, d dt St  (3) = ⟨∂W Ht, ∂SHt − Φ(∂W Ht)⟩ + ⟨∂SHt, −∂W Ht − Ψ(∂SHt)⟩ = − ∥∂W Ht∥ 2 Φ − ∥∂SHt∥ 2 Ψ ≤ 0, where we write ∂W Ht = ∂W H(Wt,St) and similarly for ∂SHt. The main idea is that the cross  terms ⟨∂W Ht, ∂SHt⟩ are canceled, leaving only the negative terms. Example 2.2. *The momentum method yields following continuous-time form and Hamiltonian:* d dt Wt = −Mt,d dt Mt = a(∇L(Wt) − Mt), with H(W,M) = L(W) + ∥M∥ 2 2a. Example 2.3. Adam [12] yields the following continuous-time form and Hamiltonian, d dt Wt = −Mt √V t + e ,d dt Mt = a(∇L(Wt) − Mt),d dt V t = b(∇L(Wt) ⊙2 − V t), with H(W,M,V ) = L(W) + 1 2a  M √V + e , M , for which we can show that d dtH(Wt,Mt,V t) ≤ 0 *when* a ≥ b/4.  Example 2.4. The Lion-K optimizer [5, *4] (without weight decay) can be written into* d dt Wt = ∇K((1 − b)Mt − b∇L(Wt)),d dt Mt = −a(∇L(Wt) + Mt), where a ≥ 0, b ∈ [0, 1] and K(X) *is any convex function that attains the minimum at* X = 0*. One* of its Hamiltonians that yields the Hamiltonian+descent structure (Eq (13) in Chen et al. [4]) is    **Remark 1.** The $\mathcal{H}$-function $\mathcal{H}$ is defined as  $$H(\boldsymbol{W},\boldsymbol{M})=aL(\boldsymbol{W})+\frac{1}{1-b}\mathcal{K}((1-b)\boldsymbol{M}).$$ 
body
Memory-Efficient Llm Training With Online Subspace Descent
Optimization Background
2408.12857
 We introduce the idea of constructing memory efficient optimzers by descending in the subspaces that dynamically changes across iterations as motivated by GaLore [25] and Sketchy [9]. We first derive static subspace descent by restricting the whole optimization on a subspace (Section 3.1), and then propose to dynamically change the subspace across iterations as a heuristic to attain the optimization in the full space while only using subspace descent (Section 3.2). In particular, we propose to update the subspaces via continuous online PCA like updates to avoids the need of exact SVD like in GaLore and Sketchy (Section 3.2). Finally, we remark in Section 3.3 the heuristic nature of the derivation of the method and highlight the difficulty in theoretical understanding, which motivates our analysis based on Hamiltonian dynamics in Section 4. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Memory-Efficient Optimizers Via Online Subspace Descent
2408.12857
 One popular approach to improving memory efficiency is to confine the optimization to a lowdimensional space. To do this, we impose a low rank structure of W = PWˆ , where P ∈ R n×kis a projection matrix to be determined later and Wˆ ∈ R k×m is a dimension-reduced parameter. When k ≪ n, P and Wˆ are much smaller in size compared to W. Now consider  $${}_{i}({\hat{\mathbf{S}}}_{t}),$$  min Wˆ L(PWˆ ).  Applying the optimizer from (1) to update Wˆ along with an optimizer state Sˆ, and mapping the update rule Wˆ t+1 = Wˆ t + ϕt(Sˆt) to that of W = PWˆ t, we get Wt+1 = Wt + P ϕt(Sˆt), Sˆt = ψt(Sˆt−1, P ⊤∇L(Wt)), (4) where we used the fact that ∇Wˆ L(PWˆ ) = P ⊤∇W L(W). This yields a more memory-efficient optimizer, as the size of Sˆt is proportional to that of Wˆt, much smaller than St in (1) when k ≪ n. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Static Subspace Descent
2408.12857
 With a static P , regardless of its values, the parameter W is restricted to have a low rank structure. Although low rank assumption is proved to be useful for fine-tuning with LoRA-like methods [11], it is often too limited for pre-training or when the desirable model weights are not inherently low-rank. To address this problem, Zhao et al. [25] suggested to keep the projected updated in (4), but use different P across the iterations: Wt+1 = Wt + P tϕt(Sˆt), Sˆt = ψt(Sˆt−1, P ⊤ t ∇L(Wt)), P t+1 = χt(P t,Wt,Sˆt), where χt is a update rule of P t that will be determined in the sequel. The intuition is to open up different projection directions at different iterations, so that optimization can be conducted in different subspaces across different iterations. This is similar to the update of coordinate descent, except in a continuous fashion. Note that the update of P t can be done in parallel with that of (Wt,Sˆt), and incurs no slowdown once it is fast enough to not cause a speed bottleneck. Example 3.1. Examples of common optimizers equipped with online subspace descent: Gradient Descent : Wt+1 = Wt − ϵtP tP ⊤ Adam : Wt+1 = Wt − ϵtP tMˆ p t Vˆ t + e t Gt, Gt = ∇L(Wt), Momentum : Wt+1 = Wt − ϵtP tMˆ t, Mˆ t = (1 − β)P Lion-K : Wt+1 = Wt − ϵtP t∇K(Nˆ t), Gˆt = P Nˆ t = (1 − β1)Gˆt + β1Mˆ t, Mˆ t = (1 − β2)Gˆt + β2Mˆ t−1,  With some transport momentum  $\mathbf{G}_t=\nabla L(\mathbf{W}_t)$,  $\hat{\mathbf{M}}_t=(1-\beta)\mathbf{P}_t^\top\mathbf{G}_t+\beta\hat{\mathbf{M}}_{t-1}$,  $\hat{\mathbf{G}}_t=\mathbf{P}_t^\top\mathbf{G}_t$  $\hat{\mathbf{M}}_t=(1-\beta_2)\hat{\mathbf{G}}_t+\beta_2\hat{\mathbf{M}}_{t-1}$,  $\hat{\mathbf{G}}_t=\mathbf{P}_t^\top\mathbf{G}_t$,  $\hat{\mathbf{V}}_t=(1-\beta_{2t})\hat{\mathbf{G}}_t^{\odot2}+\beta_{2t}\hat{\mathbf{V}}_{t-1}$.  $$t\,D e s c e n t$$  Mˆ t = (1 − β1t)Gˆt + β1tMˆ t−1, Vˆ t = (1 − β2t)Gˆ How Should P t be Updated? It is useful to draw intuition from the projected gradient descent rule Wt+1 = Wt − ϵtP tP ⊤ t Gt, Gt = ∇L(Wt), (5) in which P tP ⊤ t can be viewed as a low rank preconditioning of Gt. To make it follow the exact gradient descent, we hope to make P tP ⊤ t Gt approximate Gt as much as possible. In Galore, this is achieved by performing singular value decomposition (SVD) on Gt periodically every T iterations: P t, _, _ = torch.linalg.svd(GT⌊t/T⌋), where T⌊t/T⌋ is the largest multiple of T less than or equal to t. However, numerical SVD incurs a large computational cost for very large models. Also, since P t is fully determined by GT⌊t/T⌋ calculated from a single mini-batch at the last periodic point, it does not incorporate the gradient information from all data in a timely fashion.  In this work, we propose to update P t in a continuous online fashion that incorporates the most recent gradient information in a timely fashion, without calling torch.linalg.decompositions routines.  We view the update of P t as conducting an online principal component analysis (PCA) based on the streaming of {Gt}. In particular, we propose to update P t at time t by minimizing the following PCA objective:  $$L_{\mathbf{G}_{t}}(\mathbf{P})=\left\|\mathbf{P}\mathbf{P}^{\top}\tilde{\mathbf{G}}_{t}-\tilde{\mathbf{G}}_{t}\right\|^{2}+\lambda\left\|\mathbf{P}^{\top}\mathbf{P}-\mathbf{I}_{k\times k}\right\|^{2},\quad\tilde{\mathbf{G}}_{t}=\frac{\mathbf{G}_{t}}{\|\mathbf{G}_{t}\|},\tag{6}$$  where ∥A∥ = tr(A ⊤A) 1/2and Ik×k is the k × k identity matrix; we introduced an auxiliary loss to encourage the columns of P to be orthonormal and normalizes Gt to increase stability.  The key property of LGt (P ) in (6) is that all its stable local minimum is a global minimum, and P is a global minimum iff P P ⊤G˜t forms the optimal rank-k approximation of G˜t [e.g., 3]; moreover, we have P ⊤P = Ik×k at optima when λ > 0.  Instead of minimizing LGt (P ) exactly, to retain computational efficiency, we propose to update P t by only performing one step of optimization on LGt (P ): P t+1 = OptimizerP.step(P t, ∇P LGt (P t)), where OptimizerP.step can be a favorite optimizer, such as gradient descent or Adam. Note that when using Adam, we introduce a copy of optimizer state S P t for P t. See Algorithm 1. Compared to the exact SVD, each online update of P t here is fast and can be executed in parallel with the (Wt,Sˆt) updates to avoid slowdown. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Online Subspace Descent
2408.12857
 The idea above of projecting an arbitrary optimizer with a dynamically changing P t is heuristically motivated and lacks an immediate rigorous theoretical justification. The main challenge lies in the complex interaction between the update of Ut and the optimization state St, which could potentially degrade the convergence and other theoretical properties of the original optimizer. A key question is whether we can develop a theoretical framework to understand how P t impacts the optimizer's convergence behavior and provide guidance for the design of the update rules of P t.  To gain understanding, it is useful to first exam the simple case of projected gradient descent in (5) which does not have an optimizer state (St = ∅). In this case, since P tP ⊤ t is positive semi-finite, the update P tP ⊤ t Gt is always non-increasing direction of L(W) for any P t. The algorithm is essentially a variant of coordinate or subspace descent, where P t defines the subspace on which one step of gradient descent is conduced at iteration t. To ensure that (5) finds a local optimum, we mainly need to ensure that P tGt = 0 only if Gt = 0 to prevent the optimizer from stopping prematurely; this is a mild condition that can be satisfied e.g. when P t is updated by (online) PCA on Gt.  Unfortunately, this coordinate-descent-like interpretation does not apply to more advanced optimizers that track a momentum state St. This is because St accumulates the information from the projected gradient P τGτ at all earlier iterations τ ≤ t. As P t changes across time, it is unclear whether the gradient projected to different subspaces P τ would be coherent with each other, and useful for future updates that are conducted in different subspaces P t for *t > τ* . The difficulty is the inertia effect of St that entangles the different subspaces, making the dynamic behavior fundamentally more complicated than naive coordinate descent where the descent in different subspaces is uncoupled.  This is what we address in Section 4 via the Hamiltonian descent framework. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Difficulty In Theoretical Understanding
2408.12857
 In this section, we show a surprising result that the complication outlined above in Section 3.3 *is not* a problem for optimizers that yields the Hamiltonian+descent structure in (2). Our result is two-fold: - Section 4.1: When applying Online Subspace Descent on systems in (2), the Hamiltonian+descent structure is preserved once the update rule of P t has a smooth continuous-time limit. Hence, under very mild conditions, Online Subspace Descent equipped with common optimizers like Adam and Lion automatically yield a Lyapunov function and hence benign continuous-time convergence.  Moreover, P t can, in fact, be generalized to an arbitrary linear operator as shown in Section 4.3. - Section 4.2: For any smooth P t update rules that eliminates the degenerate case of P ⊤ t Gt = 0 while Gt = 0 at convergence, the online subspace optimizer guarantees to converge in continuous time to a stationary point of the loss L(W). This mild condition is satisfied, for example, when P t is updated by a typical optimizer on the online PCA objective LGt (P ). 
body
Memory-Efficient Llm Training With Online Subspace Descent
Hamiltonian Descent Meets Subspace Descent: A Lyapunov Analysis
2408.12857
 Applying dynamic projection to Hamiltonian descent in (2), we obtain the following systems:  $$\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{W}_{t}=\mathbf{P}_{t}\partial_{\mathbf{\hat{S}}}H(\mathbf{W}_{t},\hat{\mathbf{S}}_{t})-\Phi(\partial_{\mathbf{W}}H(\mathbf{W}_{t},\hat{\mathbf{S}}_{t}))$$ $$\frac{\mathrm{d}}{\mathrm{d}t}\hat{\mathbf{S}}_{t}=-\mathbf{P}_{t}^{\top}\partial_{\mathbf{W}}H(\mathbf{W}_{t},\hat{\mathbf{S}}_{t})-\Psi(\partial_{\mathbf{\hat{S}}}H(\mathbf{W}_{t},\hat{\mathbf{S}}_{t}))\tag{7}$$ $$\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{P}_{t}=\Gamma(\mathbf{P}_{t},\nabla L(\mathbf{W}_{t})),$$ 6 where Γ specifies the update rule of P t. Following essentially the same derivation as (3), one can show that H(W,S) remains a Lyapunov function of (7), regardless of the choice of Γ:  $$\begin{split}\frac{\mathrm{d}}{\mathrm{d}t}H(\mathbf{W}_{t},\hat{\mathbf{S}}_{t})&=-\left\|\partial_{\mathbf{W}}H_{t}\right\|_{\Phi}^{2}-\left\|\partial_{\mathbf{S}}H_{t}\right\|_{\Psi}^{2}+\left\langle\partial_{\mathbf{W}}H_{t},\mathbf{P}_{t}\partial_{\mathbf{S}}H_{t}\right\rangle-\left\langle\partial_{\mathbf{S}}H_{t},\mathbf{P}_{t}^{\top}\partial_{\mathbf{W}}H_{t}\right\rangle\\ &=-\left\|\partial_{\mathbf{W}}H_{t}\right\|_{\Phi}^{2}-\left\|\partial_{\mathbf{S}}H_{t}\right\|_{\Psi}^{2}\leq0,\end{split}\tag{8}$$  where the key is to use the *adjoint* property of P and P ⊤ that ⟨P tX,Y ⟩ = ⟨X, P ⊤ t Y ⟩, which cancels the crossing terms, independent of the values of P t. There is no requirement on Γ here, besides that the derivative in (8) should exist. As shown in Section 4.3, we can generalize (8) by replacing P t and P ⊤ t with a general linear operator Pt and its adjoint P ∗ t .  The following are examples of continuous-time Momentum, Lion-K and Adam with subspace descent and their Hamiltonian functions.  Example 4.1. *Momentum + Online Subspace Descent is*  $$\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{W}_{t}=-\mathbf{P}_{t}\tilde{\mathbf{M}}_{t},\qquad\qquad\tilde{\mathbf{G}}_{t}=\mathbf{P}_{t}^{\top}\nabla L(\mathbf{W}_{t}),\qquad\qquad\frac{\mathrm{d}}{\mathrm{d}t}\tilde{\mathbf{M}}_{t}=a(\tilde{\mathbf{G}}_{t}-\tilde{\mathbf{M}}_{t}),$$  with Lyapunov function H(W,Mˆ ) = L(W) + ∥Mˆ ∥ 2 2a, *for which*  $$\frac{\mathrm{d}}{\mathrm{d}t}H(\mathbf{W}_{t},\hat{\mathbf{M}}_{t})=-\nabla L(\mathbf{W}_{t})^{\top}\mathbf{P}_{t}\hat{\mathbf{M}}_{t}+\hat{\mathbf{M}}_{t}^{\top}(\mathbf{P}_{t}^{\top}\nabla L(\mathbf{W}_{t})-\hat{\mathbf{M}}_{t})=-\left\|\hat{\mathbf{M}}_{t}\right\|_{2}^{2}\leq0.$$  Example 4.2. *Adam + Online Subspace Descent is*  $${\frac{\mathrm{d}}{\mathrm{d}t}}\mathbf{W}_{t}=\mathbf{P}_{t}{\frac{\hat{\mathbf{M}}_{t}}{\sqrt{\hat{\mathbf{V}}_{t}}+e}},\quad\hat{\mathbf{G}}_{t}=\mathbf{P}_{t}^{\top}\nabla L(\mathbf{W}_{t}),\quad{\frac{\mathrm{d}}{\mathrm{d}t}}\hat{\mathbf{M}}_{t}=a(\hat{\mathbf{G}}_{t}-\hat{\mathbf{M}}_{t}),\quad{\frac{\mathrm{d}}{\mathrm{d}t}}\hat{\mathbf{V}}_{t}=b(\hat{\mathbf{G}}_{t}^{2}-\hat{\mathbf{V}}_{t}).$$ _with Lyapunov function_ $H(\mathbf{W},\mathbf{M},\mathbf{V})=L(\mathbf{W})+\dfrac{1}{2a}\left\langle\dfrac{\hat{\mathbf{M}}}{\sqrt{\hat{\mathbf{V}}}+e},\ \hat{\mathbf{M}}\right\rangle,$ _for which_ d dt H(Wt,Mˆ t,Vˆ t) = − * Gt, P tMˆ p t Vˆ t + e + + 1 a *Mˆ p t Vˆ t + e , a(P ⊤ t Gt − Mˆ t) + − b 4a *Mˆ ⊙2 p t Vˆ t ⊙ ( pVˆ t + e)⊙2 , (Gˆ ⊙2 t − Vˆ t) + = − * 1 − b 4a pVˆ p t Vˆ t + e ,Mˆ ⊙2 p t Vˆ t + e + − b 4a *Mˆ ⊙2 p t Vˆ t ⊙ ( pVˆ t + e)⊙2 , Gˆ ⊙2 t + ≤ − 1 − b 4a  Mˆ q t pVˆ t + e  2 − b 4a  Mˆ tGˆt p4Vˆ t( pVˆ t + e)  2 ≤ 0, where we assume a ≥ b/4. Example 4.3. The Lion-K + Online Subspace Descent is 7 $$\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{W}_{t}=\mathbf{P}_{t}\nabla\mathcal{K}((1-b)\tilde{\mathbf{M}}_{t}-b\tilde{\mathbf{G}}_{t}),\quad\frac{\mathrm{d}}{\mathrm{d}t}\mathbf{M}_{t}=-a(\tilde{\mathbf{G}}_{t}+\tilde{\mathbf{M}}_{t}),\quad\tilde{\mathbf{G}}_{t}=\mathbf{P}_{t}^{\top}\nabla L(\mathbf{W}_{t}).$$  _Consider the Hamiltonian function in Eq (13) of [-1]:_  $$H(\mathbf{W},\tilde{\mathbf{M}})=aL(\mathbf{W})+\frac{1}{1-b}\mathcal{K}((1-b)\tilde{\mathbf{M}}).$$ d dt H(Wt,Mt) = a⟨Gt, P t∇K((1 − b)Mˆ t − bGˆt)⟩ − a⟨∇K((1 − b)Mˆ t), Gˆt + Mˆ t⟩ = a⟨Gˆt, ∇K((1 − b)Mˆ t − bGˆt) − ∇K((1 − b)Mˆ t)⟩ − a⟨∇K((1 − b)Mˆ t),Mˆ t⟩ = − a b [(1 − b)Mˆ t; − bGˆt]∇K −a (1 − b) [0; (1 − b)Mˆ t]∇K ≤ 0 where we defined [X;Y ]∇K = ⟨Y , ∇K(X +Y )− ∇K(X)⟩ and used the fact that [X;Y ]∇K ≥ 0 by the convexity of K*; we used* ⟨Gt, P tXt⟩ = ⟨P ⊤ t Gt, Xt⟩ = ⟨Gˆt, Xt⟩. In all examples above, although the form of Hamiltonian H(W,Sˆ) is independent of the update rule of P t, the decreasing rate d dtH(Wt,Sˆt) depends on P t in a complicate way through Mˆ t,Vˆ t, Gˆt. An interesting direction for future investigation is to find optimal rules of P t to maximize the decreasing rate as an optimal control problem. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Online Subspace Descent Preserves The Hamiltonian+Descent Structure
2408.12857
 In addition to the Lyapunov structure, we need an additional mild condition on the update rule of P t to ensure the system converges to the local optimum of the loss L(W). The main idea is to prevent the system from stopping prematurely before reaching zero gradient Gt = 0 by excluding the degenerate case of P tGt = 0 while Gt ̸= 0 in the invariant set of the system.  Assumption 4.4. *Assume the functions in system* (7) are continuously differentiable and i) d dtH(Wt,Sˆt) = 0 *implies* Gˆt = P ⊤ t ∇L(Wt) = 0 and d dtWt = 0.  ii) When Gt ≡ G ̸= 0, the set {P : P ⊤G = 0} *is not a positive invariant set of* d dtP t = Γ(P t, Gt).  This is a mild condition. Assumption i) says that the optimizer should stop at a point with Gˆt = 0, which is easy to verify for the common optimizers like momentum, Adam, Lion-K. Assumption ii) ensures Gˆt = 0 would imply Gt = 0 in invariance sets, which is satisfied when for example, P t is updated by a reasonable optimizer of the online PCA loss that converges to a stable local minimum.  Theorem 4.5. Assume Assumption 4.4 holds. Let (Wt,St, P t)t *be a bounded solution of* (7), then all the accumulation points {Wt} as t → +∞ *are stationary points of* L(W).  Proof. By LaSalle's invariance principle, the positive limit set of (Wt,St, P t)t must be contained in I, where I = {the union of complete trajectories satisfying d dtH(Wt,Sˆt) = 0, ∀t }.  From the Assumption i), the trajectories contained in I must satisfy d dtWt = 0, which implies d dtGt = d dt∇L(Wt) = 0 and Gˆt = 0 and hence Gt ≡ G is a constant with P ⊤ t G = 0. Moreover, from Assumption ii), we must have ∇L(Wt) = Gt ≡ 0, since otherwise the trajectory is not invariant. As a result, all trajectories in the limit set I must have ∇L(Wt) = 0. Because d dtWt = 0, these trajectories are static points of Wt. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Convergence To Local Optima
2408.12857
 We can generalize the online subspace descent with general linear operators: d dt Wt = Pt(∂SˆH(Wt,Sˆt)) − Φ(∂W H(Wt,Sˆt)) d dt Sˆt = −P∗ t (∂W H(Wt,Sˆt)) − Ψ(∂SˆH(Wt,Sˆt)) d dt Pt = Γ(Pt, ∇L(Wt)), where we generalize P t to be any linear operator Pt with an adjoint operator P ∗ t , satisfying ⟨X,Pt(Y )⟩ = ⟨P∗ t (X),Y ⟩, ∀X,Y .  The derivation of Lyapunov follows a similar way: d dt H(Wt,Sˆt) = − ∥∂W Ht∥ 2 Φ − ∥∂SHt∥ 2 Ψ + ⟨∂W Ht,Pt(∂SˆHt)*⟩ − ⟨*∂SˆHt,P ∗ t (∂W Ht)⟩ = − ∥∂W Ht∥ 2 Φ − ∥∂SHt∥ 2 Ψ ≤ 0, where the crossing terms are again canceled due to the adjoint property.  As an example of the general framework, consider Pt(X) = P tXQt , where Qt is another projection matrix applied on the different dimension of X (see also [25]). The adjoint operator of Pt is P ∗ t (X) = P ⊤ t XQ⊤ t . This can be verified by ⟨P tXQt ,Y ⟩ = tr(P tXQtY ⊤) = tr(XQtY ⊤P t) = tr(X(P ⊤ t Y Q⊤ t ) ⊤) = ⟨X, P ⊤ t Y Q⊤ t ⟩.  The subspace descent system of this operator is d dt Wt = P t∂SˆH(Wt,Sˆt)Qt − Φ(∂W H(Wt,Sˆt)) d dt Sˆt = −P ⊤ t ∂W H(Wt,Sˆt))Q ⊤ t − Ψ(∂SˆH(Wt,Sˆt)) d dt P t = ΓP (P t, Qt , ∇L(Wt)) d dt Qt = ΓQ(P t, Qt , ∇L(Wt)), where P t, Qt can be updated jointly via an online SVD on Gt.  Another linear operator that involves two matrices is Pt(X) = P tX+XQt , which yields P ∗ t (X) = P ⊤ t X + XQ⊤ t . 
body
Memory-Efficient Llm Training With Online Subspace Descent
Online Subspace Descent With General Linear Projection Operators
2408.12857
 We answer a number of key questions with pretraining experiments of LLaMA [22] on the C4 dataset [20]. All experiments except for large 7B experiments are conducted on a *single* NVIDIA A100 GPU. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Experiment
2408.12857
 Overall, Online Subspace Descent offers two major advantages over previous methods that rely on SVD, better convergence and lower overhead. In this section, we discuss both in detail. First, Online Subspace Descent closes the gap between the state-of-the-art low-rank method and full rank baseline uniformly across different model sizes, as shown in figure 1. A highlight amongst these results is LLaMA 1B (SS 256). As shown in table 1, Online Subspace Descent attains significant improvement over GaLore in perplexity, while consuming a similar amount of GPU memory (8.64 GB v.s 9.01 GB). One additional observation in 1 shows as model size and sequence length grow, Online Subspace Descent becomes more effective. We hypothesize that this is due to the higher intrinsic rank of the underlying optimization problem in larger models. Hence, the positive impact on the convergence of the online update of P t becomes more obvious. See more details in Appendix A.  | Method                 | Perplexity(↓)   |       |       | |------------------------|-----------------|-------|-------| | 60M                    | 350M            | 1B    |       | | 8bit-AdamW (Full Rank) | 32.75           | 30.43 | 29.40 | | GaLore (Rank = 512)    | 57.03           | 44.34 | 35.52 | | Ours (Rank = 512)      | 56.12           | 43.67 | 31.30 |  ![8_image_0.png](8_image_0.png)  Another favorable characteristic of Online Subspace Descent is its minimum overhead. In figure 2,  ![8_image_1.png](8_image_1.png) we measure and analyze the execution time of SVD and online PCA on a popular data center GPU (A100) and a consumer GPU (RTX 3090). The typical Pytorch implementation of SVD can be up to 142 times slower than running a single-step online PCA on representative weight tensors from LLaMA architectures. Online PCA is fast because it is implemented as a single optimization step with respect to a simple loss function. Hence, each step of online PCA can be cleverly scheduled and hidden in the weight optimization step when executed in parallel, whereas SVD is too expensive to be hidden.  Figure 2: The execution time of torch.svd and that a single-step backward() call for online PCA in PyTorch, on matrices of typical shapes in linear layers in the LLaMA 60M to 7B. Thanks to the high speed of single-step online PCA, P t updates can be executed in parallel with weight updates, adding no overhead to the training process. In contrast, SVD incurs significant overhead as the model and weight tensor sizes increase. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Why Do We Need Online Subspace Descent?
2408.12857
 We conduct an ablation study on the rank of Online Subspace Descent. Figure 3 shows that the final perplexity is inversely correlated with rank: higher ranks result in lower convergent perplexity.  However, the rate of reduction of perplexity decreases as the rank increases, eventually reaching a saturation point. We propose an intuitive explanation for this phenomenon. In language modeling, high-frequency tokens can be effectively learned with low-rank training. However, learning lowerfrequency tokens requires higher ranks. Once these lower-frequency tokens are adequately learned, further increasing the rank does not significantly decrease perplexity. In conclusion, given sufficient time and resources, higher ranks yield better performance for Online Subspace Descent. It is recommended that the highest rank be selected until the perplexity reduction saturates. 
body
Memory-Efficient Llm Training With Online Subspace Descent
What Rank Should We Pick For **Online Subspace Descent**?
2408.12857
 α and λ: The parameter α controls the update speed of P t, while λ determines the regularization strength on the optimization objective of P t. Empirically, we find that the result is not sensitive to λ for small models (60M). and set λ = 0.1 for all subsequent experiments. We find that α must be kept  ![9_image_0.png](9_image_0.png) small to avoid instability (Figure 3), and we set α = 5 for all experiments.  Learning rate: For the small model (60M), learning rate choices are more flexible, producing similar results. However, for larger models (350M, 1B), we recommend using a learning rate that is 10 times smaller, specifically 0.001. Larger learning rates cause unrecoverable spikes and instability, a general characteristic observed across all methods. See additional hyperparameter choices in Appendix A. 
body
Memory-Efficient Llm Training With Online Subspace Descent
What Are The Best Hyperparameters?
2408.12857
 One straightforward extension of Online Subspace Descent is to apply it to other base optimizers beyond AdamW8bit. We conduct ablation studies on LION [6] and Adafactor [21], finding that Online Subspace Descent behaves similarly to how it does with AdamW8bit. Despite the initial observation that updating P t with AdamW8bit consistently yields better results, we discover that updating Pt with simple SGD can achieve similar performance.  | Method     | GaLore   | Ours      |             |            |             |       | |------------|----------|-----------|-------------|------------|-------------|-------| | Lion       | Adaf.    | Lion+Lion | Adaf.+Adaf. | Lion+AdamW | Adaf.+AdamW |       | | Perplexity | 46.90    | 34.32     | 57.97       | 47.61      | 44.76       | 34.15 |  Table 2: LLaMA 60M on C4 with sequence length 1024, with optimizers on P t and Wt, denote as "Ours {Wt optimizer} + {P t optimizer}". Adaf., and Adam refer to Adafactor and 8bit-AdamW, respectively. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Can Online Subspace Descent **Be Applied To Different Optimizers?**
2408.12857
 We pretrain from scratch a 7B LLaMA model on the C4 dataset for 10K steps, where the P t matrix is updated by SGD. The perplexity is the lower the better. The final perplexity and training wall-clock time are provided in Table 3. We further provide the downstream evaluation of the pretrained checkpoints using Galore and our method on the GLUE benchmark in Table 4. Our method consistently outperforms Galore when the model size scales up.  | Method   | Perplexity   | Wall Clock Time (hours)   | |----------|--------------|---------------------------| | Galore   | 51.21        | 9.7439                    | | Ours     | 43.72        | 7.1428                    |  Table 3: Perplexity and Wall Clock Time for 7B models pretrained on C4 for 10K steps. Lower perplexity is better. Online Subspace Descent can be upto 1.3x faster than GaLore. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Can Online Subspace Descent Scale To Larger Model?
2408.12857
 We discuss related works on memory-efficient optimization and low-rank adaptation techniques.  | Method   | MRPC   | RTE    | SST2   | MNLI   | QNLI   | QQP    | AVG    | |----------|--------|--------|--------|--------|--------|--------|--------| | Galore   | 0.6838 | 0.5018 | 0.5183 | 0.3506 | 0.4946 | 0.3682 | 0.4862 | | Ours     | 0.6982 | 0.4901 | 0.5233 | 0.3654 | 0.5142 | 0.3795 | 0.4951 |  Low-Rank Adaptation Low-Rank Adaptation (LoRA) [11] adds a low-rank adaptor to speficic linear layers in a model, and finetune only the low-rank adaptor. As the adaptors are small, LoRA is widely applied for finetuning large models. Many variants have been proposed since LoRA, including support for multi-task learning Wang et al. [23] and further memory reductions Dettmers et al. [8]. Notably, Lialin et al. [15] proposed ReLoRA for pretraining, requiring a full-rank training warmup to match standard performance levels. It's important to note that LoRA is fundamentally distinct from subspace descent. While subspace descent optimizes within the original model parameter space, LoRA focuses its optimization efforts within the space of the adaptors.  Memory-Efficient Optimization Several approaches aim to reduce memory costs associated with gradient statistics in adaptive optimization algorithms [21, 2, 7]. In particular, Adafactor [21] factorizes the second-order statistics by a row-column outer product and update the factorized bases on the fly, hence achieving a sub-linear memory cost. K-Fac [19] presents a factorized approximation of the Fisher information matrix which leads to a sublinear natural gradient method. More recently, Feinberg et al. [9] observes that the spectra of the Kronecker-factored gradient covariance matrix in deep learning (DL) training tasks are concentrated on a small leading eigenspace and propose to maintain a matrix preconditioner using the frequent directions sketch. However, their method requires conducting the eigendecomposition at every step, which can be costly for large models.  Other than factorization methods, quantization techniques [7, 1, 24, 16] are also widely used, where the gradient (or the momentum and the preconditioner) are directly quantized to tradeoff performance for memory. Fused gradient computation method [17] have also been used to minimize memory costs during training. GaLore [25] is the most relevant work to ours. GaLore focuses on low-rank gradient structures, reducing memory costs for both first and second-order statistics. Our method can be viewed as a general extension to GaLore where we replace the infrequent SVD by a continuous subspace descent [14, 10]. As a result, our method not only provides a more general framework to study memory-efficient subspace descent, but is also more performant than GaLore in practice. 
body
Memory-Efficient Llm Training With Online Subspace Descent
Related Works
2408.12857
 In conclusion, we provide the first convergence guarantee for arbitrary update rules of projection matrix, applicable to a range of optimizers that can be analyzed using Hamiltonian Descent, including common ones like LION, AdamW, and Adafactor. Inspired by this theoretical foundation, we introduce Dynamic Subspace Descent, a novel family of subspace descent optimizers that eschews SVD in favor of online PCA for updating projection matrix. Dynamic Subspace Descent is both flexible and minimally intrusive, and our experiments show that it achieves lower perplexity in pretraining LLaMA models (ranging from 60M to 1B parameters) on the C4 dataset compared to state-of-the-art low-rank training methods, while also closing the perplexity gap with full-rank baselines.  For future research, we propose several open and intriguing questions: (1) Are there alternative methods for updating projection matrix that could accelerate convergence? (2) What is the impact of weight decay on convergence in Dynamic Subspace Descent? (3) Can low-rank gradients and updates be combined with dynamic low-rank weights (e.g., Mixture of Experts) to further enhance training efficiency? (4) Can this method be applied to problems beyond language modeling? We hope that our work provides a strong foundation for exploring these questions. 
conclusion
Memory-Efficient Llm Training With Online Subspace Descent
Conclusion
2408.12857
 [1] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficient sgd via gradient quantization and encoding. *Advances in neural* information processing systems, 30, 2017.  [2] Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory efficient adaptive optimization. *Advances in Neural Information Processing Systems*, 32, 2019.  [3] Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. *Neural networks*, 2(1):53–58, 1989.  [4] Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu. Lion secretly solves constrained optimization: As lyapunov predicts. *arXiv preprint arXiv:2310.05898*, 2023.  [5] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. *arXiv preprint arXiv:2302.06675*, 2023.  [6] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. *Advances in Neural Information Processing Systems*, 36, 2024.  [7] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. *arXiv preprint arXiv:2110.02861*, 2021.  [8] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. *Advances in Neural Information Processing Systems*, 36, 2024.  [9] Vladimir Feinberg, Xinyi Chen, Y Jennifer Sun, Rohan Anil, and Elad Hazan. Sketchy: Memoryefficient adaptive regularization with frequent directions. Advances in Neural Information Processing Systems, 36, 2024.  [10] Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace.  arXiv preprint arXiv:1812.04754, 2018.  [11] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.  [12] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.  [13] David Kozak, Stephen Becker, Alireza Doostan, and Luis Tenorio. Stochastic subspace descent.  arXiv preprint arXiv:1904.01145, 2019.  [14] Brett W Larsen, Stanislav Fort, Nic Becker, and Surya Ganguli. How many degrees of freedom do we need to train deep networks: a loss landscape perspective. arXiv preprint arXiv:2107.05802, 2021.  [15] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: Highrank training through low-rank updates. In *Workshop on Advancing Neural Network Training:* Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023), 2023.  [16] Bo Liu, Lemeng Wu, Lizhang Chen, Kaizhao Liang, Jiaxu Zhu, Chen Liang, Raghuraman Krishnamoorthi, and Qiang Liu. Communication efficient distributed training with distributed lion. *arXiv preprint arXiv:2404.00438*, 2024.  [17] Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. *arXiv preprint arXiv:2306.09782*, 2023.  [18] Chris J Maddison, Daniel Paulin, Yee Whye Teh, Brendan O'Donoghue, and Arnaud Doucet.  Hamiltonian descent methods. *arXiv preprint arXiv:1809.05042*, 2018.  [19] James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In *International conference on machine learning*, pages 2408–2417. PMLR, 2015.  [20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. *arXiv e-prints*, 2019.  [21] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In *International Conference on Machine Learning*, pages 4596–4604. PMLR, 2018.  [22] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. *arXiv preprint arXiv:2302.13971*, 2023.  [23] Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. Multilora: Democratizing lora for better multi-task learning. *arXiv preprint arXiv:2311.11501*, 2023.  [25] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. *arXiv preprint* arXiv:2403.03507, 2024. [24] Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. *Advances in neural* information processing systems, 30, 2017. 
references
Memory-Efficient Llm Training With Online Subspace Descent
References
2408.12857
 We sweep learning rate from [0.01, 0.005, 0.001]. For GaLore as well as Adam8bit, we follow the recommended hyperparameters as much as possible. For instance, GaLore update gap is set to recommended default, 200. Warmup is set to 10% of the total training steps. Batch size is set to 512 and gradient clipping is set to 1.0. 
body
Memory-Efficient Llm Training With Online Subspace Descent
A Experiments A.1 Hyperparameters
2408.12857
 In the following table, is a sweep on different ranks and their final perplexity of LLaMA 60M (SS = 1024) on C4. All other hyperparameters are fixed and using recommended default. Notice that as the rank increases, both Dynamic Subspace Descent and GaLore improve.  | Rank                                                                                                         | Perplexity (Ours)   | Perplexity (GaLore)   | |--------------------------------------------------------------------------------------------------------------|---------------------|-----------------------| | 32                                                                                                           | 85.90               | 86.16                 | | 128                                                                                                          | 49.01               | 48.05                 | | 512                                                                                                          | 37.41               | 36.93                 | | Full                                                                                                         | 37.18               | 36.51                 | | Table 5: On LLaMA 60M SS 1024, we sweep across different ranks, the trend is clear and intuitive that higher |                     |                       | 
body
Memory-Efficient Llm Training With Online Subspace Descent
A.2 Rank Sweep
2408.12857
 | Method                     | Perplexity   | |----------------------------|--------------| | Lion                       | 52.65        | | Adafactor                  | 33.45        | | Adam8bit                   | 29.77        | | SGD                        | 3469.14      | | Galore LION                | 46.90        | | Galore Adafactor           | 34.32        | | Galore AdamW8bit           | 48.05        | | Ours LION + LION           | 57.97        | | Ours Adafactor + Adafactor | 47.61        | | Ours AdamW8bit + AdamW8bit | 49.01        | | Ours LION + Adam8bit       | 44.76        | | Ours Adafactor + AdamW8bit | 34.15        | | Ours AdamW8bit + SGD       | 53.53        |  Lion 52.65 Adafactor 33.45  Adam8bit 29.77  SGD 3469.14  Galore LION 46.90  Galore Adafactor 34.32  Galore AdamW8bit 48.05 Ours LION + LION 57.97 Ours Adafactor + Adafactor 47.61 Ours AdamW8bit + AdamW8bit 49.01  Ours LION + Adam8bit 44.76 Ours Adafactor + AdamW8bit 34.15  Ours AdamW8bit + SGD 53.53  Table 6: In this experiment, we train LLaMA 60M on C4 with sequence length of 1024. We combine different  base optimizers to update both Pt and Wt, denote as "Ours weight optimizer + Pt optimizer". 
body
Memory-Efficient Llm Training With Online Subspace Descent
A.3 Optimizer Sweep