<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

<head>  <title>[2408.08274] BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
  <link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
  <link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20240822" />
  <link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
  <script language="javascript" src="/static/browse/0.3.4/js/accordion.js" /></script>
  
    <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/slider.css?v=1.14" />
    <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="/static/browse/0.3.4/js/donate.js?v=1.11"></script><link rel="canonical" href="/abs/2408.08274"/><meta property="og:type" content="website" />
<meta property="og:site_name" content="arXiv.org" />
<meta property="og:title" content="BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts" />
<meta property="og:url" content="https://arxiv.org/abs/2408.08274v2" />
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="700" />
<meta property="og:image:alt" content="arXiv logo"/>
<meta property="og:description" content="The Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance over dense models. However, training MoEs from scratch in a large-scale regime is prohibitively expensive. Existing methods mitigate this by pre-training multiple dense expert models independently and using them to initialize an MoE. This is done by using experts&#39; feed-forward network (FFN) to initialize the MoE&#39;s experts while merging other parameters. However, this method limits the reuse of dense model parameters to only the FFN layers, thereby constraining the advantages when &#34;upcycling&#34; these models into MoEs. We propose BAM (Branch-Attend-Mix), a simple yet effective method that addresses this shortcoming. BAM makes full use of specialized dense models by not only using their FFN to initialize the MoE layers but also leveraging experts&#39; attention parameters fully by initializing them into a soft-variant of Mixture of Attention (MoA) layers. We explore two methods for upcycling attention parameters: 1) initializing separate attention experts from dense models including all attention parameters for the best model performance; and 2) sharing key and value parameters across all experts to facilitate for better inference efficiency. To further improve efficiency, we adopt a parallel attention transformer architecture to MoEs, which allows the attention experts and FFN experts to be computed concurrently. Our experiments on seed models ranging from 590 million to 2 billion parameters demonstrate that BAM surpasses baselines in both perplexity and downstream task performance, within the same computational and data constraints."/>
<meta name="twitter:site" content="@arxiv"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="BAM! Just Like That: Simple and Efficient Parameter Upcycling for..."/>
<meta name="twitter:description" content="The Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance over dense models. However, training MoEs from scratch in a..."/>
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
<meta name="twitter:image:alt" content="arXiv logo"/>
  <link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/>  <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script>  <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
  <script src="/static/browse/0.3.4/js/toggle-labs.js?20230622" type="text/javascript"></script>
  <script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts" /><meta name="citation_author" content="Zhang, Qizhen" /><meta name="citation_author" content="Gritsch, Nikolas" /><meta name="citation_author" content="Gnaneshwar, Dwaraknath" /><meta name="citation_author" content="Guo, Simon" /><meta name="citation_author" content="Cairuz, David" /><meta name="citation_author" content="Venkitesh, Bharat" /><meta name="citation_author" content="Foerster, Jakob" /><meta name="citation_author" content="Blunsom, Phil" /><meta name="citation_author" content="Ruder, Sebastian" /><meta name="citation_author" content="Ustun, Ahmet" /><meta name="citation_author" content="Locatelli, Acyr" /><meta name="citation_date" content="2024/08/15" /><meta name="citation_online_date" content="2024/08/17" /><meta name="citation_pdf_url" content="http://arxiv.org/pdf/2408.08274" /><meta name="citation_arxiv_id" content="2408.08274" /><meta name="citation_abstract" content="The Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance over dense models. However, training MoEs from scratch in a large-scale regime is prohibitively expensive. Existing methods mitigate this by pre-training multiple dense expert models independently and using them to initialize an MoE. This is done by using experts&#39; feed-forward network (FFN) to initialize the MoE&#39;s experts while merging other parameters. However, this method limits the reuse of dense model parameters to only the FFN layers, thereby constraining the advantages when &#34;upcycling&#34; these models into MoEs. We propose BAM (Branch-Attend-Mix), a simple yet effective method that addresses this shortcoming. BAM makes full use of specialized dense models by not only using their FFN to initialize the MoE layers but also leveraging experts&#39; attention parameters fully by initializing them into a soft-variant of Mixture of Attention (MoA) layers. We explore two methods for upcycling attention parameters: 1) initializing separate attention experts from dense models including all attention parameters for the best model performance; and 2) sharing key and value parameters across all experts to facilitate for better inference efficiency. To further improve efficiency, we adopt a parallel attention transformer architecture to MoEs, which allows the attention experts and FFN experts to be computed concurrently. Our experiments on seed models ranging from 590 million to 2 billion parameters demonstrate that BAM surpasses baselines in both perplexity and downstream task performance, within the same computational and data constraints." />
</head>

<body  class="with-cu-identity">
  
  <aside class="slider-wrapper bps-banner forum grey">
    <a class="close-slider bps-banner" href="#"><img src="/static/browse/0.3.4/images/icons/close-slider.png" alt="close this message"></a>
  	<div class="columns">
      <img role="presentation" class="bps-banner-image" src="/static/browse/0.3.4/images/events/arxiv-forum-logotype-title.png" alt="arXiv Accessibility Forum 2024">
      <div class="copy-donation bps-banner">
        <h2 id="forum-session-title"></h2>
        <p id="forum-session-link-small"></p>
      </div>
      <div class="amount-donation bps-banner">
          <div id="forum-session-link-box" class="donate-cta"></div>
      </div>
    </div>
  </aside>
  <script src="/static/browse/0.3.4/js/rotate_forum_info.js" type="text/javascript"></script>
  <div class="flex-wrap-footer">
    <header>
      <a href="#content" class="is-sr-only">Skip to main content</a>
      <!-- start desktop header -->
      <div class="columns is-vcentered is-hidden-mobile" id="cu-identity">
        <div class="column" id="cu-logo">
          <a href="https://www.cornell.edu/"><img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University" /></a>
        </div><div class="column banner-minimal forum">
            <!-- /from July 7 at 1:00 AM to Sept 2 at 1:00 AM --><p>This week: the arXiv Accessibility Forum</p>
            <a href="https://accessibility2024.arxiv.org/schedule" target="_blank">Forum Schedule</a></div><div class="column" id="support-ack">
          <span id="support-ack-url">We gratefully acknowledge support from the Simons Foundation, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and all contributors.</span>
          <a href="https://info.arxiv.org/about/donate.html" class="btn-header-donate">Donate</a>
        </div>
      </div>

      <div id="header" class="is-hidden-mobile">
<a aria-hidden="true" href="{url_path('ignore_me')}"></a>
  <div class="header-breadcrumbs is-hidden-mobile">
    <a href="/"><img src="/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg" alt="arxiv logo" style="height:40px;"/></a> <span>&gt;</span> <a href="/list/cs/recent">cs</a> <span>&gt;</span> arXiv:2408.08274
  </div>
          <div class="search-block level-right">
    <form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
      <div class="field has-addons">
        <div class="control">
          <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
          <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
        </div>
        <div class="control">
          <div class="select is-small">
            <select name="searchtype" aria-label="Field to search">
              <option value="all" selected="selected">All fields</option>
              <option value="title">Title</option>
              <option value="author">Author</option>
              <option value="abstract">Abstract</option>
              <option value="comments">Comments</option>
              <option value="journal_ref">Journal reference</option>
              <option value="acm_class">ACM classification</option>
              <option value="msc_class">MSC classification</option>
              <option value="report_num">Report number</option>
              <option value="paper_id">arXiv identifier</option>
              <option value="doi">DOI</option>
              <option value="orcid">ORCID</option>
              <option value="author_id">arXiv author ID</option>
              <option value="help">Help pages</option>
              <option value="full_text">Full text</option>
            </select>
          </div>
        </div>
        <input type="hidden" name="source" value="header">
        <button class="button is-small is-cul-darker">Search</button>
      </div>
    </form>
  </div>
     </div><!-- /end desktop header -->

      <div class="mobile-header">
        <div class="columns is-mobile">
          <div class="column logo-arxiv"><a href="https://arxiv.org/"><img src="/static/browse/0.3.4/images/arxiv-logomark-small-white.svg" alt="arXiv logo" style="height:60px;" /></a></div>
          <div class="column logo-cornell"><a href="https://www.cornell.edu/">
            <picture>
              <source media="(min-width: 501px)"
                srcset="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg  400w"
                sizes="400w" />
              <source srcset="/static/browse/0.3.4/images/icons/cu/cornell_seal_simple_black.svg 2x" />
              <img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University Logo" />
            </picture>
          </a></div>
          <div class="column nav" id="toggle-container" role="menubar">
            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-white"><title>open search</title><path d="M505 442.7L405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9 0 208 0S0 93.1 0 208s93.1 208 208 208c48.3 0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9 0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7 0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7 0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
                <div class="field has-addons">
                  <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
                  <input type="hidden" name="source" value="header">
                  <input type="hidden" name="searchtype" value="all">
                  <button class="button">GO</button>
                </div>
              </form>
            </div>

            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-white" role="menu"><title>open navigation menu</title><path d="M16 132h416c8.837 0 16-7.163 16-16V76c0-8.837-7.163-16-16-16H16C7.163 60 0 67.163 0 76v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16z"/ ></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <nav class="mobile-menu" aria-labelledby="mobilemenulabel">
                <h2 id="mobilemenulabel">quick links</h2>
                <ul>
                    <li><a href="https://arxiv.org/login">Login</a></li>
                    <li><a href="https://info.arxiv.org/help">Help Pages</a></li>
                    <li><a href="https://info.arxiv.org/about">About</a></li>
                </ul>
              </nav>
            </div>
          </div>
        </div>
      </div><!-- /end mobile-header -->
    </header>

    <main>
      <div id="content">
<!--
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
    <rdf:Description
        rdf:about="/abs/2408.08274"
        dc:identifier="/abs/2408.08274"
        dc:title="BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts"
        trackback:ping="/trackback/2408.08274" />
    </rdf:RDF>
--><div id="abs-outer">

  <div class="leftcolumn">
    <div class="subheader">
      <h1>Computer Science > Machine Learning</h1>
    </div>

    <div class="header-breadcrumbs-mobile">
      <strong>arXiv:2408.08274</strong> (cs)
    </div>
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
<div id="content-inner">
  <div id="abs">
    <div class="dateline">
  [Submitted on 15 Aug 2024 (<a href="https://arxiv.org/abs/2408.08274v1">v1</a>), last revised 17 Aug 2024 (this version, v2)]</div>
    <h1 class="title mathjax"><span class="descriptor">Title:</span>BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts</h1>
    <div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Zhang,+Q">Qizhen Zhang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gritsch,+N">Nikolas Gritsch</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gnaneshwar,+D">Dwaraknath Gnaneshwar</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Guo,+S">Simon Guo</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Cairuz,+D">David Cairuz</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Venkitesh,+B">Bharat Venkitesh</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Foerster,+J">Jakob Foerster</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Blunsom,+P">Phil Blunsom</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ruder,+S">Sebastian Ruder</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ustun,+A">Ahmet Ustun</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Locatelli,+A">Acyr Locatelli</a></div>            <div id="download-button-info" hidden>View a PDF of the paper titled BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts, by Qizhen Zhang and 10 other authors</div>
    <a class="mobile-submission-download" href="/pdf/2408.08274">View PDF</a>
    <a class="mobile-submission-download" href="https://arxiv.org/html/2408.08274v2">HTML (experimental)</a>



    <blockquote class="abstract mathjax">
            <span class="descriptor">Abstract:</span>The Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance over dense models. However, training MoEs from scratch in a large-scale regime is prohibitively expensive. Existing methods mitigate this by pre-training multiple dense expert models independently and using them to initialize an MoE. This is done by using experts&#39; feed-forward network (FFN) to initialize the MoE&#39;s experts while merging other parameters. However, this method limits the reuse of dense model parameters to only the FFN layers, thereby constraining the advantages when &#34;upcycling&#34; these models into MoEs. We propose BAM (Branch-Attend-Mix), a simple yet effective method that addresses this shortcoming. BAM makes full use of specialized dense models by not only using their FFN to initialize the MoE layers but also leveraging experts&#39; attention parameters fully by initializing them into a soft-variant of Mixture of Attention (MoA) layers. We explore two methods for upcycling attention parameters: 1) initializing separate attention experts from dense models including all attention parameters for the best model performance; and 2) sharing key and value parameters across all experts to facilitate for better inference efficiency. To further improve efficiency, we adopt a parallel attention transformer architecture to MoEs, which allows the attention experts and FFN experts to be computed concurrently. Our experiments on seed models ranging from 590 million to 2 billion parameters demonstrate that BAM surpasses baselines in both perplexity and downstream task performance, within the same computational and data constraints.
    </blockquote>

    <!--CONTEXT-->
    <div class="metatable">
      <table summary="Additional metadata"><tr>
          <td class="tablecell label">Subjects:</td>
          <td class="tablecell subjects">
            <span class="primary-subject">Machine Learning (cs.LG)</span></td>
        </tr><tr>
          <td class="tablecell label">Cite as:</td>
          <td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2408.08274">arXiv:2408.08274</a> [cs.LG]</span></td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxividv">(or <span class="arxivid">
              <a href="https://arxiv.org/abs/2408.08274v2">arXiv:2408.08274v2</a> [cs.LG]</span> for this version)
          </td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxivdoi">              <a href="https://doi.org/10.48550/arXiv.2408.08274"  id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2408.08274</a><div class="button-and-tooltip">
              <button class="more-info" aria-describedby="more-info-desc-1">
                <svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
                <span class="visually-hidden">Focus to learn more</span>
              </button>
              <!-- tooltip description -->
              <div role="tooltip" id="more-info-desc-1">
                <span class="left-corner"></span>                  arXiv-issued DOI via DataCite</div>
            </div>
          </td>
        </tr></table>
    </div>
  </div>
</div>
    <div class="submission-history">
      <h2>Submission history</h2> From: Qizhen Zhang [<a href="/show-email/18dc9fbc/2408.08274">view email</a>]      <br/>            <strong><a href="/abs/2408.08274v1">[v1]</a></strong>
        Thu, 15 Aug 2024 17:19:12 UTC (1,380 KB)<br/>
    <strong>[v2]</strong>
        Sat, 17 Aug 2024 01:33:30 UTC (1,380 KB)<br/>
</div>
  </div>
  <!--end leftcolumn-->
<div class="extra-services">    <div class="full-text">
      <a name="other"></a>
      <span class="descriptor">Full-text links:</span>
      <h2>Access Paper:</h2>
      <ul>
  <div id="download-button-info" hidden>
View a PDF of the paper titled BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts, by Qizhen Zhang and 10 other authors</div><li><a href="/pdf/2408.08274" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2408.08274v2" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2408.08274" class="abs-button download-eprint">TeX Source</a></li><li><a href="/format/2408.08274" class="abs-button download-format">Other Formats</a></li></ul>
      <div class="abs-license"><a href="http://creativecommons.org/licenses/by/4.0/" title="Rights to this article" class="has_license">
          <img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-4.0.png"/>
          <span>view license</span>
        </a></div>
    </div>
    <!--end full-text-->    <div class="browse">
    Current browse context: <div class="current">cs.LG</div>

  <div class="prevnext">
<span class="arrow">
      <a class="abs-button prev-url" href="/prevnext?id=2408.08274&amp;function=prev&amp;context=cs.LG"
         accesskey="p" title="previous in cs.LG (accesskey p)">&lt;&nbsp;prev</a>
    </span>
    <span class="is-hidden-mobile">&nbsp; | &nbsp;</span>    <span class="arrow">
      <a class="abs-button next-url" href="/prevnext?id=2408.08274&amp;function=next&amp;context=cs.LG" accesskey="n"
         title="next in cs.LG (accesskey n)">next&nbsp;&gt;</a>
    </span><br/>
  </div><div class="list">
    <a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.LG/new">new</a>
    <span class="is-hidden-mobile"> | </span>
    <a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.LG/recent">recent</a>
    <span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.LG/2024-08">2024-08</a>
  </div><div class="abs-switch-cat">
    Change to browse by:
    <div class="switch context-change">
        <a href="/abs/2408.08274?context=cs">cs</a><br class="is-hidden-mobile">
    </div>
  </div>

    </div>

    <div class="extra-ref-cite">
      <h3>References &amp; Citations</h3>
      <ul>
        <li><a  class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2408.08274">NASA ADS</a></li><li><a  class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2408.08274" target="_blank" rel="noopener">Google Scholar</a></li>
        <li><a  class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2408.08274" target="_blank" rel="noopener">Semantic Scholar</a></li>
      </ul>
      <div style="clear:both;"></div>
    </div>

<div class='extra-ref-cite'>
    <a id='bib-cite-css' hidden='true' href='/static/browse/0.3.4/css/cite.css'>a</a>

    <span id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</span>
    <span id='bib-cite-loading' hidden='true'>Loading...</span>
</div>

<div id='bib-cite-modal' class='bib-modal' hidden='true'>
    <div class='bib-modal-content'>
        <div class='bib-modal-title'>
            <h2>BibTeX formatted citation</h2>
            <span class='bib-modal-close' >&times;</span>
        </div>
        <div>
            <textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
        </div>
        <div>
            <span>Data provided by: </span>
            <a id='bib-cite-source-api'></a>
        </div>
    </div>
</div><div class="bookmarks">
  <div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2408.08274&amp;description=BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts"
     title="Bookmark on BibSonomy">
    <img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
         alt="BibSonomy logo"/>
  </a>
  <a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2408.08274&amp;title=BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts"
     title="Bookmark on Reddit">
    <img src="/static/browse/0.3.4/images/icons/social/reddit.png"
         alt="Reddit logo"/>
  </a>
</div>  </div>
  <!--end extra-services-->
<!-- LABS AREA -->
<div id="labstabs">
  <div class="labstabs"><input type="radio" name="tabs" id="tabone"checked="checked">
    <label for="tabone">Bibliographic Tools</label>
    <div class="tab labs-display-bib">
      <h1>Bibliographic and Citation Tools</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input id="bibex-toggle" type="checkbox" class="lab-toggle">
              <span class="slider"></span>
              <span class="is-sr-only">Bibliographic Explorer Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
          </div>
        </div>
<div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="litmaps-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
                aria-labelledby="label-for-litmaps">
              <span class="slider"></span>
              <span class="is-sr-only">Litmaps Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
          </div>
        </div>
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="scite-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
                aria-labelledby="label-for-scite">
              <span class="slider"></span>
              <span class="is-sr-only">scite.ai Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
          </div>
        </div>
      </div>
        <div class="labs-content-placeholder labs-display" style="display: none;"></div>
        <div style="min-height: 15px" id="litmaps-open-in"></div>
        <div style="min-height: 15px" id="scite-open-in"></div>
    </div>


    <input type="radio" name="tabs" id="tabtwo">
    <label for="tabtwo">Code, Data, Media</label>
    <div class="tab">
      <h1>Code, Data and Media Associated with this Article</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input        
                id="catalyzex-toggle"
                data-script-url="/static/browse/0.3.4/js/catalyzex.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="dagshub-toggle"
                data-script-url="/static/browse/0.3.4/js/dagshub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
              <span class="slider"></span>
              <span class="is-sr-only">DagsHub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
          </div>
        </div>
  
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="gotitpub-toggle"
                data-script-url="/static/browse/0.3.4/js/gotitpub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
              <span class="slider"></span>
              <span class="is-sr-only">GotitPub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="paperwithcode-toggle"
                data-script-url="/static/browse/0.3.4/js/paperswithcode.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-pwc">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-pwc">Papers with Code</span> <em>(<a href="https://paperswithcode.com/" target="_blank">What is Papers with Code?</a>)</em>
          </div>
        </div>


        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="sciencecast-toggle"
                data-script-url="/static/browse/0.3.4/js/sciencecast.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
              <span class="slider"></span>
              <span class="is-sr-only">ScienceCast Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
          </div>
        </div>
      </div>

      <div id="catalyzex-output" style="display:none"></div>
      <div id="dagshub-output" style="display:none"></div>
      <div id="gotitpub-output" style="display:none"></div>
      <div id="pwc-output" style="display:none"></div>
      <div id="pwc-data-output" style="display:none"></div>
      <div id="sciencecast-output" style="display:none"></div>
    </div>


      <input type="radio" name="tabs" id="labstabs-demos-input">
      <label for="labstabs-demos-input" id="labstabs-demos-label">Demos</label>
      <div class="tab">
        <h1>Demos</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="replicate-toggle"
                  data-script-url="/static/browse/0.3.4/js/replicate.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
                <span class="slider"></span>
                <span class="is-sr-only">Replicate Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="spaces-toggle"
                  data-script-url="/static/browse/0.3.4/js/spaces.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="txyz-toggle"
                  data-script-url="/static/browse/0.3.4/js/txyz.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
            </div>
          </div>
        </div>
        <div id="replicate-output"></div>
        <div id="spaces-output"></div>
        <div id="txyz-output"></div>
      </div>
      <input type="radio" name="tabs" id="tabfour">
      <label for="tabfour">Related Papers</label>
      <div class="tab">
        <h1>Recommenders and Search Tools</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="influenceflower-toggle"
                data-script-url="/static/browse/0.3.4/js/influenceflower.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
                <span class="slider"></span>
                <span class="is-sr-only">Link to Influence Flower</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="connectedpapers-toggle" type="checkbox" class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
                       aria-labelledby="label-for-connected-papers" >
                <span class="slider"></span>
                <span class="is-sr-only">Connected Papers Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
                <span class="slider"></span>
                <span class="is-sr-only">Core recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
            </div>
          </div>          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="iarxiv-toggle"
                       type="checkbox"
                       class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/iarxiv.js"
                       aria-labelledby="label-for-iarxiv">
                <span class="slider"></span>
                <span class="is-sr-only">IArxiv recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-iarxiv">IArxiv Recommender</span>
              <em>(<a href="https://iarxiv.org/about">What is IArxiv?</a>)</em>
            </div>
          </div>
</div>
        <div id="influenceflower-output"></div>
        <div id="influenceflower-output-graph" style="display:none">
          <ul class="flower-tabs">
            <li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
          </ul>
          <div class="flower-tab-content">
            <div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
            <div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
            <div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
            <div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
          </div>
        </div>
        <div style="min-height: 15px" id="connectedpapers-output"></div>
        <div id="coreRecommenderOutput"></div>
        <div id="iarxivOutput"></div>
      </div>

      <input type="radio" name="tabs" id="tabfive">
      <label for="tabfive">
        About arXivLabs
      </label>
      <div class="tab">
        <div class="columns">
          <div class="column">
            <h1>arXivLabs: experimental projects with community collaborators</h1>
            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
            <p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
          </div>
          <div class="column is-narrow is-full-mobile">
            <p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
          </div>
        </div>
      </div>

    </div>
</div>
<!-- END LABS AREA -->
  <div class="endorsers">
    <a href="/auth/show-endorsers/2408.08274" class="endorser-who">Which authors of this paper are endorsers?</a> |
    <a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
    <span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
  </div>
  <script type="text/javascript" language="javascript">mathjaxToggle();</script>
</div>
      </div>
    </main>

    <footer style="clear: both;">
      <div class="columns is-desktop" role="navigation" aria-label="Secondary" style="margin: -0.75em -0.75em 0.75em -0.75em">
        <!-- Macro-Column 1 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/about">About</a></li>
                <li><a href="https://info.arxiv.org/help">Help</a></li>
              </ul>
            </div>
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>contact arXiv</title><desc>Click here to contact arXiv</desc><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>
                  <a href="https://info.arxiv.org/help/contact.html"> Contact</a>
                </li>
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>subscribe to arXiv mailings</title><desc>Click here to subscribe</desc><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
                  <a href="https://info.arxiv.org/help/subscribe"> Subscribe</a>
                </li>
              </ul>
            </div>
          </div>
        </div>
        <!-- End Macro-Column 1 -->
        <!-- Macro-Column 2 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/license/index.html">Copyright</a></li>
                <li><a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy Policy</a></li>
              </ul>
            </div>
            <div class="column sorry-app-links">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/web_accessibility.html">Web Accessibility Assistance</a></li>
                <li>
                  <p class="help">
                    <a class="a11y-main-link" href="https://status.arxiv.org" target="_blank">arXiv Operational Status <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 512" class="icon filter-dark_grey" role="presentation"><path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"/></svg></a><br>
                    Get status notifications via
                    <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/email/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>email</a>
                    or <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/slack/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-black" role="presentation"><path d="M94.12 315.1c0 25.9-21.16 47.06-47.06 47.06S0 341 0 315.1c0-25.9 21.16-47.06 47.06-47.06h47.06v47.06zm23.72 0c0-25.9 21.16-47.06 47.06-47.06s47.06 21.16 47.06 47.06v117.84c0 25.9-21.16 47.06-47.06 47.06s-47.06-21.16-47.06-47.06V315.1zm47.06-188.98c-25.9 0-47.06-21.16-47.06-47.06S139 32 164.9 32s47.06 21.16 47.06 47.06v47.06H164.9zm0 23.72c25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06H47.06C21.16 243.96 0 222.8 0 196.9s21.16-47.06 47.06-47.06H164.9zm188.98 47.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06h-47.06V196.9zm-23.72 0c0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06V79.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06V196.9zM283.1 385.88c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06v-47.06h47.06zm0-23.72c-25.9 0-47.06-21.16-47.06-47.06 0-25.9 21.16-47.06 47.06-47.06h117.84c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06H283.1z"/></svg>slack</a>
                  </p>
                </li>
              </ul>
            </div>
          </div>
        </div> <!-- end MetaColumn 2 -->
        <!-- End Macro-Column 2 -->
      </div>
    </footer>
  </div>

  <script src="/static/base/1.0.1/js/member_acknowledgement.js"></script>

</body>

</html>
2408.08274
 The Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance over dense models.  However, training MoEs from scratch in a large-scale regime is prohibitively expensive. Existing methods mitigate this by pre-training multiple dense expert models independently and using them to initialize an MoE. This is done by using experts' feed-forward network (FFN) to initialize the MoE's experts while merging other parameters. However, this method limits the reuse of dense model parameters to only the FFN layers, thereby constraining the advantages when "upcycling" these models into MoEs. We propose BAM (Branch-Attend-Mix), a simple yet effective method that addresses this shortcoming. BAM makes full use of specialized dense models by not only using their FFN to initialize the MoE layers but also leveraging experts' attention parameters fully by initializing them into a soft-variant of Mixture of Attention (MoA) layers. We explore two methods for upcycling attention parameters: 1) initializing separate attention experts from dense models including all attention parameters for the best model performance; and 2) sharing key and value parameters across all experts to facilitate for better inference efficiency. To further improve efficiency, we adopt a parallel attention transformer architecture to MoEs, which allows the attention experts and FFN experts to be computed concurrently. Our experiments on seed models ranging from 590 million to 2 billion parameters demonstrate that BAM surpasses baselines in both perplexity and downstream task performance, within the same computational and data constraints. 
abstract
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Abstract
2408.08274
 Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks [2–4]. Often, LLMs use a dense architecture, where models apply all parameters to every input during training and inference [2, 3]. Consequently, increasing model capacity results in increased computational cost. An empirical finding in the scaling laws of LLMs [5] indicates that model performance scales with model capacity. However, naively increasing model size to achieve predicted improvements is challenging due to 1) increased hardware requirements and associated costs; and 2) greater training instabilities such as gradient spikes during large-scale training runs [6]. Mixture of Experts [MoEs; 7–9] have emerged as a popular solution to these issues [10–12]. During training and inference, each input activates only a subset of the model parameters (often referred to as experts), thereby decoupling computation cost from the total parameter count, allowing the total number of  2 Figure 1: BAM operates in three phases. Different colors correspond to different expert domains,  ![1_image_0.png](1_image_0.png)  which indicates the pre-trained seed model. White indicates random parameter initialization, gradient color indicates parameter merging **1) Branching**: Begin with a pre-trained dense seed model and create N copies of it. **2) Continued Pre-training**: Continue to pre-train each copy independently on its own data mixture. This process yields specialized dense expert models. **3) Mixture Model** Training: Utilize these specialized dense expert models to initialize both the FFN and attention experts of the mixture model. The router layers are initialized randomly. All other parameters are derived by averaging the corresponding layers in each of the dense experts. Note that BAM employs a parallel attention transformer architecture that concurrently computes attention experts and FFN experts. The figure is loosely based on Figure 1 from Sukhbaatar et al. [1].  parameters to grow without additional computation. Moreover, MoEs empirically outperform dense models with equivalent computational requirements [7, 13].  However, training MoEs from scratch is expensive due larger compute requirements [12, 7] and is difficult due to training instabilities[8]. To address this, recent works have explored more efficient alternatives by initializing MoEs using pre-trained dense models [14, 1], followed by a small number of MoE training steps. This approach offers several advantages: 1) By utilizing a pre-trained model, we can leverage existing knowledge rather than starting from a random initialization; 2) the dense model used for initialization is smaller in total parameter count, making it faster and easier to train; 3) various pre-trained dense models are readily available in the open source community, which can be improved further without incurring additional pre-training cost [15, 16, 3]. In particular, BranchTrain-MiX [BTX; 1] initializes an MoE with N FFN experts through a three-step upcycling process.  Initially, N copies of a pre-trained seed model are created. Each copy is then further pre-trained on domain-specific data for specialization. The MoE's FFN expert parameters are subsequently initialized from these specialized dense models by directly copying their FFN parameters. It is important to note that the MoE's non-FFN parameters, such as attention layers, are not converted to MoE layers. Since there are N dense models, each with its own set of attention parameters, BTX uses the average of the N attention parameters to initialize the MoE's attention module while the router parameters are initialized randomly. However, this approach under-utilizes the potential of the specialized dense models for two key reasons. 1) The expert knowledge embedded in the attention modules of each of these specialized models is not leveraged, despite their potential contribution to performance enhancement; 2) Uniform averaging of parameters from distinct models, even though they originate from the same seed, can compromise performance due to averaging divergent specializations [17].  Motivated by these limitations, we propose Branch-Attend-Mix (BAM), a simple yet effective improvement to BTX that upcycles both FFN and attention parameters into expert layers. See Figure 1 for an illustration of BAM. Our contributions are as follows: - To fully leverage the benefits of specialized attention experts and improve training stability, BAM uses a variant of Mixture of Attention [MoA; 18] with soft routing where each token is assigned to every attention expert. We validate through ablation studies that this is critical to surpass baseline performance.  - We propose two variants of BAM: 1) For best model performance, BAM with KV Experts includes all attention parameters in the attention experts; 2) For enhanced inference efficiency, BAM with KV sharing shares the key and value parameters across all attention experts, similar to the method done in Zhang et al..  - To alleviate the increased computations from soft-routing attention experts, BAM employs a parallel attention transformer architecture [19, 20] that allows the attention experts and FFN experts to be computed in parallel, thereby improving throughput.  - We validate our approach through experiments on seed models ranging from 590 million to 2 billion parameters. Our results demonstrate that BAM outperforms the BTX model in terms of both perplexity and downstream task performance across various domains under equivalent data and computational constraints, confirming the effectiveness of our proposed enhancements. 
introduction
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Introduction
2408.08274
 Efficient initialization for MoEs Two recent works have studied how to use pre-trained dense models to initialize MoEs [14, 1]. Sparse Upcycling [14] upcycles a single dense model by simply copying over dense model parameters as MoE parameters. It replicates FFN parameters N times into each FFN expert in sparse MoE layers. BTX [1] improves on this approach by upcycling not just a single dense seed model but also multiple specialized dense models branched from the seed model.  This is advantageous because not only can the dense models be trained in parallel, but also they are specialized in particular domains. However, unlike our method, both approaches only upcycle the FFN part of the dense (seed or specialized) models.  Alternative architecture for MoEs Alternative to the standard MoE architectures where only feed-forward blocks (FFNs) are used as MoE layers, recent work [18, 21–23] investigates the mixtureof-attention framework, which uses attention experts in addition to FFN experts. This approach has not gained widespread popularity because it only achieves modest performance improvements while requiring additional engineering tricks for optimization [22]. However, in the setting of upcycling specialized dense experts into an MoE, the usage of attention experts is much more appropriate, as the attention parameters contain specialized domain knowledge that would otherwise not be accessible in the final MoE.  Alternatives Methods for Model Merging Recent work has tried to combine many openly available, dense models to create an improved model, via layerwise merging or stitching of pre-trained model weights [24]. However, this approach creates a new dense model, not a sparse one. Other work has looked into dynamically adding new blocks to MoA whenever the model is trained on a new domain, but uses random initialization for the new experts instead of utilizing existing models [25]. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Related Work
2408.08274
 The typical architecture for LLMs is based on stacking multiple blocks of the transformers [26]. The conventional transformer block comprises a Multi-Headed Attention (MHA) module, commonly referred to as the attention layer, followed by a residual connection and a feed-forward neural network (FFN). Recent works [19] introduced the parallel attention transformer, also known as parallel layers [20]. Parallel attention transformers is a variant of the original architecture which computes the outputs of the attention and the FFN layers in parallel using the same input. The final transformer output is the projected concatenation of the attention output and the FFN output with its residual connection (See Figure 1). Processing the two layers in parallel increases the computational throughput without degrading the performance [20]. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Background 3.1 Parallel Attention Transformers
2408.08274
 In an attention layer with h heads, the input is linearly projected to form the query, key, and value vectors through learned parameters. Each head applies the attention mechanism to its vectors independently to compute its attention outputs. The individual outputs from all h heads are concatenated and linearly projected back to the dimensionality of the original model, ensuring consistent input and output dimensions throughout the network layers.  $$\begin{split}\text{Head}_{i}&=\text{softmax}\left(\frac{QW_{i}^{q}(KW_{i}^{k})^{T}}{\sqrt{d_{k}}}\right)VW_{i}^{v}\\ \text{MHA}(Q,K,V)&=\text{concatenate}\left(\text{head}_{1},\ldots,\text{head}_{h}\right)W^{o},\end{split}\tag{1}$$  where $W_{i}^{q},W_{i}^{k}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}$, $W_{i}^{v}\in\mathbb{R}^{d_{\text{model}}\times d_{v}}$ and $W^{o}\in\mathbb{R}^{h_{d_{k}}\times d_{\text{model}}}$, $d_{k}$ is the key dimension.   4 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Multi-Headed Attention
2408.08274
 The Mixture of Experts [MoE; 9] model replaces the FFN layer by an MoE layer in the conventional dense transformer. An MoE layer consists of a linear router, parameterized by WFFN router ∈ R dmodel×N , and a set of N FFN experts, denoted as {FFNi(x)} N i=1. The router outputs the normalized router logits p(x) = softmax (WFFN router · x), where pi(x) corresponds to the gate value for the i-th expert, FFNi. The router assigns the input token representation x to the k experts with the highest gate values. Denoting the set of selected top-k indices as T , the MoE layer's final output is the weighted sum of the selected experts' outputs, weighted by their respective gating values:  $$\mathrm{FFN}_{\mathrm{MoE}}=\sum_{i\in{\mathcal{T}}}p_{i}(x)\mathrm{FFN}_{i}(x).$$ 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Mixture Of Experts
2408.08274
 Mixture of Attention [MoA; 18, 23] extends MoEs by also replacing the conventional attention layer with attention experts. Each MoA layer comprises of a linear router parameterized by Wattn router ∈ R  dmodel×N and a set of N attention experts, {MHAj (x)} N j=1. The router outputs the normalized router logits g(x) = softmax (Wattn router · x), with each gj (x) represents the gate value for the j-th expert.  Each attention expert is composed of query-projection experts W q j N j=1 and the attention output projection experts Wo j N j=1. The key projection Wkand value projection Wvare shared among all experts for computation efficiency. The final output of the MoA layer is a gate-value weighted sum of the computations from the top-k selected experts M:  $$\mathrm{MHA}_{\mathrm{MoA}}=\sum_{i\in\mathcal{M}}g_{i}(x)\mathrm{MHA}_{i}(x).$$ 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Mixture Of Attention
2408.08274
 1) Branching: We create N copies of the model from a pre-trained dense model. We will refer to this dense model as the seed model from here on. We use a pre-trained seed model with a parallel attention transformer architecture. This is motivated by two reasons: 1) Parallel attention transformers offer better throughput compared to traditional transformers, with only minor performance degradation at smaller scales but no degradation at larger scales [20]. 2) Employing a dense seed model with a parallel attention transformer architecture allows the upcycled mixture model to inherit this parallel attention setup. This architecture enables the attention experts and FFN experts to be computed concurrently, thereby enhancing both training and inference efficiency.  2) Continued Pre-training: Each copy of the seed model undergoes independent pre-training on a specialized data mixture, each tailored to different domains such as law, mathematics, and coding.  This phase results in specialized dense expert models that have enhanced performance within their respective domains compared to the seed model. However, these models might perform wose in domains outside their specialization.  3) Mixture Model Training: We initialize the mixture model using the specialized dense models from step 2). We also incorporate a copy of the original pre-trained dense seed model (similar to the BTX approach), ensuring the seed model's general knowledge acquired in the pre-training stage is transferred to the mixture model. A key differentiator for BAM is it takes full advantage of the dense specialized models by leveraging both the FFN and attention layers. The parameters for each attention expert and FFN expert in the mixture model are directly inherited from their corresponding expert or seed dense model. Non-expert parameters in the mixture model, such as layer normalization, input and output embeddings, are initialized by uniformly averaging the corresponding parameters from the dense models. The router layers are randomly initialized. The upcycling of both FFN experts and attention experts is beneficial for two reasons: 1) It maximizes the utilization of specialized capabilities in the dense models; and 2) It avoids averaging the attention parameters, which has shown to degrade model performance [17]. In the subsequent section, we will discuss in more detail the training methodologies of the mixture model. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Bam: Branch-Attend-Mix There Are Three Phases To Bam:
2408.08274
 The MoA architecture proposed by Zhang et al. uses only the query projection (Wq) and the attention output projection (Wo) as attention experts, while sharing the key-value projections among all experts to enhance efficiency. Like conventional MoEs, Zhang et al. also employs top-k routing, which assigns input tokens to only k experts with the highest attention gate values.  To adapt MoA to our BAM framework, we propose two modifications to the original architecture.  First, instead of the traditional top-k routing, BAM employs soft routing for the attention experts, where the attention router assigns each input token to all attention experts. We show in Section 7 that using soft-routing is crucial for achieving performance superior to the baseline. Second, we propose a variant of MoA where that also incorporates the key-value projections (Wkand Wv) as part of the attention experts (i.e. we have one copy of Wk i , Wv i for each attention expert i). This is as oppose to than sharing them across all experts. These two modifications are motivated by three reasons:  1. Employing soft-routing and including KV experts allow us to fully exploit the capabilities of the specialized attention experts.  2. In transformer architectures, FFNs computations typically account for a larger portion of computational load than attention computations [27, 28]. By adopting the parallel attention transformer architecture, BAM enables the concurrent computation of both the attention experts module and the FFN experts module. In addition, we implement expert parallelism [7] where the individual FFNs and attention are computed in parallel across experts. Thus, we are partially "masking" the extra computations introduced by soft-routing and KV experts under the more expensive FFN experts computations.  3. Using soft-routing allows for more stable training dynamics since the router is no longer a discrete optimization problem. This alleviates the attention experts training from common MoE problems such as imbalanced router load[7]. We discuss in Section 5.3 on how to evaluate BAM fairly comparing to baselines. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Mixture Model Training: Attention Experts
2408.08274
 We use top-1 routing of FFN experts for all experiments unless otherwise stated. For training stability, we use the auxiliary load balancing loss LLB [7] to encourage uniform load across all experts and the auxilliary router z-loss [8] to penalize large logits going into the router. Let B be the number of tokens in the batch, fi be the fraction of tokens assigned to FFN expert i, and Pi be the fraction of the expert i's gate value, then the two auxiliary losses are as follows:  $$\mathcal{L}_{\mathrm{LB}}=N\sum_{i=1}^{N}f_{i}P_{i}$$ $$\mathcal{L}_{z}=\frac{1}{B}\sum_{i=1}^{B}\left(\mathrm{LogSumExp}_{j=1:N}\left(x_{j}^{i}\right)\right)^{2}\tag{1}$$ $${\mathrm{(2)}}$$  The model's final loss function L = LNLL +P∀ MoE Layers (αLLB + βLz) is a weighted sum, where LNLL is the negative log-likelihood loss, and α and β are hyperparameters.  5 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Mixture Model Training Phase: Ffn Experts
2408.08274
 We validate BAM against the baseline model, BTX, through two sets of experiments that vary in scale. Detailed architecture specifications are shown in Table 8 in the Appendix. For both sets of experiments, we train a mixture model upcycled from four dense experts: a general pre-training seed model, along with specialized experts in code, mathematics, and law. Each expert undergoes a continued pre-training phase using 100 billion tokens. We use the AdamW optimizer [29] with a weight decay of 0.1. To ensure training stability, the peak learning rate for the continued pre-training phase is set to 50% that of the pre-training phase.  Small-Scale Experiments Our first set of experiments uses a dense seed model with 590 million parameters, pre-trained on 400 billion token with a batch size of 1 million tokens. During the pre-training and mixture model training phases, we employ a learning rate schedule that warms up from 0 to 1.2e − 4 over 1500 steps, then undergoes a cosine decay to 10% of the peak learning rate.  Large-Scale Experiments The second set of experiments utilizes a dense seed model with 2 billion parameters, pre-trained on 750 billion tokens with a batch size of 4 million tokens. During the pre-training phase, the learning rate schedule warms up from 0 to 1e − 2 over 2000 steps, then cosine-decays to 3% of the peak learning rate. In the mixture model training phase, we observes gradient spikes when using the same learning rate schedule as in the pre-training phase. To mitigate this, we lower the peak learning rate to 1e − 4, warm-up steps to 1000. And as before, cosine-decay to 10% of the peak learning rate. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Experiment Details 5.1 Experimental Setup
2408.08274
 Below we describe the data details for both BAM and BTX experiments. In the **continued pretraining phase**, each dense expert is trained on a specialized data domain for 100 billion tokens. The data domains contain the following data:  - **Mathematics**: Utilizes the MathGLM [30], GSM8K [31], proof-pile-2 [15], and MathPile [32] datasets. Notably, we ensure that there is no overlap between the GSM8K train and evaluation splits.  - **Code**: Comprises the Starcoder dataset [33]. - Law: Includes the pile-of-law [34] and HUPD [35] datasets. Additionally, each data domain is augmented with 10% data sourced from Common Crawl. In our ablation experiments, we observed that incorporating a portion of general-text data into the expert training mix enhances model performance performance as it allows the expert data distribution to more matching with the data distribution of the seed model.  In the **mixture model training phase**, the training data comprises an equal distribution of data from each domain (including the pre-training domain), with each domain contributing 25% to the overall mix. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Data Details
2408.08274
 We compare BAM to BTX, as well as against the original dense seed model and the dense expert models. Similar to the evaluations in Sukhbaatar et al. [1], BAM is evaluated in two settings:  - **Data-Matching (DM)**: We use exactly the same training data in terms of both content and quantity for BAM as used for the BTX baseline.  - **Compute-Matching (CM)**: We train BAM with the same amount of computation resources as BTX is trained on, measured in TPU-core-days. In the small-scale experiments, all MoE training phases are allocated 25 TPU-core-days, while in the large-scale experiments, the allocation increases to 305 TPU-core-days.  Base Model **23.98** 8.62 8.92 14.12 13.91  Specialized Models  Code Dense Expert 39.50 3.71 10.83 8.77 15.70 Law Dense Expert 87.82 23.45 7.53 30.79 37.40 Math Dense Expert 57.61 6.78 10.43 5.67 20.12  Generalist Models  BTX (Baseline) 26.72 3.78 6.63 5.77 10.72  BAM DM (Expert KV), ours 24.02 3.45 6.04 5.31 **9.70**  BAM CM (Expert KV), our 24.84 3.53 6.21 5.43 10.00 BAM DM (Shared KV), ours 25.67 3.64 6.39 5.57 10.32 BAM CM (Shared KV), ours 26.00 3.72 6.50 5.63 10.46  Table 1: Perplexity evaluation (↓) of BAM versus BTX for small-scale experiments (using a seed  model with **590M parameters**). Highlighted entries denote models that outperform the BTX  baseline. BAM consistently outperforms the baseline under both compute matching and token  matching regimes.  Pretrain Code Law Math Average  Base Model **9.83** 2.41 3.86 3.71 3.33  Specialized Models  Code Dense Expert 15.39 **2.18** 5.22 4.34 3.91  Law Dense Expert 32.69 6.84 **3.09** 8.61 6.18  Math Dense Expert 20.32 3.20 5.11 **3.20** 3.84  Generalist Models  BTX (Baseline) 10.35 2.40 3.76 3.64 3.27 BAM DM (Expert KV), ours 10.11 2.36 3.66 3.55 **3.19**  BAM CM (Expert KV), our 10.19 2.37 3.69 3.57 3.21  BAM DM (Shared KV), ours 10.20 2.37 3.69 3.59 3.22  BAM CM (Shared KV), ours 10.28 2.38 3.72 3.61 3.24  Table 2: Perplexity evaluation (↓) of BAM versus BTX for large-scale experiments (using a seed  model of **2B parameters**). Highlighted entries indicate models outperforming the BTX baseline.  BAM consistently surpasses BTX in all domains under both compute matching and token matching  regimes.  Pretrain Code Law Math Average  In addition to evaluating perplexity, we also asses large-scale models' performances with zero-shot and few-shot downstream tasks relevant to the expert domains1.  For each domain, we report the average of task scores within that domain.  - **Math**: For mathematical reasoning, we report the average performance on GSM8K (8-shot) [31] and MATH (4-shot) [36].  - **Code**: For code generation, we evaluate on HumanEval (0-shot) [37] and MBPP (3-shot) [38].  - Law: We use the International Citizenship Questions sub-task in LegalBench [39].  - **World Knowledge**: For general knowledge on facts, we report the performance on Natural Questions (0-shot) [40] and TriviaQA (0-shot) [41].  - **Reasoning**: For reasoning abilities, we use ARC-Easy, ARC-Challenge [42], SIQA [43], PIQA [44], WinoGrande [45], and QUAC [46] (all 0-shot).  - **General Knowledge**: We use MMLU (5-shot) [47] to test general language understanding.  1For LegalBench, we specifically selected the International Citizenship QA sub-task, as it was the only sub-task among those implemented in our code-base that scored above random.  Math Code Law Know. Reason. MMLU Average  Seed Model 3.68% 9.41% 73.34% 21.33% **47.73%** 34.13% 31.60%  Specialized Models  Math Dense Expert **4.92%** 12.39% 68.21% 13.32% 46.11% 34.29% 29.87% Code Dense Expert 3.19% **18.80%** 21.49% 12.18% 44.29% 31.50% 21.91% Law Dense Expert 3.05% 0.20% **88.80%** 10.41% 44.08% 32.18% 29.79% Generalist Models  BTX (Baseline) 3.86% 10.05% 81.85% 19.07% 47.36% 34.07% 32.71%  BAM DM  (Expert KV), ours 4.44% 12.83% 85.47% 19.89% 47.11% 34.42% **34.02%**  BAM CM  (Expert KV), ours 4.34% 12.48% 82.79% 19.51% 47.43% 34.43% 33.50%  BAM DM  (Shared KV), ours 4.10% 11.76% 86.73% 19.48% 47.27% **34.55%** 33.98%  BAM DM  (Shared KV), ours 3.65% 11.77% 80.98% 19.22% 47.56% 34.16% 32.89%  Table 3: Benchmark evaluations (↑) for BAM versus BTX. Table shows large-scale experiments (using a seed model of **2B parameters**). Highlighted entries indicate models outperform the BTX baseline. All BAM variants outperform BTX on average. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Evaluation
2408.08274
 We evaluate two variants of BAM: *BAM (Expert KV)*, where all attention parameters are utilized in the attention experts, and *BAM (Shared KV)*, where key-value (KV) projections are shared among all attention experts. We report evaluations on both variants because each presents its own advantages.  BAM (Expert KV) yields better model performance because it leverages more specialization from expert dense models. Conversely, *BAM (Shared KV)* offers greater inference efficiency because of reduced memory requirements and the shared KV cache. A more detailed analysis on inference efficiency is provided in Section 7. Perplexity Evaluations We present perplexity results for both dense models (seed and specialized dense models) and generalist models (BAM and BTX). Results from small-scale experiments are shown in Table 1, and large-scale experiments in Table 2. In all experiments, BAM consistent outperforms BTX under both the compute and token-matching regimes. While dense expert models sometimes excel in their respective specialized domain, they significantly under perform compared to generalist MoE models in all other domains.  Benchmark Evaluations See Table 3 for downstream tasks evaluations on large scale experiments where we show that all BAM variants outperform BTX on average. We omit the small-scale downstream results as most of the performance was indistinguishable from random guessing, likely due to the limited scale of the models. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Results
2408.08274
 We ablate the importance of upcycling attention experts in BAM by comparing it to BTX, ensuring both models have matching total parameters (equivalent computations) and/or active parameters (equivalent model capacity). All ablation experiment use the same number of 100B tokens in the mixture model training phase. See Table 9 in the Appendix for a detailed walk-through on calculating the number of total and active parameters of each model.  Matching the Number of Total Parameters with BTX To match the number of total parameters in BTX with BAM, we simply increase the number of FFN experts while still using top-1 routing.  All newly added FFN experts are initialized from the seed model's FFN. As shown in Table 4, BTX matches BAM's total parameter count by employing 6 FFN experts. Even when BTX's total number of FFN experts is increased to 8, thereby exceeding the total parameter count of BAM, BAM continues to outperform BTX.  Total Params Active Params Code Law Math Pre-train  BAM, *ours* 776M 663M **3.44 6.04 5.31 24.06**  BTX 4 Experts 700M 587M 3.78 6.63 5.77 26.72 BTX 5 Experts 738M 587M 4.11 7.19 6.25 29.08  BTX 6 Experts 776M 587M 3.99 7.03 6.10 28.50 BTX 7 Experts 813M 587M 3.90 6.84 6.00 27.36 BTX 8 Experts 851M 587M 3.94 6.92 6.05 27.79  Table 4: Perplexity (↓) ablation studies for small-scale experiments assess how BAM's total parameter  count compares with BTX. To match BTX's total parameters with those of BAM, we incrementally  increase the number of FFN experts in BTX from 4 up to a maximum of 8. The additional experts are  upcycled using the FFN parameters from the same dense seed model.  Matching the Number of Total & Active Parameters with BTX To match the number of active parameters with BAM, we simply increase the number of top-k from the usual top-1 to top-3 in every MoE layer of BTX. We see that BTX matches total parameters and active parameters with BAM when using top-3 routing with a total of 6 experts in Table 5. 3 of the 6 experts are upcycled from the 3 expert dense models while the other 3 are upcycled from the same copy of the dense seed model.  Despite matching parameters, BTX does not outperform BAM.  Total Params Active Params Code Law Math Pre-train BAM, *ours* 776M 662M **3.44 6.04 5.31 24.06** BTX top-3 w/ 6 Experts 776M 662M 3.55 6.23 5.48 24.13 Table 5: Perplexity ablation studies on small-scale experiments compare the performance of BAM to BTX under conditions where both models have equivalent numbers of active parameters and total parameters. In addition to utilizing six FFN experts, we have adjusted BTX's routing mechanism from top-1 to top-3 routing. This change means that instead of activating just one FFN expert per token representation, three are now activated, effectively increasing the number of active parameters per input. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Ablations 7.1 On The Importance Of Upcycling Attention Experts
2408.08274
 We compare soft routing with the most commonly used sparse routing approaches (i.e. top-1 and top-2 routing) in BAM's attention experts layers. See Table 6 for ablation results. All experiments are compute matched at the 590M scale. Our results indicate that BAM with top-1 and top-2 attention experts routing does not show improvement over the baseline BTX on most domains. Conversely, BAM with soft-routing consistently outperforms BTX across all domains. This demonstrates that using soft routing in MoA layers is crucial and supports our decision to implement this approach.  Pretrain Code Law Math Average  BTX 26.72 3.78 6.63 5.77 10.72 BAM soft-routing MoA 26.00 3.72 6.51 5.64 **10.47**  BAM top-2 routing MoA 26.68 3.78 6.69 5.75 10.72  BAM top-1 routing MoA 26.89 3.83 6.69 5.82 10.81  Table 6: Perplexity ablation (↓) of BAM compute matched with different attention experts routing  methods. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
On The Importance Of Using Soft-Routing For Attention Experts
2408.08274
 We analyze the FLOPs count per token [5, 48] for BTX versus BAM during inference. In Table 7, we provide the estimates for non-embedding parameter per Transformer layer, excluding non-linearities, biases, and layer normalization as they are negligible. Our results are given for the small-scale model dimensions stated in Table 9 in the Appendix. For the detailed FLOPs calculation steps, refer to Table 10 in the Appendix. Compared to the standard BTX, BAM consumes more FLOPs due to the soft-routing attention experts. However, the increase in FLOPs is partially mitigated by our implementation of a parallel attention transformer architecture and expert parallelism. In addition, when we compare FLOPs between BAM and the parameter-matching variant of BTX, BAM contains approximately the same number of FLOPS while still performing better model-quality wise (see Table 5).  For empirical inference latency, we measure the time taken to generate 16 tokens from a prompt of length 256, averaged over 10 runs on TPUs. We observe that BAM is slightly slower (6.17s) than standard BTX (4.81s). The increased latency in BAM is likely attributable to increased memory pressure from the individual KV caches maintained by each attention expert, as attention mechanisms are typically memory-bound [49]. Motivated by this, the KV-sharing version of BAM reduces the inference latency from 6.17s to 5.96s, as all attention experts share a single KV cache. In future work, can further optimize inference time to be closer to the standard BTX baseline (representative of standard MoEs) as our hardware and framework code is optimized for training rather than inference latency.  | Config                                                                                          | Params   | FLOPs   |       |        |            |            |            | |-------------------------------------------------------------------------------------------------|----------|---------|-------|--------|------------|------------|------------| | Method                                                                                          | nexperts | ntopk   | Total | Active | Attention  | FFN        | Total      | | BAM                                                                                             | 4        | 1       | 776M  | 663M   | 35,651,584 | 12,589,056 | 48,257,024 | | BTX                                                                                             | 4        | 1       | 700M  | 587M   | 8,912,896  | 12,589,056 | 21,510,144 | | BTX                                                                                             | 6        | 3       | 776M  | 663M   | 8,912,896  | 37,767,168 | 46,692,352 | | Table 7: Estimates for FLOPs per token during inference. Row 1 shows BAM with KV experts, row 2 |          |         |       |        |            |            |            | 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Analysis On Inference Efficiency
2408.08274
 In this work, we introduce BAM (Branch-Attend-Mix), a simple yet effective improvement for upcycling dense models into MoE models. In addition to upcycling just the FFN parameters, we also upcycle the dense model's attention parameters into attention experts. To maximize the benefits of specialized dense models and enhance training stability, BAM uses a variant of Mixture of Attention with soft routing whereby each token is assigned to every attention expert. Our ablation studies confirm that this approach is crucial for surpassing baseline performance levels. We propose two variants of BAM to address different needs: BAM with KV Experts, which includes all attention parameters in the attention experts for optimal model performance, and BAM with KV Sharing, which shares the key and value parameters across the attention experts to boost inference efficiency.  To alleviate the increased computations from soft-routing attention experts, BAM employs a parallel attention transformer architecture. This setup enables concurrent computation of the attention expert sand FFN experts. Finally, we validate our approach through experiments on seed models ranging from 590 million to 2 billion parameters. Our results consistently show that BAM outperforms the BTX model in both perplexity and downstream task performance across various domains. These outcomes validate the effectiveness of the enhancements proposed in BAM. Future work could focus on optimizing the training data mixture for BAM's three phases and improving the training framework to accelerate both training and inference processes. 
conclusion
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Conclusion & Future Work
2408.08274
 The authors would like to thank Siddhartha Rao Kamalakara for his helpful pointers on debugging the inference code. The authors thank Felipe Cruz for his tremendous help in training and evaluating the seed model used in the large scale experiments. The authors also thank Amr Kayid, Cécile Robert-Michon, Joon Kim, Milad Alizadehand, Louise Rust, and Autumn Moulder for their amazing support on GPU and TPU infrastructure. The authors are grateful to John Lin, Viraat Aryabumi, and Roman Castagné for providing valuable suggestions and insights on data and evaluation. QZ also thanks Joanna Yoo, Kuba Perlin, and Sid for their help and discussions on a previous related project on MoEs sometime ago. QZ thanks Chris Lu for his discussions and feedback on the project. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
Acknowledgements
2408.08274
 [1] Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. Branch-train-mix: Mixing expert llms into a mixture-of-experts llm. *arXiv preprint arXiv:2403.07816*, 2024.  [2] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023.  [3] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. *arXiv preprint arXiv:2307.09288*, 2023.  [4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.  [5] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. *arXiv preprint arXiv:2001.08361*, 2020.  [6] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. *arXiv preprint arXiv:2205.01068*, 2022.  [7] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. *Journal of Machine Learning Research*, 23 (120):1–39, 2022.  [8] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. *arXiv* preprint arXiv:2202.08906, 2022.  [9] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. *arXiv preprint arXiv:1701.06538*, 2017.  [10] DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.  [11] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. *arXiv preprint arXiv:2401.04088*, 2024.  [12] Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts. *Proceedings of Machine Learning and Systems*, 5, 2023.  [13] Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygó´zd´z, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. *arXiv preprint arXiv:2402.07871*, 2024.  [14] Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. *arXiv preprint arXiv:2212.05055*, 2022.  [15] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. *arXiv preprint arXiv:2310.10631*, 2023.  [16] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. *arXiv preprint arXiv:2308.12950*, 2023.  [17] Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. *arXiv preprint arXiv:2208.03306*, 2022.  [18] Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, and Zhang Xiong. Mixture of attention heads: Selecting attention heads per token. *arXiv preprint arXiv:2210.05144*, 2022. [19] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021.  [20] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. *Journal of Machine Learning Research*, 24(240): 1–113, 2023.  [21] Hao Peng, Roy Schwartz, Dianqi Li, and Noah A Smith. A mixture of h − 1 heads is better than h heads. *arXiv preprint arXiv:2005.06537*, 2020.  [22] Róbert Csordás, Piotr Pi˛ekos, and Kazuki Irie. Switchhead: Accelerating transformers with mixture-of-experts attention. *arXiv preprint arXiv:2312.07987*, 2023.  [23] Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching llama2 performance with 0.1 m dollars. *arXiv preprint arXiv:2404.07413*, 2024. [24] Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes, 2024.  [25] Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan.  Moduleformer: Modularity emerges from mixture-of-experts, 2023.  [26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.  [27] Longteng Zhang, Xiang Liu, Zeyu Li, Xinglin Pan, Peijie Dong, Ruibo Fan, Rui Guo, Xin Wang, Qiong Luo, Shaohuai Shi, et al. Dissecting the runtime performance of the training, fine-tuning, and inference of large language models. *arXiv preprint arXiv:2311.03687*, 2023.  [28] Andrej Karpathy. I fixed the transformer diagram, 2023. URL https://x.com/karpathy/ status/1658161721251602432. [29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.  [30] Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Guo, Jinfeng Bai, and Jie Tang. Gpt can solve mathematical problems without a calculator. *arXiv preprint* arXiv:2309.03241, 2023.  [31] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. *arXiv preprint arXiv:2110.14168*, 2021.  [32] Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part i–mathpile: A billiontoken-scale pretraining corpus for math. *arXiv preprint arXiv:2312.17120*, 2023. [33] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! *arXiv preprint arXiv:2305.06161*, 2023.  [34] Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. Pile of law: Learning responsible data filtering from the law and a 256gb opensource legal dataset. *Advances in Neural Information Processing Systems*, 35:29217–29234, 2022.  [35] Mirac Suzgun, Luke Melas-Kyriazi, Suproteem Sarkar, Scott D Kominers, and Stuart Shieber.  The harvard uspto patent dataset: A large-scale, well-structured, and multi-purpose corpus of patent applications. *Advances in Neural Information Processing Systems*, 36, 2024.  [36] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.  NeurIPS, 2021.  [38] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.  [39] Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.  Advances in Neural Information Processing Systems, 36, 2024.  [40] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.  [41] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017.  [42] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. *ArXiv*, abs/1803.05457, 2018. URL https://api.semanticscholar.org/ CorpusID:3922816.  [43] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions, 2019.  [45] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.  [46] Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac : Question answering in context, 2018.  [47] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. [37] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.  [44] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.  [48] Alexandre Muzio, Alex Sun, and Churan He. Seer-moe: Sparse expert efficiency through regularization for mixture-of-experts. *arXiv preprint arXiv:2404.05089*, 2024.  [49] Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W Mahoney, et al. Full stack optimization of transformer inference: a survey. *arXiv preprint arXiv:2302.14017*, 2023. 
references
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
References
2408.08274
 | Large Scale                                                               | Small Scale   |             | |---------------------------------------------------------------------------|---------------|-------------| | Embedding dimension                                                       | 2304          | 1024        | | FFN dimension                                                             | 18432         | 4096        | | # Heads                                                                   | 18            | 8           | | # KV heads                                                                | 6             | 8           | | Vocabulary Size                                                           | 256000        | 256000      | | Activation Function                                                       | swiglu        | swiglu      | | # layers                                                                  | 18            | 6           | | Positional Embedding                                                      | rope          | rope        | | Share Input & Output Embedding                                            | Yes           | No          | | Seed Model Parameters                                                     | 2 Billion     | 590 Million | | Table 8: Model architecture details for large and small scale experiments |               |             |  | layernorm / Block   | 1024        | 1024        | 1024        | 1024        | 1024        | |---------------------|-------------|-------------|-------------|-------------|-------------| | attn_out / Block    | 1,048,576   | 4,194,304   | 4,194,304   | 1,048,576   | 1,048,576   | | qkv_proj / Block    | 3,145,728   | 1,258,2912  | 4,194,304   | 3,145,728   | 3,145,728   | | ffn_exp / Block     | 4,194,304   | 4,194,304   | 4,194,304   | 4,194,304   | 12,582,912  | | ffn_red / Block     | 2,097,152   | 2,097,152   | 2,097,152   | 2,097,152   | 6,291,456   | | router / Block      | 0           | 4096        | 4096        | 4096        | 4096        | | input embedding     | 262,144,000 | 262,144,000 | 262,144,000 | 262,144,000 | 262,144,000 | | output embedding    | 262,144,000 | 262,144,000 | 262,144,000 | 262,144,000 | 262,144,000 | | layernorm           | 1024        | 1024        | 1024        | 1024        | 1024        | | Active Params       | 587,209,728 | 662,731,776 | 612,400,128 | 587,234,304 | 662731776   | | Total Params        | 587,209,728 | 776,002,560 | 738,253,824 | 700,480,512 | 700,480,512 |  Table 9: Active and total parameter counts for small scale ablation experiments. All transformer parameters are reported as per transformer block. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
A Model Architecture Details
2408.08274
 | Ops                                                                                                 | BTX              | BAM                   |                       |                 | |-----------------------------------------------------------------------------------------------------|------------------|-----------------------|-----------------------|-----------------| | Attention Router                                                                                    | Params           | -                     | nexpertsdmodel        |                 | | FLOPs                                                                                               | -                | 2nexpertsdmodel       |                       |                 | | Attention: QKV                                                                                      | Params           | dmodel3dattn          | nexpertsdmodel3dattn  |                 | | 2                                                                                                   | 2                |                       |                       |                 | | FLOPs                                                                                               | 6d model         | 6nexpertsd model      |                       |                 | | Attention: Mask                                                                                     | Params           | -                     | -                     |                 | | FLOPs                                                                                               | 2nctxdmodel      | 2nexpertsnctxdmodel   |                       |                 | | Attention: Projection                                                                               | Params           | dattndmodel           | nexpertsdattndmodel   |                 | | FLOPs                                                                                               | 2d 2 model       | 2nexpertsd 2 model    |                       |                 | | Attention                                                                                           | FFN Router       | Params                | nexpertsdmodel        | nexpertsdmodel  | | FFN                                                                                                 |                  | FLOPs                 | 2nexpertsdmodel       | 2nexpertsdmodel | | FFN                                                                                                 | Params           | 1.5nexpertsdmodeldf f | 1.5nexpertsdmodeldf f |                 | | FLOPs                                                                                               | 3ntopkdmodeldf f | 3ntopkdmodeldf f      |                       |                 | | Table 10: Detailed breakdown of parameters and FLOPs per token in forward pass (inference)  for each |                  |                       |                       |                 |  FFN  FFN Router Params nexpertsdmodel nexpertsd*model*  FLOPs 2nexpertsdmodel 2nexpertsd*model*  FFN Params 1.5nexpertsdmodeldf f 1.5nexpertsdmodeldf f  FLOPs 3ntopkdmodeldf f 3ntopkdmodeldf f  Table 10: Detailed breakdown of parameters and FLOPs per token in forward pass (inference)  for each  MoE layer, comparing the standard BTX and BAM with KV experts. Note that BAM uses soft-gating MoA which utilizes all the attention experts, increasing total attention FLOPs and introduces a second router / gating network for attention experts. 
body
Bam! Just Like That: Simple And Efficient Parameter Upcycling For Mixture Of Experts
B Inference Efficiency Analysis