<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

<head>  <title>[2408.02442] Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
  <link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
  <link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20240822" />
  <link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
  <script language="javascript" src="/static/browse/0.3.4/js/accordion.js" /></script>
  
    <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/slider.css?v=1.14" />
    <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="/static/browse/0.3.4/js/donate.js?v=1.11"></script><link rel="canonical" href="/abs/2408.02442"/><meta property="og:type" content="website" />
<meta property="og:site_name" content="arXiv.org" />
<meta property="og:title" content="Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models" />
<meta property="og:url" content="https://arxiv.org/abs/2408.02442v1" />
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="700" />
<meta property="og:image:alt" content="arXiv logo"/>
<meta property="og:description" content="Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large language models (LLMs). This study investigates whether such constraints on generation space impact LLMs&#39; abilities, including reasoning and domain knowledge comprehension. Specifically, we evaluate LLMs&#39; performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. Surprisingly, we observe a significant decline in LLMs&#39; reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks."/>
<meta name="twitter:site" content="@arxiv"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="Let Me Speak Freely? A Study on the Impact of Format Restrictions..."/>
<meta name="twitter:description" content="Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large..."/>
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
<meta name="twitter:image:alt" content="arXiv logo"/>
  <link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/>  <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script>  <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
  <script src="/static/browse/0.3.4/js/toggle-labs.js?20230622" type="text/javascript"></script>
  <script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models" /><meta name="citation_author" content="Tam, Zhi Rui" /><meta name="citation_author" content="Wu, Cheng-Kuang" /><meta name="citation_author" content="Tsai, Yi-Lin" /><meta name="citation_author" content="Lin, Chieh-Yen" /><meta name="citation_author" content="Lee, Hung-yi" /><meta name="citation_author" content="Chen, Yun-Nung" /><meta name="citation_date" content="2024/08/05" /><meta name="citation_online_date" content="2024/08/05" /><meta name="citation_pdf_url" content="http://arxiv.org/pdf/2408.02442" /><meta name="citation_arxiv_id" content="2408.02442" /><meta name="citation_abstract" content="Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large language models (LLMs). This study investigates whether such constraints on generation space impact LLMs&#39; abilities, including reasoning and domain knowledge comprehension. Specifically, we evaluate LLMs&#39; performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. Surprisingly, we observe a significant decline in LLMs&#39; reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks." />
</head>

<body  class="with-cu-identity">
      <aside class="slider-wrapper bps-banner forum blue">
      <a class="close-slider bps-banner" href="#"><img src="/static/browse/0.3.4/images/icons/close-slider.png" alt="close this message"></a>
      <div class="columns">
        <img role="presentation" class="bps-banner-image" src="/static/browse/0.3.4/images/icons/smileybones-infinity-large.png" alt="arXiv Accessibility Forum 2024">
        <div class="copy-donation bps-banner">
          <h2>The Accessibility Forum is back!</h2>
          <p>Coming this September, the Forum is free, virtual, and open to all. <a target="_blank" href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y">Sign Up</a> and <a href="https://accessibility2024.arxiv.org/" target="_blank">Learn more</a>.</p>
        </div>
        <div class="amount-donation bps-banner">
          <div class="donate-cta"><a class="banner_link banner-btn-grad" target="_blank" href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y"><b>Sign Up</b></a></div>
        </div>
      </div>
    </aside>
  <div class="flex-wrap-footer">
    <header>
      <a href="#content" class="is-sr-only">Skip to main content</a>
      <!-- start desktop header -->
      <div class="columns is-vcentered is-hidden-mobile" id="cu-identity">
        <div class="column" id="cu-logo">
          <a href="https://www.cornell.edu/"><img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University" /></a>
        </div><div class="column banner-minimal forum">
            <!-- /from July 7 at 1:00 AM to Sept 2 at 1:00 AM --><p>arXiv's Accessibility Forum starts next month!</p>
            <a href="https://cornell.ca1.qualtrics.com/jfe/form/SV_eEZ1d27LF2fVM7Y" target="_blank">Sign Up</a>

            <!-- /from Sept 2 at 1:00 AM to Sept 13 at 11:30PM --></div><div class="column" id="support-ack">
          <span id="support-ack-url">We gratefully acknowledge support from the Simons Foundation, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and all contributors.</span>
          <a href="https://info.arxiv.org/about/donate.html" class="btn-header-donate">Donate</a>
        </div>
      </div>

      <div id="header" class="is-hidden-mobile">
<a aria-hidden="true" href="{url_path('ignore_me')}"></a>
  <div class="header-breadcrumbs is-hidden-mobile">
    <a href="/"><img src="/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg" alt="arxiv logo" style="height:40px;"/></a> <span>&gt;</span> <a href="/list/cs/recent">cs</a> <span>&gt;</span> arXiv:2408.02442
  </div>
          <div class="search-block level-right">
    <form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
      <div class="field has-addons">
        <div class="control">
          <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
          <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
        </div>
        <div class="control">
          <div class="select is-small">
            <select name="searchtype" aria-label="Field to search">
              <option value="all" selected="selected">All fields</option>
              <option value="title">Title</option>
              <option value="author">Author</option>
              <option value="abstract">Abstract</option>
              <option value="comments">Comments</option>
              <option value="journal_ref">Journal reference</option>
              <option value="acm_class">ACM classification</option>
              <option value="msc_class">MSC classification</option>
              <option value="report_num">Report number</option>
              <option value="paper_id">arXiv identifier</option>
              <option value="doi">DOI</option>
              <option value="orcid">ORCID</option>
              <option value="author_id">arXiv author ID</option>
              <option value="help">Help pages</option>
              <option value="full_text">Full text</option>
            </select>
          </div>
        </div>
        <input type="hidden" name="source" value="header">
        <button class="button is-small is-cul-darker">Search</button>
      </div>
    </form>
  </div>
     </div><!-- /end desktop header -->

      <div class="mobile-header">
        <div class="columns is-mobile">
          <div class="column logo-arxiv"><a href="https://arxiv.org/"><img src="/static/browse/0.3.4/images/arxiv-logomark-small-white.svg" alt="arXiv logo" style="height:60px;" /></a></div>
          <div class="column logo-cornell"><a href="https://www.cornell.edu/">
            <picture>
              <source media="(min-width: 501px)"
                srcset="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg  400w"
                sizes="400w" />
              <source srcset="/static/browse/0.3.4/images/icons/cu/cornell_seal_simple_black.svg 2x" />
              <img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University Logo" />
            </picture>
          </a></div>
          <div class="column nav" id="toggle-container" role="menubar">
            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-white"><title>open search</title><path d="M505 442.7L405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9 0 208 0S0 93.1 0 208s93.1 208 208 208c48.3 0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9 0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7 0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7 0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
                <div class="field has-addons">
                  <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
                  <input type="hidden" name="source" value="header">
                  <input type="hidden" name="searchtype" value="all">
                  <button class="button">GO</button>
                </div>
              </form>
            </div>

            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-white" role="menu"><title>open navigation menu</title><path d="M16 132h416c8.837 0 16-7.163 16-16V76c0-8.837-7.163-16-16-16H16C7.163 60 0 67.163 0 76v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16z"/ ></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <nav class="mobile-menu" aria-labelledby="mobilemenulabel">
                <h2 id="mobilemenulabel">quick links</h2>
                <ul>
                    <li><a href="https://arxiv.org/login">Login</a></li>
                    <li><a href="https://info.arxiv.org/help">Help Pages</a></li>
                    <li><a href="https://info.arxiv.org/about">About</a></li>
                </ul>
              </nav>
            </div>
          </div>
        </div>
      </div><!-- /end mobile-header -->
    </header>

    <main>
      <div id="content">
<!--
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
    <rdf:Description
        rdf:about="/abs/2408.02442"
        dc:identifier="/abs/2408.02442"
        dc:title="Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models"
        trackback:ping="/trackback/2408.02442" />
    </rdf:RDF>
--><div id="abs-outer">

  <div class="leftcolumn">
    <div class="subheader">
      <h1>Computer Science > Computation and Language</h1>
    </div>

    <div class="header-breadcrumbs-mobile">
      <strong>arXiv:2408.02442</strong> (cs)
    </div>
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
<div id="content-inner">
  <div id="abs">
    <div class="dateline">
  [Submitted on 5 Aug 2024]</div>
    <h1 class="title mathjax"><span class="descriptor">Title:</span>Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models</h1>
    <div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Tam,+Z+R">Zhi Rui Tam</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Wu,+C">Cheng-Kuang Wu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Tsai,+Y">Yi-Lin Tsai</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lin,+C">Chieh-Yen Lin</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lee,+H">Hung-yi Lee</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Chen,+Y">Yun-Nung Chen</a></div>            <div id="download-button-info" hidden>View a PDF of the paper titled Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models, by Zhi Rui Tam and 5 other authors</div>
    <a class="mobile-submission-download" href="/pdf/2408.02442">View PDF</a>
    <a class="mobile-submission-download" href="https://arxiv.org/html/2408.02442v1">HTML (experimental)</a>



    <blockquote class="abstract mathjax">
            <span class="descriptor">Abstract:</span>Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large language models (LLMs). This study investigates whether such constraints on generation space impact LLMs&#39; abilities, including reasoning and domain knowledge comprehension. Specifically, we evaluate LLMs&#39; performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. Surprisingly, we observe a significant decline in LLMs&#39; reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks.
    </blockquote>

    <!--CONTEXT-->
    <div class="metatable">
      <table summary="Additional metadata">        <tr>
          <td class="tablecell label">Comments:</td>
          <td class="tablecell comments mathjax">18 pages</td>
        </tr>
<tr>
          <td class="tablecell label">Subjects:</td>
          <td class="tablecell subjects">
            <span class="primary-subject">Computation and Language (cs.CL)</span></td>
        </tr><tr>
          <td class="tablecell label">Cite as:</td>
          <td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2408.02442">arXiv:2408.02442</a> [cs.CL]</span></td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxividv">(or <span class="arxivid">
              <a href="https://arxiv.org/abs/2408.02442v1">arXiv:2408.02442v1</a> [cs.CL]</span> for this version)
          </td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxivdoi">              <a href="https://doi.org/10.48550/arXiv.2408.02442"  id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2408.02442</a><div class="button-and-tooltip">
              <button class="more-info" aria-describedby="more-info-desc-1">
                <svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
                <span class="visually-hidden">Focus to learn more</span>
              </button>
              <!-- tooltip description -->
              <div role="tooltip" id="more-info-desc-1">
                <span class="left-corner"></span>                  arXiv-issued DOI via DataCite</div>
            </div>
          </td>
        </tr></table>
    </div>
  </div>
</div>
    <div class="submission-history">
      <h2>Submission history</h2> From: Zhi Rui Tam [<a href="/show-email/227c2f27/2408.02442">view email</a>]      <br/>    <strong>[v1]</strong>
        Mon, 5 Aug 2024 13:08:24 UTC (252 KB)<br/>
</div>
  </div>
  <!--end leftcolumn-->
<div class="extra-services">    <div class="full-text">
      <a name="other"></a>
      <span class="descriptor">Full-text links:</span>
      <h2>Access Paper:</h2>
      <ul>
  <div id="download-button-info" hidden>
View a PDF of the paper titled Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models, by Zhi Rui Tam and 5 other authors</div><li><a href="/pdf/2408.02442" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2408.02442v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2408.02442" class="abs-button download-eprint">TeX Source</a></li><li><a href="/format/2408.02442" class="abs-button download-format">Other Formats</a></li></ul>
      <div class="abs-license"><a href="http://creativecommons.org/licenses/by-sa/4.0/" title="Rights to this article" class="has_license">
          <img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-sa-4.0.png"/>
          <span>view license</span>
        </a></div>
    </div>
    <!--end full-text-->    <div class="browse">
    Current browse context: <div class="current">cs.CL</div>

  <div class="prevnext">
<span class="arrow">
      <a class="abs-button prev-url" href="/prevnext?id=2408.02442&amp;function=prev&amp;context=cs.CL"
         accesskey="p" title="previous in cs.CL (accesskey p)">&lt;&nbsp;prev</a>
    </span>
    <span class="is-hidden-mobile">&nbsp; | &nbsp;</span>    <span class="arrow">
      <a class="abs-button next-url" href="/prevnext?id=2408.02442&amp;function=next&amp;context=cs.CL" accesskey="n"
         title="next in cs.CL (accesskey n)">next&nbsp;&gt;</a>
    </span><br/>
  </div><div class="list">
    <a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.CL/new">new</a>
    <span class="is-hidden-mobile"> | </span>
    <a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.CL/recent">recent</a>
    <span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.CL/2024-08">2024-08</a>
  </div><div class="abs-switch-cat">
    Change to browse by:
    <div class="switch context-change">
        <a href="/abs/2408.02442?context=cs">cs</a><br class="is-hidden-mobile">
    </div>
  </div>

    </div>

    <div class="extra-ref-cite">
      <h3>References &amp; Citations</h3>
      <ul>
        <li><a  class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2408.02442">NASA ADS</a></li><li><a  class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2408.02442" target="_blank" rel="noopener">Google Scholar</a></li>
        <li><a  class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2408.02442" target="_blank" rel="noopener">Semantic Scholar</a></li>
      </ul>
      <div style="clear:both;"></div>
    </div>

<div class='extra-ref-cite'>
    <a id='bib-cite-css' hidden='true' href='/static/browse/0.3.4/css/cite.css'>a</a>

    <span id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</span>
    <span id='bib-cite-loading' hidden='true'>Loading...</span>
</div>

<div id='bib-cite-modal' class='bib-modal' hidden='true'>
    <div class='bib-modal-content'>
        <div class='bib-modal-title'>
            <h2>BibTeX formatted citation</h2>
            <span class='bib-modal-close' >&times;</span>
        </div>
        <div>
            <textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
        </div>
        <div>
            <span>Data provided by: </span>
            <a id='bib-cite-source-api'></a>
        </div>
    </div>
</div><div class="bookmarks">
  <div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2408.02442&amp;description=Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models"
     title="Bookmark on BibSonomy">
    <img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
         alt="BibSonomy logo"/>
  </a>
  <a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2408.02442&amp;title=Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models"
     title="Bookmark on Reddit">
    <img src="/static/browse/0.3.4/images/icons/social/reddit.png"
         alt="Reddit logo"/>
  </a>
</div>  </div>
  <!--end extra-services-->
<!-- LABS AREA -->
<div id="labstabs">
  <div class="labstabs"><input type="radio" name="tabs" id="tabone"checked="checked">
    <label for="tabone">Bibliographic Tools</label>
    <div class="tab labs-display-bib">
      <h1>Bibliographic and Citation Tools</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input id="bibex-toggle" type="checkbox" class="lab-toggle">
              <span class="slider"></span>
              <span class="is-sr-only">Bibliographic Explorer Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
          </div>
        </div>
<div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="litmaps-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
                aria-labelledby="label-for-litmaps">
              <span class="slider"></span>
              <span class="is-sr-only">Litmaps Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
          </div>
        </div>
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="scite-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
                aria-labelledby="label-for-scite">
              <span class="slider"></span>
              <span class="is-sr-only">scite.ai Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
          </div>
        </div>
      </div>
        <div class="labs-content-placeholder labs-display" style="display: none;"></div>
        <div style="min-height: 15px" id="litmaps-open-in"></div>
        <div style="min-height: 15px" id="scite-open-in"></div>
    </div>


    <input type="radio" name="tabs" id="tabtwo">
    <label for="tabtwo">Code, Data, Media</label>
    <div class="tab">
      <h1>Code, Data and Media Associated with this Article</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input        
                id="catalyzex-toggle"
                data-script-url="/static/browse/0.3.4/js/catalyzex.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="dagshub-toggle"
                data-script-url="/static/browse/0.3.4/js/dagshub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
              <span class="slider"></span>
              <span class="is-sr-only">DagsHub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
          </div>
        </div>
  
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="gotitpub-toggle"
                data-script-url="/static/browse/0.3.4/js/gotitpub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
              <span class="slider"></span>
              <span class="is-sr-only">GotitPub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="paperwithcode-toggle"
                data-script-url="/static/browse/0.3.4/js/paperswithcode.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-pwc">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-pwc">Papers with Code</span> <em>(<a href="https://paperswithcode.com/" target="_blank">What is Papers with Code?</a>)</em>
          </div>
        </div>


        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="sciencecast-toggle"
                data-script-url="/static/browse/0.3.4/js/sciencecast.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
              <span class="slider"></span>
              <span class="is-sr-only">ScienceCast Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
          </div>
        </div>
      </div>

      <div id="catalyzex-output" style="display:none"></div>
      <div id="dagshub-output" style="display:none"></div>
      <div id="gotitpub-output" style="display:none"></div>
      <div id="pwc-output" style="display:none"></div>
      <div id="pwc-data-output" style="display:none"></div>
      <div id="sciencecast-output" style="display:none"></div>
    </div>


      <input type="radio" name="tabs" id="labstabs-demos-input">
      <label for="labstabs-demos-input" id="labstabs-demos-label">Demos</label>
      <div class="tab">
        <h1>Demos</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="replicate-toggle"
                  data-script-url="/static/browse/0.3.4/js/replicate.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
                <span class="slider"></span>
                <span class="is-sr-only">Replicate Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="spaces-toggle"
                  data-script-url="/static/browse/0.3.4/js/spaces.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="txyz-toggle"
                  data-script-url="/static/browse/0.3.4/js/txyz.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
            </div>
          </div>
        </div>
        <div id="replicate-output"></div>
        <div id="spaces-output"></div>
        <div id="txyz-output"></div>
      </div>
      <input type="radio" name="tabs" id="tabfour">
      <label for="tabfour">Related Papers</label>
      <div class="tab">
        <h1>Recommenders and Search Tools</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="influenceflower-toggle"
                data-script-url="/static/browse/0.3.4/js/influenceflower.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
                <span class="slider"></span>
                <span class="is-sr-only">Link to Influence Flower</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="connectedpapers-toggle" type="checkbox" class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
                       aria-labelledby="label-for-connected-papers" >
                <span class="slider"></span>
                <span class="is-sr-only">Connected Papers Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
                <span class="slider"></span>
                <span class="is-sr-only">Core recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
            </div>
          </div></div>
        <div id="influenceflower-output"></div>
        <div id="influenceflower-output-graph" style="display:none">
          <ul class="flower-tabs">
            <li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
          </ul>
          <div class="flower-tab-content">
            <div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
            <div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
            <div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
            <div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
          </div>
        </div>
        <div style="min-height: 15px" id="connectedpapers-output"></div>
        <div id="coreRecommenderOutput"></div>
        <div id="iarxivOutput"></div>
      </div>

      <input type="radio" name="tabs" id="tabfive">
      <label for="tabfive">
        About arXivLabs
      </label>
      <div class="tab">
        <div class="columns">
          <div class="column">
            <h1>arXivLabs: experimental projects with community collaborators</h1>
            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
            <p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
          </div>
          <div class="column is-narrow is-full-mobile">
            <p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
          </div>
        </div>
      </div>

    </div>
</div>
<!-- END LABS AREA -->
  <div class="endorsers">
    <a href="/auth/show-endorsers/2408.02442" class="endorser-who">Which authors of this paper are endorsers?</a> |
    <a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
    <span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
  </div>
  <script type="text/javascript" language="javascript">mathjaxToggle();</script>
</div>
      </div>
    </main>

    <footer style="clear: both;">
      <div class="columns is-desktop" role="navigation" aria-label="Secondary" style="margin: -0.75em -0.75em 0.75em -0.75em">
        <!-- Macro-Column 1 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/about">About</a></li>
                <li><a href="https://info.arxiv.org/help">Help</a></li>
              </ul>
            </div>
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>contact arXiv</title><desc>Click here to contact arXiv</desc><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>
                  <a href="https://info.arxiv.org/help/contact.html"> Contact</a>
                </li>
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>subscribe to arXiv mailings</title><desc>Click here to subscribe</desc><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
                  <a href="https://info.arxiv.org/help/subscribe"> Subscribe</a>
                </li>
              </ul>
            </div>
          </div>
        </div>
        <!-- End Macro-Column 1 -->
        <!-- Macro-Column 2 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/license/index.html">Copyright</a></li>
                <li><a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy Policy</a></li>
              </ul>
            </div>
            <div class="column sorry-app-links">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/web_accessibility.html">Web Accessibility Assistance</a></li>
                <li>
                  <p class="help">
                    <a class="a11y-main-link" href="https://status.arxiv.org" target="_blank">arXiv Operational Status <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 512" class="icon filter-dark_grey" role="presentation"><path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"/></svg></a><br>
                    Get status notifications via
                    <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/email/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>email</a>
                    or <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/slack/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-black" role="presentation"><path d="M94.12 315.1c0 25.9-21.16 47.06-47.06 47.06S0 341 0 315.1c0-25.9 21.16-47.06 47.06-47.06h47.06v47.06zm23.72 0c0-25.9 21.16-47.06 47.06-47.06s47.06 21.16 47.06 47.06v117.84c0 25.9-21.16 47.06-47.06 47.06s-47.06-21.16-47.06-47.06V315.1zm47.06-188.98c-25.9 0-47.06-21.16-47.06-47.06S139 32 164.9 32s47.06 21.16 47.06 47.06v47.06H164.9zm0 23.72c25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06H47.06C21.16 243.96 0 222.8 0 196.9s21.16-47.06 47.06-47.06H164.9zm188.98 47.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06h-47.06V196.9zm-23.72 0c0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06V79.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06V196.9zM283.1 385.88c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06v-47.06h47.06zm0-23.72c-25.9 0-47.06-21.16-47.06-47.06 0-25.9 21.16-47.06 47.06-47.06h117.84c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06H283.1z"/></svg>slack</a>
                  </p>
                </li>
              </ul>
            </div>
          </div>
        </div> <!-- end MetaColumn 2 -->
        <!-- End Macro-Column 2 -->
      </div>
    </footer>
  </div>

  <script src="/static/base/1.0.1/js/member_acknowledgement.js"></script>

</body>

</html>
2408.02442
 Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large language models (LLMs). This study investigates whether such constraints on generation space impact LLMs' abilities, including reasoning and domain knowledge comprehension. Specifically, we evaluate LLMs' performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. Surprisingly, we observe a significant decline in LLMs' reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks. 
abstract
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Abstract
2408.02442
 The few-shot in-context learning (Brown et al., 2020) and instruction-following (Wei et al., 2021) capabilities of large language models (LLMs) have enabled their out-of-the-box usage to solve downstream tasks. However, a major obstacle to incorporating LLMs into industrial applications is their lack of adherence to standardized output formats.  This inconsistency complicates output parsing and undermines the reliability of these models.  One common approach to overcoming this obstacle is *structured generation*, which involves providing output in standardized formats like JSON or XML through *format restrictions*. These restrictions can be implemented in various ways, such as instructing LLMs to adhere to specified formats with *format-restricting instructions*, or using industrial solutions like JSON mode (OpenAI, 2024; Gemini, 2024), Instructor (Liu, 2024), or Guardrails (PrefectHQ, 2024). These strategies  ![0_image_0.png](0_image_0.png)  simplify parsing workflows and streamline the integration of LLMs into real-world applications.  Due to the growing demand for structured generation, the research community has shown increased interest in investigating LLMs' format-following abilities. For example, IFEval (Zhou et al., 2023), INFOBENCH (Qin et al., 2024), and FOFO (Xia et al., 2024) focus on evaluating LLMs' instructionfollowing capabilities, including format adherence.  However, these studies do not address a critical question for industrial applications: Do formatrestricting instructions affect the quality of LLMs' generated content? In other words, they fail to explore whether format restrictions degrade LLMs' performance, which has great business impacts.  This performance degradation is shown in Figure 1.  In this work, we address the aforementioned research question through extensive empirical experiments. We present a comprehensive analysis of the potential impacts of format-restricting instructions on LLMs' performance across a wide range of tasks. The formats studied include commonly used schemas such as JSON, XML, and YAML.  To the best of our knowledge, this is the first systematic investigation into the relationship between format-restricting instructions and the quality of generated content. Our contributions are twofold:  - We observe declines in LLMs' reasoning abilities under format restrictions, with stricter constraints generally leading to greater performance degradation in reasoning tasks.  - We offer insights into why performance degrades due to format constraints and propose simple approaches to mitigate these issues, thereby achieving both consistent formats and optimal performance. 
introduction
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Introduction
2408.02442
 To study different levels of format restrictions on downstream performance, we adopt the following three common methodologies in our experiments: Constrained Decoding (JSON-mode): Constrained decoding is a technique that limits the output of LLMs by enforcing predefined token space during the generation process. Among mainstream LLM providers, **JSON mode** is a widely implemented instance of this technique, especially due to its extensive use in industrial settings. This mode, available as a hyperparameter flag in OpenAI and Gemini APIs, ensures the output is valid JSON.  It is assumed that the implementation is similar to the constrained decoding methods described by (Willard and Louf, 2023; Koo et al., 2024), and provided in Text-Generation-Inference1.  Format-Restricting Instructions (FRI): They direct the LLM to generate responses in standardized formats such as JSON, XML, and YAML, adhering to specified schemas. These instructions ensure  ![1_image_0.png](1_image_0.png)  that the generated output follows a structured format, facilitating the extraction and evaluation of the final answer. This approach is more relaxed than constrained decoding, as it does not enforce a predefined token space.  NL-to-Format: This two-step process first instructs the LLM to answer the question in natural language, and then instructs it to convert its response into the target format schema. As the most relaxed version of structured generation, this method decouples *content generation* from format adherence, aiming to maintain the performance of unrestricted natural language responses while still providing structured output. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
**Methodology For Structured Generation**
2408.02442
 We adopt datasets from various domains, categorized by the primary skills they assess: 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Experiments 3.1 Datasets
2408.02442
 GSM8K (Cobbe et al., 2021): A collection of mathematical problems set in natural language contexts,  ![2_image_0.png](2_image_0.png)  reflecting daily life scenarios. This dataset challenges LLMs to generate necessary intermediate reasoning steps.  Last Letter Concatenation (Wei et al., 2022): This task requires LLMs to produce a string by concatenating the last letters of a sequence of words, testing their ability to perform symbolic reasoning.  Shuffled Objects (Ghazal et al., 2013): This evaluate set from BigBench evaluates the ability to infer the final state given an initial state and a sequence of shuffling events. We use the entire validation set in our experiments. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
1 Reasoning Tasks
2408.02442
 DDXPlus (Tchango et al., 2022): A multiplechoice medical diagnosis dataset where LLMs must select the most appropriate diagnosis from 49 possible diseases based on a given patient profile. We use a subset provided by StreamBench (Wu et al., 2024) due to the extensive number of questions.  MultiFin (Jørgensen et al., 2023): A multi-choice financial dataset that requires classifying a given paragraph into one of five categories.  Sports Understanding (Ghazal et al., 2013): This task from BigBench tests LLMs' ability to determine whether an artificially constructed sentence relating to sports is plausible or implausible.  NI - Task 280 (Mishra et al., 2022): A multiplechoice stereotype classification task based on a given paragraph. We included this task as it has been found to be sensitive to change in prompt formatting, with performance variations of up to 56% (Sclar et al., 2023). 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
2 Classification Tasks
2408.02442
 For all experiments, we compare *gpt-3.5-turbo0125* (OpenAI, 2023), *claude-3-haiku-20240307* (Team, 2024a), *gemini-1.5-flash* (Team et al., 2023). For open weights model we use *LLaMA3-8B-Instruct* (Team, 2024b) and *Gemma-2-9BInstruct* (Team et al., 2024) inference using TextGeneration-Server for its support in **JSON mode**2. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Model
2408.02442
 Metrics. To assess the performance of the models across the diverse range of tasks, we employ taskspecific evaluation metrics. For the classificationbased tasks (Sports Understanding, DDXPlus, Natural Instruction Task 280, and MultiFin), we use accuracy as the primary metric. For the Last Letter Concatenation and GSM8K, we utilize the exact match metric where the final answer must be the extact string match with the actual answer. Perfect Text Parser. To disentangle format errors from the actual performance of the generated content, we use an LLM prompted to extract the final answer from the text, rather than relying on regex or string parsers. This approach acts as a perfect parser, minimizing errors introduced when switching between different models. Our ablation study, comparing different models, found that claude-3haiku-20240307 is the most consistent when using gpt-4-turbo as a human reference, compared to four other low-cost APIs. Full results can be found in Appendix B.  Consideration for Prompt Sensitivity. Previous studies (Chen et al., 2023; Sclar et al., 2023; Zhu et al., 2023) have shown that LLMs are sensitive to slight variations in prompts. To account for this, we evaluate our approach by nine prompt combinations: three task descriptions and three JSON, XML, and YAML schemas with slight variations in wording or format. For natural language prompting, we include three variations in text formats (e.g., Give your reason first followed by your answers).  Details of the task description prompts and FRI prompts can be found in Appendix F. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Evaluation Method
2408.02442
 We investigate the effects of format restrictions on LLM performance by examining three progressively relaxed prompting approaches: JSON-mode, FRI, and NL-to-Format conversion.  We evaluate these approaches on datasets with exact match scores: GSM8K and Last Letter Concatenation presented in Figure 2. Surprisingly, JSON-mode performs significantly worse than FRI (JSON) on the Last Letter task. Upon inspection, we found that 100% of GPT 3.5 Turbo JSON-mode responses placed the "answer" key before the "reason" key, resulting in zero-shot direct answering instead of zero-shot chain-of-thought reasoning.  Comparing NL-to-Format with unrestricted Natural Language responses, we observe nearly identical performance across most models, as both derive answers from the same initial natural language response. However, NL-to-Format occasionally introduces generation errors, leading to slightly lower performance for LLaMA 3 8B Instruct, while other models maintain consistent scores across both settings. These findings suggest that the degree and implementation of format restrictions can significantly impact LLM performance, particularly in reasoning tasks. The order of keys in structured outputs and the decoupling of reasoning from format adherence emerge as important factors in maintaining LLM capabilities while providing structured responses.  When evaluating classification datasets, we observe a different trend compared to reasoning tasks, as illustrated in Figure 3. Notably, in the DDXPlus dataset, Gemini 1.5 Flash demonstrates a significant performance boost when JSON-mode is enabled. Across other classification datasets, JSONmode performs competitively, and in some cases, surpasses the other three methodologies.  We hypothesize that JSON-mode improves classification task performance by constraining possible answers resulted in reducing errors in answer selection. Conversely, natural language responses may introduce distractions, leading to parsing errors. These findings suggest format restrictions' impact on LLM performance is task-dependent: stringent formats may hinder reasoning-intensive tasks but enhance accuracy in classification tasks requiring structured outputs. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Main Results 4.1 Impact Of Format Restriction On Final Results
2408.02442
 To further investigate the effects of format restrictions, we examine a variation of the Soft Restrict setting where we remove the schema restriction from the prompt description. Instead of providing a specific schema (e.g., *"Reply your answer in JSON* format with the following schema: { "reason": ..., "answer": ... }"), we simply instruct the LLM to output in the target format language (e.g., "Reply your answer in JSON format."). Table 1 illustrates the effects of removing the schema restriction on the GSM8K dataset. We observe significant improvements in average scores and lower standard deviations across different prompt perturbations for Claude 3 Haiku, GPT-3.5 Turbo, and LLaMA 3 8B Instruct. These results suggest that while structured outputs can be beneficial for downstream processing, overly restrictive schemas may hinder LLM performance, particularly in reasoning-intensive tasks.  This finding suggests that a balance must be struck between the desire for easily parseable, structured outputs and the need to preserve the LLM's inherent reasoning abilities. Practitioners may want to consider using looser format restrictions when dealing with complex reasoning tasks, while still maintaining some level of structure to facilitate downstream processing. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Discussion 5.1 Impact On Looser Format Restriction
2408.02442
 In this section we ablate the format language by comparing not just JSON but also XML and YAML format. Since all 3 language comes in different grammar syntax rules and restriction. We deduce each models might perform differently for example Claude-3-Haiku uses XML for tool use schema so On hint sight we do not see any structure format which consistency stands out from others which generalized across all models in Figure 4. For Gemini model, we found JSON is more consistent however it does not always outperform other format.  In Table 8 we found in classification task JSONmode performs much better than text due to the restriction on answer space. However in reasoning related task, JSON-mode failed to adhere to the 4  ![4_image_0.png](4_image_0.png)  Model Text JSON XML YAML  gemini-1.5-flash 89.33 **89.66 89.26 89.21**  (0.8) (0.3) (0.3) (0.4)  + schema constraint - 89.21 88.20 87.42  - (1.5) (2.2) (3.7)  claude-3-haiku 86.51 **86.99 86.96 82.89**  (0.8) (0.2) (0.6) (5.7)  + schema constraint - 23.44 79.76 80.63  - (22.9) (7.0) (2.8)  gpt-3.5-turbo 75.99 **74.70 60.45** 71.58  (3.1) (1.1) (7.2) (3.0)  + schema constraint - 49.25 45.06 **73.85**  - (12.0) (19.9) (5.6)  LLaMA-3-8B 75.13 **64.67 65.07 69.41**  (0.9) (2.23) (0.56) (0.95)  + schema constraint - 48.90 56.74 46.08  - (6.7) (8.3) (16.8)  order of reasoning first followed by answer causing a large drop in final performance. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Comparison Across Different Formats
2408.02442
 We initially hypothesized that the performance gap between text and structured formats might be attributed to parsing errors during answer extraction.  However, our analysis of error rates across different formats and models, as shown in Table 2, reveals that this is not the primary factor. In fact, Gemini 1.5 Flash and GPT 3.5 Turbo exhibit near zero parsing failures in all three format. In the LLaMA 3 8B setting, the parsing error rate for the Last Letter task in JSON format is only 0.148%, yet there exists a substantial 38.15% performance gap as seen in Table 1.  This finding suggests that the performance dif-  ![4_image_1.png](4_image_1.png)  ferences between formats are not primarily due to parsing errors, but rather to the impact of format restrictions on the LLM's reasoning and generation processes. However, we discovered that parsing errors, when present, can be effectively mitigated through a simple corrective step.  By prompting Claude-3-Haiku to reformat any output with parsing errors for both Claude 3 Haiku and LLaMA 3 8B (the two models with the highest percentage of parsing errors), we observed improved scores in JSON and YAML formats, as illustrated in Figure 5. This approach demonstrates the potential for enhancing the reliability of structured outputs without sacrificing the benefits of format-specific optimizations. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Structure Format And Parsing Error Rates
2408.02442
 Our study can be summarized into two genres : reasoning ability of LLM and format following.  In study of LLMs reasoning ability, early work  | Task           | Reasoning   | Classification   |       |         |        |         |          | |----------------|-------------|------------------|-------|---------|--------|---------|----------| | Model          | Format      | Last Letter      | GSM8K | DDXPlus | Sports | Task280 | MultiFin | | Gemini-Flash   | JSON        | 0.0              | 0.03  | 0.37    | 0.0    | 0.0     | 0.0      | | XML            | 0.0         | 0.19             | 1.26  | 0.0     | 0.22   | 0.0     |          | | YAML           | 0.0         | 0.0              | 0.68  | 0.06    | 6.46   | 0.0     |          | | Claude-3-Haiku | JSON        | 3.48             | 60.07 | 0.09    | 0.0    | 10.26   | 0.0      | | XML            | 0.0         | 1.85             | 0.48  | 0.0     | 0.41   | 0.0     |          | | YAML           | 0.0         | 0.0              | 86.66 | 1.02    | 0.13   | 0.0     |          | | GPT-3.5-Turbo  | JSON        | 0.0              | 0.13  | 0.0     | 0.0    | 0.0     | 0.0      | | XML            | 0.0         | 0.24             | 0.35  | 0.0     | 0.0    | 0.0     |          | | YAML           | 0.0         | 0.0              | 0.32  | 1.23    | 0.08   | 0.0     |          | | LLaMA 3 8B     | JSON        | 0.15             | 22.75 | 1.63    | 0.28   | 1.61    | 0.0      | | XML            | 17.93       | 7.62             | 32.45 | 6.54    | 22.04  | 5.78    |          | | YAML           | 32.40       | 33.18            | 34.40 | 7.16    | 2.19   | 0.14    |          |  ![5_image_0.png](5_image_0.png)  by (Kojima et al., 2022) found using "Think stepby-step" can elicit reasoning ability without few shot examples. Subsequent study (Jin et al., 2024) shows that the number of reasoning steps correlates with the final accuracy. Recent work by (Wang and Zhou, 2024) found Chain-of-Thought (CoT) reasoning seed prompt (Kojima et al., 2022) can be removed with a carefully crafted CoT decoding schema.  The exploration of LLMs' ability to follow instructions and produce responses in specified formats was first addressed by IFEval (Zhou et al., 2023) which designed to evaluate the general instruction-following ability of LLMs, and it contains a subset of test instances specifically assessing format-following. INFOBENCH (Qin et al., 2024) introduces a broader coverage of instructions and conducts a more fine-grained analysis by decomposing the instructions into different categories, including format specifications. FOFO (Xia et al., 2024) is a benchmark solely focused on the formatfollowing ability of LLMs. However, these works do not explore if format instruction interfere with downstream performance. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Related Work
2408.02442
 Our study reveals that structured generation constraints significantly impact LLM performance across various tasks. Format restrictions, particularly constrained decoding (JSON-mode), can hinder reasoning abilities while enhance classification task accuracy. Looser format restrictions generally improve performance and reduce variance in reasoning tasks. Parsing errors, while not the primary cause of performance differences, can be mitigated through corrective prompting. These findings underscore the importance of balancing format adherence, reasoning capabilities, and cost efficiency in LLM applications. Given that our study focuses on reasoning-intensive tasks, future work should explore how reasoning tasks of varying difficulty, from intensive to simple, are affected by restrictive formats and LLMs. To mitigate the performance degradation of LLMs due to restrictive formats, future studies should include a wider range of training data that contains instructions in various restrictive formats in local LLMs. 
conclusion
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Conclusion
2408.02442
 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.  Yulin Chen, Ning Ding, Xiaobin Wang, Shengding Hu, Haitao Zheng, Zhiyuan Liu, and Pengjun Xie. 2023.  Exploring lottery prompts for pre-trained language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15428–15444.  Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  2021. Training verifiers to solve math word problems. *arXiv preprint arXiv:2110.14168*.  Google Gemini. 2024. Generate json output with the gemini api. https://ai.google.dev/ gemini-api/docs/json-mode?lang=python. Accessed on 2024-07-02.  Ahmad Ghazal, Tilmann Rabl, Minqing Hu, Francois Raab, Meikel Poess, Alain Crolotte, and Hans-Arno Jacobsen. 2013. Bigbench: Towards an industry standard benchmark for big data analytics. In *Proceedings of the 2013 ACM SIGMOD international conference on Management of data*, pages 1197–1208.  Mingyu Jin, Qinkai Yu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, Mengnan Du, et al.  2024. The impact of reasoning step length on large language models. *arXiv preprint arXiv:2401.04925*.  Rasmus Kær Jørgensen, Oliver Brandt, Mareike Hartmann, Xiang Dai, C. Igel, and Desmond Elliott. 2023.  Multifin: A dataset for multilingual financial nlp. In ACL Findings.  Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.  Terry Koo, Frederick Liu, and Luheng He. 2024.  Automata-based constraints for language model decoding. *arXiv e-prints*. Jason Liu. 2024. instructor.  Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions.  In ACL. OpenAI. 2023. Gpt-4 technical report.  OpenAI. 2024. Json mode. https://platform.  openai.com/docs/guides/text-generation/ json-mode. Accessed on 2024-07-02.  7 
references
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
References
2408.02442
 Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. Infobench: Evaluating instruction following ability in large language models. *arXiv preprint arXiv:2401.03601*.  Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.  In The Twelfth International Conference on Learning Representations.  Arsène Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. 2022. Ddxplus: a new dataset for automatic medical diagnosis. In *Proceedings of the 36th International Conference on Neural Information Processing Systems*, pages 31306– 31318. Anthropic Team. 2024a. Introducing the next generation of claude.  Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. *arXiv preprint* arXiv:2312.11805.  Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. Meta LLaMA Team. 2024b. Introducing meta llama 3: The most capable openly available llm to date.  Xuezhi Wang and Denny Zhou. 2024. Chain-ofthought reasoning without prompting. *ArXiv*, abs/2402.10200.  Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. *arXiv preprint* arXiv:2109.01652.  Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In *Advances in* Neural Information Processing Systems.  Brandon T Willard and Rémi Louf. 2023. Efficient guided generation for large language models. arXiv e-prints, pages arXiv–2307.  Cheng-Kuang Wu, Zhi Rui Tam, Chieh-Yen Lin, YunNung Chen, and Hung yi Lee. 2024. Streambench: Towards benchmarking continuous improvement of language agents.  Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. Fofo: A benchmark to evaluate llms' format-following capability. *arXiv preprint* arXiv:2402.18667.  Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. *arXiv preprint* arXiv:2311.07911.  Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, et al. 2023. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. *arXiv preprint* arXiv:2306.04528. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Prefecthq. 2024. Marvin.
2408.02442
 This study contains two primary limitations. First, due to cost constraints, we were unable to include results from more powerful language models such as LLaMA 70B or GPT-4o in our experiments. The inclusion of these models could potentially provide additional insights into how performance scales with model size and architecture. Second, our evaluation dataset, while diverse, is limited in scope. A broader range of tasks and domains could offer a more comprehensive assessment of the proposed approach's effectiveness and generalizability. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
A Limitation
2408.02442
 To select the best and low cost answer LLM parser, we select 200 samples from six datasets response in natural language format which a total of 1,200 samples. We then use *gpt-4-turbo* as best LLM answer parser as our reference and calculate the kappa cohen score with 3 LLMs candidates: *gemini-1.5-* flash, *claude-3-haiku-20240307* and *llama-3-8binstruct* in Figure 7. Result shows *claude-3-haiku20240307* has the highest aggreement with gpt-4turbo at 0.86 followed by *llama-3-8b-instruct*. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
B Choosing Which Llms As Answer Extraction
2408.02442
 An important consideration in deploying LLM applications in industry settings is the associated token cost. We analyzed the input and output tokens across our experiments for all models and formats.  ![7_image_0.png](7_image_0.png)   | Model            | text   | json   | xml   | yaml   | |------------------|--------|--------|-------|--------| | LLaMA-3-8b       | 0.11   | 0.09   | 0.09  | 0.08   | | Gemini-1.5-Flash | 0.20   | 0.21   | 0.21  | 0.19   | | Claude-3-Haiku   | 0.20   | 0.30   | 0.30  | 0.29   | | GPT-3.5-Turbo    | 0.35   | 0.23   | 0.24  | 0.23   |  For brevity, we present the averaged results from all six datasets in Table 3. Our analysis reveals that text and YAML formats generally incur similar costs. Interestingly, we found that YAML is the most cost-effective format for LLaMA-3-8B, Gemini-1.5-Flash, and GPT-3.5-Turbo. Surprisingly, for Claude-3-Haiku, the lowest cost is associated with the text format, which is unexpected given the prevalence of XML examples in their documentation for tool use. The full cost breakdown for each dataset can be found in Table 4, providing a more detailed view for practitioners interested in fine-tuning their approach for specific use cases. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
C Cost Comparison Across Different Formats
2408.02442
 We also tested additional models from Mistral and OpenAI : Mistral-7b-v0.3, *GPT-4o-mini-2024* on format prompt variation in GSM8K, Last Letter, Shuffled Object, Sports Understanding, MultiFin, NL Task 280 and DDXPlus. The result is visualized in Figure 8.  | gemini-1.5-flash   | llama-3-8b   | claude-3-haiku   | gpt-3.5-turbo   |      |      |      |      |      |      |      |      |      |      | |--------------------|--------------|------------------|-----------------|------|------|------|------|------|------|------|------|------|------| | Dataset            | Format       | In               | Out             | Tot  | In   | Out  | Tot  | In   | Out  | Tot  | In   | Out  | Tot  | | text               | 0.04         | 0.09             | 0.12            | 0.02 | 0.02 | 0.04 | 0.03 | 0.12 | 0.15 | 0.05 | 0.07 | 0.12 |      | | json               | 0.04         | 0.10             | 0.14            | 0.02 | 0.03 | 0.05 | 0.03 | 0.17 | 0.21 | 0.06 | 0.05 | 0.11 |      | | lastletter         | xml          | 0.04             | 0.10            | 0.14 | 0.02 | 0.03 | 0.05 | 0.03 | 0.15 | 0.18 | 0.06 | 0.07 | 0.13 | | yaml               | 0.04         | 0.09             | 0.13            | 0.02 | 0.02 | 0.05 | 0.03 | 0.14 | 0.18 | 0.06 | 0.09 | 0.14 |      | | text               | 0.05         | 0.13             | 0.18            | 0.03 | 0.03 | 0.06 | 0.04 | 0.23 | 0.27 | 0.07 | 0.16 | 0.23 |      | | json               | 0.05         | 0.14             | 0.20            | 0.03 | 0.03 | 0.07 | 0.04 | 0.29 | 0.33 | 0.08 | 0.12 | 0.19 |      | | gsm8k              | xml          | 0.06             | 0.14            | 0.19 | 0.03 | 0.03 | 0.07 | 0.05 | 0.27 | 0.32 | 0.08 | 0.12 | 0.20 | | yaml               | 0.05         | 0.13             | 0.18            | 0.03 | 0.03 | 0.06 | 0.04 | 0.28 | 0.32 | 0.08 | 0.14 | 0.22 |      | | text               | 0.05         | 0.01             | 0.06            | 0.03 | 0.00 | 0.03 | 0.03 | 0.02 | 0.05 | 0.07 | 0.02 | 0.09 |      | | json               | 0.05         | 0.02             | 0.07            | 0.03 | 0.00 | 0.03 | 0.04 | 0.05 | 0.09 | 0.07 | 0.02 | 0.09 |      | | multifin           | xml          | 0.05             | 0.02            | 0.07 | 0.03 | 0.01 | 0.04 | 0.04 | 0.04 | 0.08 | 0.08 | 0.03 | 0.10 | | yaml               | 0.05         | 0.01             | 0.06            | 0.03 | 0.00 | 0.03 | 0.04 | 0.02 | 0.06 | 0.07 | 0.01 | 0.08 |      | | text               | 0.04         | 0.04             | 0.08            | 0.02 | 0.01 | 0.03 | 0.03 | 0.10 | 0.13 | 0.05 | 0.05 | 0.10 |      | | json               | 0.04         | 0.06             | 0.10            | 0.02 | 0.01 | 0.04 | 0.03 | 0.11 | 0.15 | 0.06 | 0.07 | 0.12 |      | | sports             | xml          | 0.04             | 0.07            | 0.11 | 0.02 | 0.02 | 0.04 | 0.03 | 0.14 | 0.17 | 0.06 | 0.08 | 0.14 | | yaml               | 0.04         | 0.05             | 0.08            | 0.02 | 0.01 | 0.04 | 0.03 | 0.12 | 0.15 | 0.05 | 0.06 | 0.11 |      | | text               | 0.04         | 0.05             | 0.09            | 0.03 | 0.01 | 0.03 | 0.03 | 0.05 | 0.08 | 0.06 | 0.04 | 0.11 |      | | json               | 0.05         | 0.04             | 0.08            | 0.03 | 0.01 | 0.03 | 0.04 | 0.07 | 0.11 | 0.07 | 0.04 | 0.11 |      | | task280            | xml          | 0.05             | 0.04            | 0.09 | 0.03 | 0.01 | 0.04 | 0.04 | 0.08 | 0.11 | 0.07 | 0.05 | 0.12 | | yaml               | 0.04         | 0.03             | 0.07            | 0.03 | 0.01 | 0.03 | 0.04 | 0.05 | 0.09 | 0.06 | 0.03 | 0.10 |      | | text               | 0.26         | 0.15             | 0.41            | 0.15 | 0.04 | 0.18 | 0.19 | 0.20 | 0.38 | 0.38 | 0.21 | 0.59 |      | | json               | 0.22         | 0.18             | 0.41            | 0.13 | 0.06 | 0.19 | 0.19 | 0.33 | 0.52 | 0.34 | 0.15 | 0.48 |      | | ddxplus            | xml          | 0.23             | 0.19            | 0.42 | 0.14 | 0.06 | 0.19 | 0.19 | 0.37 | 0.56 | 0.34 | 0.18 | 0.51 | | yaml               | 0.22         | 0.15             | 0.37            | 0.13 | 0.05 | 0.18 | 0.19 | 0.31 | 0.50 | 0.33 | 0.15 | 0.48 |      |  Table 4: Performance comparison of different models across various datasets and formats. Values represent processing times in seconds for Input (In), Output (Out), and Total (Tot). 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
D Additional Models
2408.02442
 To answer the difference between using regex parser to extract the final strict match answer, we calculate the Exact Match score in GSM8K results using the prompt format template "The final answer is". Table 5 results reveal a significant gap between regex match and LLM as final answer parser in EM score across various language models, highlighting the limitations of using only one strict regex matching for different models. For example, GPT-3.5-Turbo shows a 31.8 percentage point improvement from regex match (43.7%) to overall accuracy (75.5%), while Gemini-1.5-Flash exhibits an even larger 43.5 point difference. This pattern is consistent across all models, with mistral7b demonstrating the most dramatic 42 point increase. These disparities underscore the value of using LLMs as answer parsers, as they can understand and evaluate responses beyond literal string matching, accounting for paraphrases and contextual understanding, thus providing a more nuanced and accurate assessment in text-based tasks.  | Model            | Regex Match   | LLM Match   | |------------------|---------------|-------------| | GPT-3.5-Turbo    | 43.7          | 75.5        | | Gemini-1.5-Flash | 25.8          | 69.3        | | Claude-3-Haiku   | 67.4          | 85.8        | | Gemma2-9b        | 82.5          | 86.0        | | LLaMA-3-8b       | 46.9          | 55.7        | | Mistral-7b-v0.3  | 10.4          | 52.4        |  Table 5: Comparison of model performance on regex match "*The final answer is (\d+)*" accuracy and using Claude-3-Haiku as answer parser. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
E Comparison Between Using Regex And Llm As Answer Parser In Gsm8K
2408.02442
 For each task we fix the same template and only swapping the task description, format description, few shots example and question text.  ![9_image_0.png](9_image_0.png)  ![9_image_1.png](9_image_1.png)  Task Description A task description describes the task and the final goal of the task.  Format Description A format description includes the target format (ie JSON, XML or YAML) and a targeted schema we intent the LLM response to adhere to.  For each description slot, we create 3 variations each which results in 9 prompt combinations. Each variation must retain the original meaning with slight change in wording, order of instruction. For each model we prompt all 9 prompts to calculate the sensitivity and variance of the final result.  If the current task requires reasoning, we include the zero shot chain-of-thought prompting : "Think step-by-step" in task description and ensures the LLM response to generate reasoning before giving the final answer. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
F Prompt F.1 Prompt Format
2408.02442
 The 3 task description variations for GSM8K are shown in as below: Task description variation1: You are a math tutor who helps students of all levels understand and solve mathematical problems. Read the last question carefully and think step by step before answering, the final answer must be only a number.  Task description variation2: Read the last question carefully and think step by step before answering, the final answer must be only a number. You are a math tutor who helps students of all levels understand and solve mathematical problems.  Task description variation3: Mathematical problem-solving task: - Given: A mathematical question or problem - Required: A numerical answer only - Role: You are a math tutor assisting students of all levels - Process: Think step by step to solve the problem Note: Read the question carefully before beginning your analysis.  Format descriptions are shown in Section F.10. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
F.2 Prompt Variation For Gsm8K
2408.02442
 The 3 task description variations for Last Letter Concatenation are shown in as below:  | Task description variation1: You are given a string of words and you need to take the last letter of each words and concate them. Read the last question carefully and think step by step before answering. Task description variation2: Read carefully for each of the last question and think step by step before answering. You are given a string of words and you need to take the last letter of each words and concatenate them. Task description variation3: String manipulation task: - Given: A sequence of words - Required: A new string made from the last letter of each word - Process: Think step by step to solve this challenge Note: Ensure you've read the question thoroughly before beginning.   | |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|  | Task description variation1: In this task, you are tasked to answer the following commonsense knowledge task. Read carefully for each of the last question and think step by step before answering. Make sure the answer only contain one of these four choice : A, B, C, D, E, F, G Task description variation2: Read carefully for each of the last question and think step by step before answering. Make sure the answer only contain one of these four choice : A, B, C, D, E, F, G In this task, you are tasked to answer the following commonsense knowledge task. Task description variation3: Context understanding assessment: - Given: A story related to many person in the same place - Required: Determine if the person who is in the end of the story - Process: Think step by step to analyze the context - Output: Answer the correct answer and only contain one of these seven choice : A, B, C, D, E, F, G   | |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
F.3 Prompt Variation For Last Letter Concatenation
2408.02442
 | answer, your answer should be the one which match any of these valid diagnoses: - Possible NSTEMI / STEMI - Spontaneous rib fracture - Pulmonary embolism - Pulmonary neoplasm ... - Scombroid food poisoning RESPONSE: Task description variation2: Act as a medical doctor and diagnose the patient based on the given patient profile All possible valid diagnoses for you to choose from are as follows: - Possible NSTEMI / STEMI - Spontaneous rib fracture - Pulmonary embolism - Pulmonary neoplasm ... - Scombroid food poisoning Task description variation3: Medical diagnosis task: - Given: A patient profile - Required: Diagnose the patient based on the provided information - Process: Think step by step to analyze the patient's symptoms and history - Output: Select one diagnosis from the provided list of valid options Note: Carefully review the patient profile and the list of possible diagnoses before making your determination. Do not answer "Insufficient information" - you must choose from the given options. Valid diagnoses (select one): - Possible NSTEMI / STEMI - Spontaneous rib fracture - Pulmonary embolism - Pulmonary neoplasm ... - Scombroid food poisoning   | |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|  F.7 Prompt variation for Natural Language - Task 280  | You are given a sentence and your task is to determine whether a sentence relating to sports is plausible or implausible Read carefully for each of the last question and think step by step before answering. Answer yes if its plausible, no if implausible Task description variation2: You are given a sentence and your task is to determine whether a sentence relating to sports is plausible or implausible. Read carefully for each of the last question and think step by step before answering. Answer yes if its plausible, no if implausible Task description variation3: Sentence plausibility assessment: - Given: A sentence related to sports - Required: Determine if the sentence is plausible or implausible - Process: Think step by step to analyze the sentence - Output: Answer "yes" if plausible, "no" if implausible   | |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
F.4 Prompt Variation For Shuffle Object Tracking F.5 Prompt Variation For Ddxplus
2408.02442
 The 3 task description variations for Sports Understanding are shown in as below: The 3 task description variations for Natural Language - Task 280 are shown in as below: Task description variation1: In this task, you are given a short passage that conveys stereotype or anti-stereotype about a specific target. A stereotype is an over-generalized belief about a particular group of people. An anti-stereotype is an idea that goes against a common stereotype. You are expected to classify the passage into four types of stereotype or anti-stereotype: gender, profession, race, and religion.  Task description variation2: You are expected to classify the passage into four types of stereotype or anti-stereotype: gender, profession, race, and religion.  In this task, you are given a short passage that conveys stereotype or anti-stereotype about a specific target. A stereotype is an over-generalized belief about a particular group of people. An anti-stereotype is an idea that goes against a common stereotype.  Task description variation3: Sentence stereotype assessment: - Given: A passage related to stereotype or anti-stereotype - Required: Determine if the paragraph is one of these four category : gender, profession, race, and religion - Output: Answer only one of the four category 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
F.6 Prompt Variation For Sports Understanding
2408.02442
 Task description variation1: Act as a finance expert and assign the content based to the valid category All possible valid category for you to choose from are as follows (one category per line, in the format of <category>): - Finance - Technology - Tax and Accounting - Business and Management - Government and Controls - Industry Your answer MUST based on the above options, do not answer Insufficient information Task description variation2: Act as a finance expert and assign the content based to the valid category Your answer MUST based on the above options, do not answer Insufficient information All possible valid category for you to choose from are as follows (one category per line, in the format of <category>): - Finance - Technology - Tax and Accounting - Business and Management - Government and Controls - Industry Task description variation3: Act as a finance expert and assign the content based to the valid category All possible valid category for you to choose from are as follows (one category per line, in the format of <category>): Finance Technology Tax and Accounting Business and Management Government and Controls Industry Your answer MUST based on the above options, do not answer Insufficient information The 3 task description variations for MultiFin are shown in as below: 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
F.8 Prompt Variation For Multifin
2408.02442
 For tasks which doesn't need chain of though reasoning, we use these three prompt variations. In our experiments, we use 3 format variations in MultiFin. 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
F.9 Direct Answer Format Prompts
2408.02442
 Natural language: Derive the most likely category to answer key.  Provide your output in the following valid text format: Answer: ...  JSON: Derive the most likely category to answer key.  Provide your output in the following valid JSON format: "'json { "answer": "..." } "' YAML: Derive the most likely category to answer key.  Provide your output in the following valid YAML format: "'yaml answer: ...  "' XML: Derive the most likely category to answer block Provide your output in the following valid YAML format: "'xml <root> <answer>...</answer> </root> "'  DA prompt description variation 2:  Natural language:  Provide your output in the following text  format:  Step by step reasoning: ...  Answer: The final answer is ...  JSON:  Provide your output in the following valid  JSON format:  "'json  {  "step_by_step_reasoning": ..., "answer": ...  }  "'  YAML:  Provide your output in the following valid  YAML format:  "'yaml  step_by_step_reasoning: | ... answer: ...  "'  XML:  Provide your output in the following valid  XML format:  "'xml  <root> <step_by_step_reasoning>...  </step_by_step_reasoning> <answer>...</answer> </root>  "'  | DA prompt description variation 3: Natural language: Provide your output in the following text format: Answer: <think step by step>. The final answer is <answer> JSON: Provide your output in the following valid JSON format: "'json { "reason": "<think step by step>", "answer": <answer> } "' YAML: Provide your output in the following valid YAML format: "'yaml reasoning: | <think step by step>, answer: <answer> "' XML: Provide your output in the following valid XML format: "'xml <root> <reason>[think step by step]</reason> <answer>[answer]</answer> </root> "'   | CoT prompt description variation 1: Natural language: Provide your output in the following text format: Answer: <reasoning first>. The final answer is <answer> JSON: Provide your output in the following valid JSON format: "'json { "reason": ..., "answer": ... } "' YAML: Provide your output in the following valid YAML format: "'yaml reasoning: | ... answer: ... "' XML: Provide your output in the following valid XML format: "'xml <root> <reason>...</reason> <answer>...</answer> </root> "'   | |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
Da Prompt Description Variation 1:
2408.02442
 For tasks which include chain of thought reasoning, we use these three prompt variations. In our experiments, we use 3 format variations in GSM8K, Last Letter, Shuffle Object Tracking, DDXPlus, Sports, NL-Task 280.  CoT prompt description variation 2:  Natural language:  Provide your output in the following text  format:  Step by step reasoning: ...  Answer: The final answer is ...  JSON:  Provide your output in the following valid  JSON format:  "'json  {  "step_by_step_reasoning": ..., "answer": ...  }  "'  YAML:  Provide your output in the following valid  YAML format:  "'yaml  step_by_step_reasoning: | ... answer: ...  "'  XML:  Provide your output in the following valid  XML format:  "'xml  <root> <step_by_step_reasoning>...  </step_by_step_reasoning> <answer>...</answer> </root>  "' 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
F.10 Chain Of Thought Format Prompts
2408.02442
 | CoT prompt description variation 3: Natural language: Provide your output in the following text format: Answer: <think step by step>. The final answer is <answer> JSON: Provide your output in the following valid JSON format: "'json { "reason": "<think step by step>", "answer": <answer> } "' YAML: Provide your output in the following valid YAML format: "'yaml reasoning: | <think step by step>, answer: <answer> "' XML: Provide your output in the following valid XML format: "'xml <root> <reason>[think step by step]</reason> <answer>[answer]</answer> </root> "'   | |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
G Averaged Numbers For All Datasets
2408.02442
 Table ?? shows all the number with standard deviation on all 4 format (NL, JSON, XML, YAML) .  The JSON-mode scores for GPT 3.5 turbo, Gemini 1.5 Flash and LLaMA 3 8B  | Last Letter           | GSM8K        | ShuffleObj   |              | |-----------------------|--------------|--------------|--------------| | Gemini-1.5-Flash Text | 65.45 (3.1)   | 89.33 (0.8)   | 58.21 (13.0)  | | JSON                  | 77.02 (7.3)   | 89.21 (1.5)   | 65.07 (5.3)   | | XML                   | 74.17 (10.4)  | 88.20 (2.2)   | 50.39 (10.5)  | | YAML                  | 71.43 (20.3)  | 87.42 (3.7)   | 34.27 (17.1)  | | GPT-3.5 Turbo Text    | 56.74 (7.1)   | 76.60 (2.8)   | 20.37 (3.6)   | | JSON                  | 25.20 (29.1)  | 49.25 (12.0)  | 20.93 (1.1)   | | XML                   | 22.30 (27.8)  | 45.06 (19.9)  | 15.35 (1.8)   | | YAML                  | 66.87 (22.0)  | 73.85 (5.6)   | 20.79 (1.3)   | | Claude 3 Haiku Text   | 57.67 (21.1)  | 86.51 (0.8)   | 36.62 (8.2)   | | JSON                  | 56.74 (16.7)  | 23.44 (22.8)  | 49.33 (4.8)   | | XML                   | 33.80 (31.5)  | 79.76 (7.0)   | 48.10 (5.2)   | | YAML                  | 31.60 (32.4)  | 80.63 (2.8)   | 18.10 (14.7)  | | LLaMA 3 8B Text       | 70.07 (5.3)   | 74.73 (0.6)   | 27.01 (5.5)   | | JSON                  | 28.00 (12.2)  | 48.90 (6.7)   | 15.72 (11.0)  | | XML                   | 15.93 (4.8)   | 56.74 (8.3)   | 11.14 (3.6)   | | YAML                  | 16.10 (10.4)  | 46.08 (16.8)  | 9.57 (3.6)    | | Gemma2 9B IT Text     | 56.81 (9.8)   | 86.50 (0.6)   | 49.39 (5.8)   | | JSON                  | 38.96 (6.8)   | 84.17 (3.7)   | 50.49 (8.9)   | | XML                   | 43.70 (3.8)   | 85.56 (0.6)   | 38.47 (5.0)   | | YAML                  | 23.41 (15.7)  | 79.52 (4.1)   | 22.96 (16.4)  |  Table 6: Zero shot prompting results for gemini-1.5-flash, gpt-3.5-turbo, claude-3-haiku, llama-3-8B, and gemma29B-IT averaged on 3 reasoning tasks with standard deviation in reasoning related task.  Table 7: Zero shot prompting results for gemini-1.5-flash, gpt-3.5-turbo, claude-3-haiku, llama-3-8B, and gemma29B-IT averaged on 4 classification tasks with standard deviation in classification related task  | DDXPlus               | Sports       | Task280      | MultiFin     |             | |-----------------------|--------------|--------------|--------------|-------------| | Gemini-1.5-Flash Text | 41.59 (6.6)   | 79.91 (3.2)   | 68.60 (2.5)   | 63.45 (0.3)  | | JSON                  | 60.36 (0.8)   | 78.85 (1.3)   | 70.27 (5.4)   | 65.24 (1.1)  | | XML                   | 59.44 (1.4)   | 80.15 (0.7)   | 69.96 (4.9)   | 64.54 (1.6)  | | YAML                  | 60.41 (1.6)   | 80.10 (1.2)   | 65.31 (12.7)  | 64.14 (0.4)  | | GPT-3.5 Turbo Text    | 44.07 (3.2)   | 67.15 (26.8)  | 72.65 (6.3)   | 63.03 (0.5)  | | JSON                  | 55.51 (0.4)   | 79.99 (3.3)   | 70.56 (11.2)  | 64.03 (0.9)  | | XML                   | 52.98 (1.4)   | 80.71 (1.1)   | 66.22 (16.2)  | 62.24 (1.1)  | | YAML                  | 55.01 (0.8)   | 80.93 (2.3)   | 72.10 (8.0)   | 65.38 (0.9)  | | Claude 3 Haiku Text   | 33.78 (13.5)  | 77.81 (5.8)   | 61.09 (11.0)  | 61.95 (1.9)  | | JSON                  | 52.04 (1.1)   | 78.74 (2.8)   | 49.46 (27.2)  | 63.68 (1.3)  | | XML                   | 50.75 (0.8)   | 77.78 (3.8)   | 45.03 (25.0)  | 62.39 (1.1)  | | YAML                  | 6.91 (5.3)    | 76.37 (8.3)   | 44.51 (24.2)  | 61.75 (1.7)  | | LLaMA 3 8B Text       | 12.04 (15.2)  | 69.49 (12.7)  | 65.28 (3.4)   | 60.26 (1.4)  | | JSON                  | 23.37 (0.7)   | 73.38 (3.5)   | 39.46 (22.4)  | 57.74 (2.0)  | | XML                   | 11.35 (1.9)   | 69.20 (5.5)   | 35.36 (22.5)  | 58.77 (3.2)  | | YAML                  | 13.08 (4.1)   | 68.25 (5.7)   | 45.42 (24.4)  | 49.74 (4.2)  | | Gemma2 9B IT Text     | 22.86 (5.8)   | 76.08 (2.3)   | 69.84 (7.7)   | 69.69 (0.4)  | | JSON                  | 52.97 (0.2)   | 72.72 (1.6)   | 65.57 (11.7)  | 70.18 (0.7)  | | XML                   | 52.92 (2.8)   | 73.29 (2.4)   | 68.07 (11.7)  | 67.98 (0.7)  | | YAML                  | 44.94 (2.2)   | 72.97 (1.7)   | 60.45 (11.0)  | 69.81 (0.7)  |  | Dataset    | GPT3.5T     | Flash       | LLaMA3      | |------------|-------------|-------------|-------------| | LastLetter | 1.78 (0.3)   | 0.67 (0.5)   | 7.56 (2.7)   | | GSM8K      | 29.87 (0.8)  | 47.78 (3.1)  | 65.38 (1.3)  | | MultiFin   | 66.00 (1.3)  | 66.79 (0.4)  | 54.82 (1.5)  | | Sports     | 76.82 (0.9)  | 77.79 (0.4)  | 72.08 (2.6)  | | Task 280   | 78.07 (2.3)  | 67.19 (4.1)  | 74.57 (2.0)  | | DDXPlus    | 51.87 (2.8)  | 84.92 (2.1)  | 22.59 (0.1)  |  Table 8: Applying JSON-mode to all 6 datasets and found 18
body
Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Performance Of Large Language Models Zhi Rui Tam1*, Cheng-Kuang Wu1*, Yi-Lin Tsai1**, Chieh-Yen Lin**1,
G.1 Zero Shot Prompting Comparing Text, Json, Xml, Yaml