<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">

<head>  <title>[2408.04682] ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="apple-touch-icon" sizes="180x180" href="/static/browse/0.3.4/images/icons/apple-touch-icon.png">
  <link rel="icon" type="image/png" sizes="32x32" href="/static/browse/0.3.4/images/icons/favicon-32x32.png">
  <link rel="icon" type="image/png" sizes="16x16" href="/static/browse/0.3.4/images/icons/favicon-16x16.png">
  <link rel="manifest" href="/static/browse/0.3.4/images/icons/site.webmanifest">
  <link rel="mask-icon" href="/static/browse/0.3.4/images/icons/safari-pinned-tab.svg" color="#5bbad5">
  <meta name="msapplication-TileColor" content="#da532c">
  <meta name="theme-color" content="#ffffff">
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/arXiv.css?v=20240822" />
  <link rel="stylesheet" type="text/css" media="print" href="/static/browse/0.3.4/css/arXiv-print.css?v=20200611" />
  <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/browse_search.css" />
  <script language="javascript" src="/static/browse/0.3.4/js/accordion.js" /></script>
  
    <link rel="stylesheet" type="text/css" media="screen" href="/static/browse/0.3.4/css/slider.css?v=1.14" />
    <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
    <script type="text/javascript" src="/static/browse/0.3.4/js/donate.js?v=1.11"></script><link rel="canonical" href="/abs/2408.04682"/><meta property="og:type" content="website" />
<meta property="og:site_name" content="arXiv.org" />
<meta property="og:title" content="ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities" />
<meta property="og:url" content="https://arxiv.org/abs/2408.04682v1" />
<meta property="og:image" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:secure_url" content="/static/browse/0.3.4/images/arxiv-logo-fb.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="700" />
<meta property="og:image:alt" content="arXiv logo"/>
<meta property="og:description" content="Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at https://github.com/apple/ToolSandbox"/>
<meta name="twitter:site" content="@arxiv"/>
<meta name="twitter:card" content="summary"/>
<meta name="twitter:title" content="ToolSandbox: A Stateful, Conversational, Interactive Evaluation..."/>
<meta name="twitter:description" content="Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use..."/>
<meta name="twitter:image" content="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png"/>
<meta name="twitter:image:alt" content="arXiv logo"/>
  <link rel="stylesheet" media="screen" type="text/css" href="/static/browse/0.3.4/css/tooltip.css"/><link rel="stylesheet" media="screen" type="text/css" href="https://static.arxiv.org/js/bibex-dev/bibex.css?20200709"/>  <script src="/static/browse/0.3.4/js/mathjaxToggle.min.js" type="text/javascript"></script>  <script src="//code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js" type="text/javascript"></script>
  <script src="//cdn.jsdelivr.net/npm/dompurify@2.3.5/dist/purify.min.js"></script>
  <script src="/static/browse/0.3.4/js/toggle-labs.js?20230622" type="text/javascript"></script>
  <script src="/static/browse/0.3.4/js/cite.js" type="text/javascript"></script><meta name="citation_title" content="ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities" /><meta name="citation_author" content="Lu, Jiarui" /><meta name="citation_author" content="Holleis, Thomas" /><meta name="citation_author" content="Zhang, Yizhe" /><meta name="citation_author" content="Aumayer, Bernhard" /><meta name="citation_author" content="Nan, Feng" /><meta name="citation_author" content="Bai, Felix" /><meta name="citation_author" content="Ma, Shuang" /><meta name="citation_author" content="Ma, Shen" /><meta name="citation_author" content="Li, Mengyu" /><meta name="citation_author" content="Yin, Guoli" /><meta name="citation_author" content="Wang, Zirui" /><meta name="citation_author" content="Pang, Ruoming" /><meta name="citation_date" content="2024/08/08" /><meta name="citation_online_date" content="2024/08/08" /><meta name="citation_pdf_url" content="http://arxiv.org/pdf/2408.04682" /><meta name="citation_arxiv_id" content="2408.04682" /><meta name="citation_abstract" content="Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at https://github.com/apple/ToolSandbox" />
</head>

<body  class="with-cu-identity">
  
  <aside class="slider-wrapper bps-banner forum grey">
    <a class="close-slider bps-banner" href="#"><img src="/static/browse/0.3.4/images/icons/close-slider.png" alt="close this message"></a>
  	<div class="columns">
      <img role="presentation" class="bps-banner-image" src="/static/browse/0.3.4/images/events/arxiv-forum-logotype-title.png" alt="arXiv Accessibility Forum 2024">
      <div class="copy-donation bps-banner">
        <h2 id="forum-session-title"></h2>
        <p id="forum-session-link-small"></p>
      </div>
      <div class="amount-donation bps-banner">
          <div id="forum-session-link-box" class="donate-cta"></div>
      </div>
    </div>
  </aside>
  <script src="/static/browse/0.3.4/js/rotate_forum_info.js" type="text/javascript"></script>
  <div class="flex-wrap-footer">
    <header>
      <a href="#content" class="is-sr-only">Skip to main content</a>
      <!-- start desktop header -->
      <div class="columns is-vcentered is-hidden-mobile" id="cu-identity">
        <div class="column" id="cu-logo">
          <a href="https://www.cornell.edu/"><img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University" /></a>
        </div><div class="column banner-minimal forum">
            <!-- /from July 7 at 1:00 AM to Sept 2 at 1:00 AM --><p>This week: the arXiv Accessibility Forum</p>
            <a href="https://accessibility2024.arxiv.org/schedule" target="_blank">Forum Schedule</a></div><div class="column" id="support-ack">
          <span id="support-ack-url">We gratefully acknowledge support from the Simons Foundation, <a href="https://info.arxiv.org/about/ourmembers.html">member institutions</a>, and all contributors.</span>
          <a href="https://info.arxiv.org/about/donate.html" class="btn-header-donate">Donate</a>
        </div>
      </div>

      <div id="header" class="is-hidden-mobile">
<a aria-hidden="true" href="{url_path('ignore_me')}"></a>
  <div class="header-breadcrumbs is-hidden-mobile">
    <a href="/"><img src="/static/browse/0.3.4/images/arxiv-logo-one-color-white.svg" alt="arxiv logo" style="height:40px;"/></a> <span>&gt;</span> <a href="/list/cs/recent">cs</a> <span>&gt;</span> arXiv:2408.04682
  </div>
          <div class="search-block level-right">
    <form class="level-item mini-search" method="GET" action="https://arxiv.org/search">
      <div class="field has-addons">
        <div class="control">
          <input class="input is-small" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
          <p class="help"><a href="https://info.arxiv.org/help">Help</a> | <a href="https://arxiv.org/search/advanced">Advanced Search</a></p>
        </div>
        <div class="control">
          <div class="select is-small">
            <select name="searchtype" aria-label="Field to search">
              <option value="all" selected="selected">All fields</option>
              <option value="title">Title</option>
              <option value="author">Author</option>
              <option value="abstract">Abstract</option>
              <option value="comments">Comments</option>
              <option value="journal_ref">Journal reference</option>
              <option value="acm_class">ACM classification</option>
              <option value="msc_class">MSC classification</option>
              <option value="report_num">Report number</option>
              <option value="paper_id">arXiv identifier</option>
              <option value="doi">DOI</option>
              <option value="orcid">ORCID</option>
              <option value="author_id">arXiv author ID</option>
              <option value="help">Help pages</option>
              <option value="full_text">Full text</option>
            </select>
          </div>
        </div>
        <input type="hidden" name="source" value="header">
        <button class="button is-small is-cul-darker">Search</button>
      </div>
    </form>
  </div>
     </div><!-- /end desktop header -->

      <div class="mobile-header">
        <div class="columns is-mobile">
          <div class="column logo-arxiv"><a href="https://arxiv.org/"><img src="/static/browse/0.3.4/images/arxiv-logomark-small-white.svg" alt="arXiv logo" style="height:60px;" /></a></div>
          <div class="column logo-cornell"><a href="https://www.cornell.edu/">
            <picture>
              <source media="(min-width: 501px)"
                srcset="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg  400w"
                sizes="400w" />
              <source srcset="/static/browse/0.3.4/images/icons/cu/cornell_seal_simple_black.svg 2x" />
              <img src="/static/browse/0.3.4/images/icons/cu/cornell-reduced-white-SMALL.svg" alt="Cornell University Logo" />
            </picture>
          </a></div>
          <div class="column nav" id="toggle-container" role="menubar">
            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-white"><title>open search</title><path d="M505 442.7L405.3 343c-4.5-4.5-10.6-7-17-7H372c27.6-35.3 44-79.7 44-128C416 93.1 322.9 0 208 0S0 93.1 0 208s93.1 208 208 208c48.3 0 92.7-16.4 128-44v16.3c0 6.4 2.5 12.5 7 17l99.7 99.7c9.4 9.4 24.6 9.4 33.9 0l28.3-28.3c9.4-9.4 9.4-24.6.1-34zM208 336c-70.7 0-128-57.2-128-128 0-70.7 57.2-128 128-128 70.7 0 128 57.2 128 128 0 70.7-57.2 128-128 128z"/></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <form class="mobile-search-form" method="GET" action="https://arxiv.org/search">
                <div class="field has-addons">
                  <input class="input" type="text" name="query" placeholder="Search..." aria-label="Search term or terms" />
                  <input type="hidden" name="source" value="header">
                  <input type="hidden" name="searchtype" value="all">
                  <button class="button">GO</button>
                </div>
              </form>
            </div>

            <button class="toggle-control"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-white" role="menu"><title>open navigation menu</title><path d="M16 132h416c8.837 0 16-7.163 16-16V76c0-8.837-7.163-16-16-16H16C7.163 60 0 67.163 0 76v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16zm0 160h416c8.837 0 16-7.163 16-16v-40c0-8.837-7.163-16-16-16H16c-8.837 0-16 7.163-16 16v40c0 8.837 7.163 16 16 16z"/ ></svg></button>
            <div class="mobile-toggle-block toggle-target">
              <nav class="mobile-menu" aria-labelledby="mobilemenulabel">
                <h2 id="mobilemenulabel">quick links</h2>
                <ul>
                    <li><a href="https://arxiv.org/login">Login</a></li>
                    <li><a href="https://info.arxiv.org/help">Help Pages</a></li>
                    <li><a href="https://info.arxiv.org/about">About</a></li>
                </ul>
              </nav>
            </div>
          </div>
        </div>
      </div><!-- /end mobile-header -->
    </header>

    <main>
      <div id="content">
<!--
rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/">
    <rdf:Description
        rdf:about="/abs/2408.04682"
        dc:identifier="/abs/2408.04682"
        dc:title="ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities"
        trackback:ping="/trackback/2408.04682" />
    </rdf:RDF>
--><div id="abs-outer">

  <div class="leftcolumn">
    <div class="subheader">
      <h1>Computer Science > Computation and Language</h1>
    </div>

    <div class="header-breadcrumbs-mobile">
      <strong>arXiv:2408.04682</strong> (cs)
    </div>
<link rel="stylesheet" type="text/css" href="/static/base/1.0.1/css/abs.css">
<div id="content-inner">
  <div id="abs">
    <div class="dateline">
  [Submitted on 8 Aug 2024]</div>
    <h1 class="title mathjax"><span class="descriptor">Title:</span>ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities</h1>
    <div class="authors"><span class="descriptor">Authors:</span><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lu,+J">Jiarui Lu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Holleis,+T">Thomas Holleis</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Zhang,+Y">Yizhe Zhang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Aumayer,+B">Bernhard Aumayer</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Nan,+F">Feng Nan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Bai,+F">Felix Bai</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ma,+S">Shuang Ma</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ma,+S">Shen Ma</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Li,+M">Mengyu Li</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Yin,+G">Guoli Yin</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Wang,+Z">Zirui Wang</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Pang,+R">Ruoming Pang</a></div>            <div id="download-button-info" hidden>View a PDF of the paper titled ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities, by Jiarui Lu and 11 other authors</div>
    <a class="mobile-submission-download" href="/pdf/2408.04682">View PDF</a>
    <a class="mobile-submission-download" href="https://arxiv.org/html/2408.04682v1">HTML (experimental)</a>



    <blockquote class="abstract mathjax">
            <span class="descriptor">Abstract:</span>Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights into tool-use LLM capabilities. ToolSandbox evaluation framework is released at <a href="https://github.com/apple/ToolSandbox" rel="external noopener nofollow" class="link-external link-https">this https URL</a>
    </blockquote>

    <!--CONTEXT-->
    <div class="metatable">
      <table summary="Additional metadata"><tr>
          <td class="tablecell label">Subjects:</td>
          <td class="tablecell subjects">
            <span class="primary-subject">Computation and Language (cs.CL)</span>; Artificial Intelligence (cs.AI); Machine Learning (cs.LG)</td>
        </tr><tr>
          <td class="tablecell label">Cite as:</td>
          <td class="tablecell arxivid"><span class="arxivid"><a href="https://arxiv.org/abs/2408.04682">arXiv:2408.04682</a> [cs.CL]</span></td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxividv">(or <span class="arxivid">
              <a href="https://arxiv.org/abs/2408.04682v1">arXiv:2408.04682v1</a> [cs.CL]</span> for this version)
          </td>
        </tr>
        <tr>
          <td class="tablecell label">&nbsp;</td>
          <td class="tablecell arxivdoi">              <a href="https://doi.org/10.48550/arXiv.2408.04682"  id="arxiv-doi-link">https://doi.org/10.48550/arXiv.2408.04682</a><div class="button-and-tooltip">
              <button class="more-info" aria-describedby="more-info-desc-1">
                <svg height="15" role="presentation" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path fill="currentColor" d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z" class=""></path></svg>
                <span class="visually-hidden">Focus to learn more</span>
              </button>
              <!-- tooltip description -->
              <div role="tooltip" id="more-info-desc-1">
                <span class="left-corner"></span>                  arXiv-issued DOI via DataCite</div>
            </div>
          </td>
        </tr></table>
    </div>
  </div>
</div>
    <div class="submission-history">
      <h2>Submission history</h2> From: Jiarui Lu [<a href="/show-email/8e4f5921/2408.04682">view email</a>]      <br/>    <strong>[v1]</strong>
        Thu, 8 Aug 2024 05:45:42 UTC (21,548 KB)<br/>
</div>
  </div>
  <!--end leftcolumn-->
<div class="extra-services">    <div class="full-text">
      <a name="other"></a>
      <span class="descriptor">Full-text links:</span>
      <h2>Access Paper:</h2>
      <ul>
  <div id="download-button-info" hidden>
View a PDF of the paper titled ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities, by Jiarui Lu and 11 other authors</div><li><a href="/pdf/2408.04682" aria-describedby="download-button-info" accesskey="f" class="abs-button download-pdf">View PDF</a></li><li><a href="https://arxiv.org/html/2408.04682v1" class="abs-button" id="latexml-download-link">HTML (experimental)</a></li><li><a href="/src/2408.04682" class="abs-button download-eprint">TeX Source</a></li><li><a href="/format/2408.04682" class="abs-button download-format">Other Formats</a></li></ul>
      <div class="abs-license"><a href="http://creativecommons.org/licenses/by-sa/4.0/" title="Rights to this article" class="has_license">
          <img alt="license icon" role="presentation" src="https://arxiv.org/icons/licenses/by-sa-4.0.png"/>
          <span>view license</span>
        </a></div>
    </div>
    <!--end full-text-->    <div class="browse">
    Current browse context: <div class="current">cs.CL</div>

  <div class="prevnext">
<span class="arrow">
      <a class="abs-button prev-url" href="/prevnext?id=2408.04682&amp;function=prev&amp;context=cs.CL"
         accesskey="p" title="previous in cs.CL (accesskey p)">&lt;&nbsp;prev</a>
    </span>
    <span class="is-hidden-mobile">&nbsp; | &nbsp;</span>    <span class="arrow">
      <a class="abs-button next-url" href="/prevnext?id=2408.04682&amp;function=next&amp;context=cs.CL" accesskey="n"
         title="next in cs.CL (accesskey n)">next&nbsp;&gt;</a>
    </span><br/>
  </div><div class="list">
    <a class="abs-button abs-button-grey abs-button-small context-new" href="/list/cs.CL/new">new</a>
    <span class="is-hidden-mobile"> | </span>
    <a class="abs-button abs-button-grey abs-button-small context-recent" href="/list/cs.CL/recent">recent</a>
    <span class="is-hidden-mobile"> | </span><a class="abs-button abs-button-grey abs-button-small context-id" href="/list/cs.CL/2024-08">2024-08</a>
  </div><div class="abs-switch-cat">
    Change to browse by:
    <div class="switch context-change">
        <a href="/abs/2408.04682?context=cs">cs</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.04682?context=cs.AI">cs.AI</a><br class="is-hidden-mobile">
        <a class="subclass" href="/abs/2408.04682?context=cs.LG">cs.LG</a><br class="is-hidden-mobile">
    </div>
  </div>

    </div>

    <div class="extra-ref-cite">
      <h3>References &amp; Citations</h3>
      <ul>
        <li><a  class="abs-button abs-button-small cite-ads" href="https://ui.adsabs.harvard.edu/abs/arXiv:2408.04682">NASA ADS</a></li><li><a  class="abs-button abs-button-small cite-google-scholar" href="https://scholar.google.com/scholar_lookup?arxiv_id=2408.04682" target="_blank" rel="noopener">Google Scholar</a></li>
        <li><a  class="abs-button abs-button-small cite-semantic-scholar" href="https://api.semanticscholar.org/arXiv:2408.04682" target="_blank" rel="noopener">Semantic Scholar</a></li>
      </ul>
      <div style="clear:both;"></div>
    </div>

<div class='extra-ref-cite'>
    <a id='bib-cite-css' hidden='true' href='/static/browse/0.3.4/css/cite.css'>a</a>

    <span id='bib-cite-trigger' class="bib-cite-button abs-button">export BibTeX citation</span>
    <span id='bib-cite-loading' hidden='true'>Loading...</span>
</div>

<div id='bib-cite-modal' class='bib-modal' hidden='true'>
    <div class='bib-modal-content'>
        <div class='bib-modal-title'>
            <h2>BibTeX formatted citation</h2>
            <span class='bib-modal-close' >&times;</span>
        </div>
        <div>
            <textarea id='bib-cite-target' class="bib-citation-content" aria-label="loading the citation">loading...</textarea>
        </div>
        <div>
            <span>Data provided by: </span>
            <a id='bib-cite-source-api'></a>
        </div>
    </div>
</div><div class="bookmarks">
  <div><h3>Bookmark</h3></div><a class="abs-button abs-button-grey abs-button-small" href="http://www.bibsonomy.org/BibtexHandler?requTask=upload&amp;url=https://arxiv.org/abs/2408.04682&amp;description=ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities"
     title="Bookmark on BibSonomy">
    <img src="/static/browse/0.3.4/images/icons/social/bibsonomy.png"
         alt="BibSonomy logo"/>
  </a>
  <a class="abs-button abs-button-grey abs-button-small" href="https://reddit.com/submit?url=https://arxiv.org/abs/2408.04682&amp;title=ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities"
     title="Bookmark on Reddit">
    <img src="/static/browse/0.3.4/images/icons/social/reddit.png"
         alt="Reddit logo"/>
  </a>
</div>  </div>
  <!--end extra-services-->
<!-- LABS AREA -->
<div id="labstabs">
  <div class="labstabs"><input type="radio" name="tabs" id="tabone"checked="checked">
    <label for="tabone">Bibliographic Tools</label>
    <div class="tab labs-display-bib">
      <h1>Bibliographic and Citation Tools</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input id="bibex-toggle" type="checkbox" class="lab-toggle">
              <span class="slider"></span>
              <span class="is-sr-only">Bibliographic Explorer Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-bibex">Bibliographic Explorer</span> <em>(<a href="https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer">What is the Explorer?</a>)</em>
          </div>
        </div>
<div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="litmaps-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/litmaps.js?20210617"
                aria-labelledby="label-for-litmaps">
              <span class="slider"></span>
              <span class="is-sr-only">Litmaps Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-litmaps">Litmaps</span> <em>(<a href="https://www.litmaps.co/" target="_blank">What is Litmaps?</a>)</em>
          </div>
        </div>
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="scite-toggle"
                type="checkbox"
                class="lab-toggle"
                data-script-url="/static/browse/0.3.4/js/scite.js?20210617"
                aria-labelledby="label-for-scite">
              <span class="slider"></span>
              <span class="is-sr-only">scite.ai Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-scite">scite Smart Citations</span> <em>(<a href="https://www.scite.ai/" target="_blank">What are Smart Citations?</a>)</em>
          </div>
        </div>
      </div>
        <div class="labs-content-placeholder labs-display" style="display: none;"></div>
        <div style="min-height: 15px" id="litmaps-open-in"></div>
        <div style="min-height: 15px" id="scite-open-in"></div>
    </div>


    <input type="radio" name="tabs" id="tabtwo">
    <label for="tabtwo">Code, Data, Media</label>
    <div class="tab">
      <h1>Code, Data and Media Associated with this Article</h1>
      <div class="toggle">
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input        
                id="catalyzex-toggle"
                data-script-url="/static/browse/0.3.4/js/catalyzex.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-cx">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-cx">CatalyzeX Code Finder for Papers</span> <em>(<a href="https://www.catalyzex.com" target="_blank">What is CatalyzeX?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="dagshub-toggle"
                data-script-url="/static/browse/0.3.4/js/dagshub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-dagshub">
              <span class="slider"></span>
              <span class="is-sr-only">DagsHub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-dagshub">DagsHub</span> <em>(<a href="https://dagshub.com/" target="_blank">What is DagsHub?</a>)</em>
          </div>
        </div>
  
        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="gotitpub-toggle"
                data-script-url="/static/browse/0.3.4/js/gotitpub.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-gotitpub">
              <span class="slider"></span>
              <span class="is-sr-only">GotitPub Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-gotitpub">Gotit.pub</span> <em>(<a href="http://gotit.pub/faq" target="_blank">What is GotitPub?</a>)</em>
          </div>
        </div>

        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="paperwithcode-toggle"
                data-script-url="/static/browse/0.3.4/js/paperswithcode.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-pwc">
              <span class="slider"></span>
              <span class="is-sr-only">Links to Code Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-pwc">Papers with Code</span> <em>(<a href="https://paperswithcode.com/" target="_blank">What is Papers with Code?</a>)</em>
          </div>
        </div>


        <div class="columns is-mobile lab-row">
          <div class="column lab-switch">
            <label class="switch">
              <input
                id="sciencecast-toggle"
                data-script-url="/static/browse/0.3.4/js/sciencecast.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-sciencecast">
              <span class="slider"></span>
              <span class="is-sr-only">ScienceCast Toggle</span>
            </label>
          </div>
          <div class="column lab-name">
            <span id="label-for-sciencecast">ScienceCast</span> <em>(<a href="https://sciencecast.org/welcome" target="_blank">What is ScienceCast?</a>)</em>
          </div>
        </div>
      </div>

      <div id="catalyzex-output" style="display:none"></div>
      <div id="dagshub-output" style="display:none"></div>
      <div id="gotitpub-output" style="display:none"></div>
      <div id="pwc-output" style="display:none"></div>
      <div id="pwc-data-output" style="display:none"></div>
      <div id="sciencecast-output" style="display:none"></div>
    </div>


      <input type="radio" name="tabs" id="labstabs-demos-input">
      <label for="labstabs-demos-input" id="labstabs-demos-label">Demos</label>
      <div class="tab">
        <h1>Demos</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="replicate-toggle"
                  data-script-url="/static/browse/0.3.4/js/replicate.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-replicate">
                <span class="slider"></span>
                <span class="is-sr-only">Replicate Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-replicate">Replicate</span> <em>(<a href="https://replicate.com/docs/arxiv/about" target="_blank">What is Replicate?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="spaces-toggle"
                  data-script-url="/static/browse/0.3.4/js/spaces.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-spaces">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-spaces">Hugging Face Spaces</span> <em>(<a href="https://huggingface.co/docs/hub/spaces" target="_blank">What is Spaces?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input
                  id="txyz-toggle"
                  data-script-url="/static/browse/0.3.4/js/txyz.js"
                  type="checkbox" class="lab-toggle" aria-labelledby="label-for-txyz">
                <span class="slider"></span>
                <span class="is-sr-only">Spaces Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-txyz">TXYZ.AI</span> <em>(<a href="https://txyz.ai" target="_blank">What is TXYZ.AI?</a>)</em>
            </div>
          </div>
        </div>
        <div id="replicate-output"></div>
        <div id="spaces-output"></div>
        <div id="txyz-output"></div>
      </div>
      <input type="radio" name="tabs" id="tabfour">
      <label for="tabfour">Related Papers</label>
      <div class="tab">
        <h1>Recommenders and Search Tools</h1>
        <div class="toggle">
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="influenceflower-toggle"
                data-script-url="/static/browse/0.3.4/js/influenceflower.js"
                type="checkbox" class="lab-toggle" aria-labelledby="label-for-influenceflower">
                <span class="slider"></span>
                <span class="is-sr-only">Link to Influence Flower</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-influenceflower">Influence Flower</span> <em>(<a href="https://influencemap.cmlab.dev/" target="_blank">What are Influence Flowers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="connectedpapers-toggle" type="checkbox" class="lab-toggle"
                       data-script-url="/static/browse/0.3.4/js/connectedpapers.js"
                       aria-labelledby="label-for-connected-papers" >
                <span class="slider"></span>
                <span class="is-sr-only">Connected Papers Toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-connected-papers">Connected Papers</span> <em>(<a href="https://www.connectedpapers.com/about" target="_blank">What is Connected Papers?</a>)</em>
            </div>
          </div>
          <div class="columns is-mobile lab-row">
            <div class="column lab-switch">
              <label class="switch">
                <input id="core-recommender-toggle" type="checkbox" class="lab-toggle" aria-labelledby="label-for-core">
                <span class="slider"></span>
                <span class="is-sr-only">Core recommender toggle</span>
              </label>
            </div>
            <div class="column lab-name">
              <span id="label-for-core">CORE Recommender</span> <em>(<a href="https://core.ac.uk/services/recommender">What is CORE?</a>)</em>
            </div>
          </div></div>
        <div id="influenceflower-output"></div>
        <div id="influenceflower-output-graph" style="display:none">
          <ul class="flower-tabs">
            <li class="active"><a class="btn tab-btn" onclick="openTab(event, 'tab-author')">Author</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-venue')">Venue</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-inst')">Institution</a></li>
            <li><a class="btn tab-btn" onclick="openTab(event, 'tab-topic')">Topic</a></li>
          </ul>
          <div class="flower-tab-content">
            <div class="tab-flower active" id="tab-author"><svg id="flower-graph-author"></svg></div>
            <div class="tab-flower" id="tab-venue"><svg id="flower-graph-venue"></svg></div>
            <div class="tab-flower" id="tab-inst"><svg id="flower-graph-inst"></svg></div>
            <div class="tab-flower" id="tab-topic"><svg id="flower-graph-topic"></svg></div>
          </div>
        </div>
        <div style="min-height: 15px" id="connectedpapers-output"></div>
        <div id="coreRecommenderOutput"></div>
        <div id="iarxivOutput"></div>
      </div>

      <input type="radio" name="tabs" id="tabfive">
      <label for="tabfive">
        About arXivLabs
      </label>
      <div class="tab">
        <div class="columns">
          <div class="column">
            <h1>arXivLabs: experimental projects with community collaborators</h1>
            <p>arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.</p>
            <p>Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.</p>
            <p>Have an idea for a project that will add value for arXiv's community? <a href="https://info.arxiv.org/labs/index.html"><strong>Learn more about arXivLabs</strong></a>.</p>
          </div>
          <div class="column is-narrow is-full-mobile">
            <p class="icon-labs"><svg xmlns="http://www.w3.org/2000/svg" role="presentation" viewBox="0 0 635.572 811"><path d="M175.6 676v27h-27v-27zm-54 27v27h27v-27zm-27 27v27h27v-27zm396-54v27h-27v-27zm0 27v27h27v-27zm27 27v27h27v-27zm-27-414h27v27h-27zm27 0h27v-27h-27zm27-27h27v-27h-27zm-396 45h-27v-27h27zm-27-54h-27v27h27zm-27-27h-27v27h27z"/><path d="M94.6 730v27h-27v-27zm477 0v27h-27v-27zm-27-495h27v27h-27zm-450 18h-27v-27h27zm477 9h27v27h-27zm-54 495h27v27h-27zm-423 0h27v27h-27zm-54-504h27v27h-27z" fill="#666"/><path d="M67.6 730v27h-27v-27zm54 54v27h-27v-27zm0-108v27h27v-27zm-27 27v27h27v-27zm-81 0v27h27v-27zm585 27v27h-27v-27zm-108-54v27h27v-27zm27 27v27h27v-27zm81 0v27h27v-27zm-54-495h27v27h-27zm-54 108h27v-27h-27zm27-27h27v-27h-27zm0-81h27v-27h-27zm-423 18h-27v-27h27zm54 54h-27v27h27zm-27-27h-27v27h27zm0-81h-27v27h27zm423 612v27h-27v-27zm81-522v27h-27v-27zm-585-9v27h-27v-27z" fill="#999"/><path d="M94.6 784v27h-27v-27zm-27-27v27h27v-27zm-27-54v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm0-27v27h27v-27zm27 0v27h27v-27zm-108 81v27h27v-27zm558 54v27h-27v-27zm-27-27v27h27v-27zm27-54v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm0-27v27h27v-27zm-27 0v27h27v-27zm108 81v27h27v-27zm0-495h27v27h-27zm-27 27h27v-27h-27zm-54-27h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm-27 0h27v-27h-27zm0 27h27v-27h-27zm81-108h27v-27h-27zm-504 45h-27v-27h27zm27-27h-27v27h27zm54-27h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm27 0h-27v27h27zm0 27h-27v27h27zm-81-108h-27v27h27z" fill="#ccc"/><path d="M598.6 665.1H41.5C-76.5 667 176 280.2 176 280.2h53a46.5 46.5 0 0162.8-56.3 29.2 29.2 0 1128.5 35.9h-1a46.5 46.5 0 01-1.5 20.3l142.5-.1s255.3 387 138.3 385.1zM291 181a29.3 29.3 0 10-29.2-29.3A29.3 29.3 0 00291 181zm65.4-66.8a22.4 22.4 0 10-22.5-22.4 22.4 22.4 0 0022.5 22.4z" fill="#fc0"/><path d="M245.5 172V10h153v162s324 495 198 495h-558c-126 0 207-495 207-495zm126 54h56m-13 72h56m-9 72h56m-20 72h56m-22 72h56m-29 72h56m-457-45c20.8 41.7 87.3 81 160.7 81 72.1 0 142.1-38.2 163.4-81" fill="none" stroke="#000" stroke-miterlimit="10" stroke-width="20"/><path d="M273.3 421.7c0 31-9.8 56.3-21.9 56.3s-21.8-25.2-21.8-56.3 9.8-56.3 21.8-56.3 21.9 25.2 21.9 56.3zm114.4-56.3c-12 0-21.8 25.2-21.8 56.3s9.7 56.3 21.8 56.3 21.9-25.2 21.9-56.3-9.8-56.3-21.9-56.3zM150.1 526.6c-18.2 6.7-27.5 22.9-23.2 30.2s14.8-5.5 33-12.2 37.4-4.9 33-12.2-24.5-12.6-42.8-5.8zm296 5.8c-4.2 7.3 14.9 5.5 33.1 12.2s28.7 19.5 33 12.2-5-23.5-23.2-30.2-38.5-1.5-42.8 5.8z"/></svg></p>
          </div>
        </div>
      </div>

    </div>
</div>
<!-- END LABS AREA -->
  <div class="endorsers">
    <a href="/auth/show-endorsers/2408.04682" class="endorser-who">Which authors of this paper are endorsers?</a> |
    <a id="mathjax_toggle" href="javascript:setMathjaxCookie()">Disable MathJax</a> (<a href="https://info.arxiv.org/help/mathjax.html">What is MathJax?</a>)
    <span class="help" style="font-style: normal; float: right; margin-top: 0; margin-right: 1em;"></span>
  </div>
  <script type="text/javascript" language="javascript">mathjaxToggle();</script>
</div>
      </div>
    </main>

    <footer style="clear: both;">
      <div class="columns is-desktop" role="navigation" aria-label="Secondary" style="margin: -0.75em -0.75em 0.75em -0.75em">
        <!-- Macro-Column 1 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/about">About</a></li>
                <li><a href="https://info.arxiv.org/help">Help</a></li>
              </ul>
            </div>
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>contact arXiv</title><desc>Click here to contact arXiv</desc><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>
                  <a href="https://info.arxiv.org/help/contact.html"> Contact</a>
                </li>
                <li>
                  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><title>subscribe to arXiv mailings</title><desc>Click here to subscribe</desc><path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z"/></svg>
                  <a href="https://info.arxiv.org/help/subscribe"> Subscribe</a>
                </li>
              </ul>
            </div>
          </div>
        </div>
        <!-- End Macro-Column 1 -->
        <!-- Macro-Column 2 -->
        <div class="column" style="padding: 0;">
          <div class="columns">
            <div class="column">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/license/index.html">Copyright</a></li>
                <li><a href="https://info.arxiv.org/help/policies/privacy_policy.html">Privacy Policy</a></li>
              </ul>
            </div>
            <div class="column sorry-app-links">
              <ul style="list-style: none; line-height: 2;">
                <li><a href="https://info.arxiv.org/help/web_accessibility.html">Web Accessibility Assistance</a></li>
                <li>
                  <p class="help">
                    <a class="a11y-main-link" href="https://status.arxiv.org" target="_blank">arXiv Operational Status <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 512" class="icon filter-dark_grey" role="presentation"><path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"/></svg></a><br>
                    Get status notifications via
                    <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/email/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" class="icon filter-black" role="presentation"><path d="M502.3 190.8c3.9-3.1 9.7-.2 9.7 4.7V400c0 26.5-21.5 48-48 48H48c-26.5 0-48-21.5-48-48V195.6c0-5 5.7-7.8 9.7-4.7 22.4 17.4 52.1 39.5 154.1 113.6 21.1 15.4 56.7 47.8 92.2 47.6 35.7.3 72-32.8 92.3-47.6 102-74.1 131.6-96.3 154-113.7zM256 320c23.2.4 56.6-29.2 73.4-41.4 132.7-96.3 142.8-104.7 173.4-128.7 5.8-4.5 9.2-11.5 9.2-18.9v-19c0-26.5-21.5-48-48-48H48C21.5 64 0 85.5 0 112v19c0 7.4 3.4 14.3 9.2 18.9 30.6 23.9 40.7 32.4 173.4 128.7 16.8 12.2 50.2 41.8 73.4 41.4z"/></svg>email</a>
                    or <a class="is-link" href="https://subscribe.sorryapp.com/24846f03/slack/new" target="_blank"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512" class="icon filter-black" role="presentation"><path d="M94.12 315.1c0 25.9-21.16 47.06-47.06 47.06S0 341 0 315.1c0-25.9 21.16-47.06 47.06-47.06h47.06v47.06zm23.72 0c0-25.9 21.16-47.06 47.06-47.06s47.06 21.16 47.06 47.06v117.84c0 25.9-21.16 47.06-47.06 47.06s-47.06-21.16-47.06-47.06V315.1zm47.06-188.98c-25.9 0-47.06-21.16-47.06-47.06S139 32 164.9 32s47.06 21.16 47.06 47.06v47.06H164.9zm0 23.72c25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06H47.06C21.16 243.96 0 222.8 0 196.9s21.16-47.06 47.06-47.06H164.9zm188.98 47.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06s-21.16 47.06-47.06 47.06h-47.06V196.9zm-23.72 0c0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06V79.06c0-25.9 21.16-47.06 47.06-47.06 25.9 0 47.06 21.16 47.06 47.06V196.9zM283.1 385.88c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06-25.9 0-47.06-21.16-47.06-47.06v-47.06h47.06zm0-23.72c-25.9 0-47.06-21.16-47.06-47.06 0-25.9 21.16-47.06 47.06-47.06h117.84c25.9 0 47.06 21.16 47.06 47.06 0 25.9-21.16 47.06-47.06 47.06H283.1z"/></svg>slack</a>
                  </p>
                </li>
              </ul>
            </div>
          </div>
        </div> <!-- end MetaColumn 2 -->
        <!-- End Macro-Column 2 -->
      </div>
    </footer>
  </div>

  <script src="/static/base/1.0.1/js/member_acknowledgement.js"></script>

</body>

</html>
2408.04682
 Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, TOOLSANDBOX1includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. We show that open source and proprietary models have a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in TOOLSANDBOX are challenging even the most capable SOTA LLMs, providing brandnew insights into tool-use LLM capabilities. 
abstract
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Abstract
2408.04682
 | TOOLSANDBOX BFCL            | ToolEval   | API-Bank   |    |    | |-----------------------------|------------|------------|----|----| | State Dependency            | "          | %          | %  | %  | | Conversational              | "          | %          | %  | "  | | Interactive                 | "          | %          | "  | %  | | Human Authored Ground Truth | "          | "          | %  | "  |  Table 1: A comparison between TOOLSANDBOX and other tool-use Benchmarks.  Recent advancements in Large Language Models (LLMs) brought forth new opportunities treating LLMs as autonomous agents, capable of observing real-world environments and deciding upcoming actions. Among which, tool-use agents (Schick et al., 2023; Qin et al., 2023a; Patil et al., 2023; Qin et al., 2024) follow human instructions and utilize real-world APIs to complete complex tasks. Contrary to prior approaches like dialog state tracking 1TOOLSANDBOX evaluation framework is released at https://github.com/apple/ToolSandbox (Henderson et al., 2014; Budzianowski et al., 2018; Rastogi et al., 2020), which require the model to explicitly generate dialog states and actions under a predefined ontology, and derive a tool call from those structured outputs, tool-use studies allow the model to directly generate tool calls based on its observations, while keeping dialog and world state tracking implicit.  Despite the paradigm shift towards a more simplified problem formulation, the stateful, **conversational** and **interactive** nature of task oriented dialog remains, and poses a significant challenge for systematic and accurate evaluation of tool-using LLMs. Existing benchmarks like the Berkeley Function Calling Leaderboard (BFCL) (Yan et al., 2024), ToolEval (Qin et al., 2024) and API-Bank (Li et al., 2023) attempted to tackle some of these challenges, but there is yet to be an all encompassing solution.  Stateful Task oriented dialog often involves tools that are strongly coupled with a *World State*, e.g. a database. This can be a tool that can alter the world state, like turning on internet connection. More interestingly, there can be a tool that implicitly depends on a world state, for example, one cannot search for a nearby restaurant when internet connection is off. Sometimes, actions that deal with both of these scenarios need to be taken to complete a task, even if the user is agnostic to the underlying world state and only gives general instructions.  The agent needs to use its own knowledge about the world and environment feedback to come up with a plan to modify the world state and complete the task. An example can be found in Figure 1.  BFCL (Yan et al., 2024) and ToolEval (Qin et al., 2024) both rely on stateless tools interacting with web services (through RESTful APIs). As such, these evaluation benchmarks are designed to assess how agents make trials with a static environment.  API-Bank (Li et al., 2023) does include a set of  ![1_image_0.png](1_image_0.png)  tools to modify world states, but does not study the impact of state dependencies.  Conversational Conversational evaluation is crucial yet challenging when assessing a dialog policy, due to the interdependency between a user and said policy, as well as the ambiguous nature of natural language. To facilitate automated conversational evaluation, a common practice is to implement a simulated user (Zhang et al., 2024; Sekulic et al., 2024). However, BFCL and ToolEval only evaluate self-contained, unambiguous single-turn user queries, which is hardly realistic. API-Bank evaluates on unrolled predefined off-policy dialog trajectories, and thus cannot assess the agent's performance based on its own policy.  Interactive Real world scenarios are full of surprises. The agent could issue an erroneous tool call. Tool execution could raise an unexpected exception. And the user could issue a follow-up correcting a previous statement. An interactive evaluation framework assessing the immediate return of key interactions with user or environment would be necessary to capture the intricate interaction between different roles. Such an interactive evaluation should provide full spectrum and finegrained evaluation of any multi-turn session. In this regard, BFCL and API-Bank both rely on a predefined trajectory, and by extension rely on static turn wise evaluation metrics. Even though ToolEval allows multiple rounds of interaction between the Agent and tools, it relies solely on an LLM evaluator to judge the final pass rate and win rate of trajectories, which raises questions to its reliability and interpretability.  Driven by these motivations, we propose TOOLSANDBOX, a stateful, conversational and interactive tool-use benchmark. To the best of our knowledge, TOOLSANDBOX is the first LLM tool-use benchmark which - Includes implicit state dependencies between stateful tools, allowing the agent to track and  alter the world state based on its world/commonsense knowledge, which is implicit from the user query; - Includes an LLM simulated user, allowing for realistic, on-policy conversational evaluation to measure the agent's ability on implicit dialog state tracking; - Allows for fully interactive, dynamic trajectory collection with a representative set of highly composable tools, and a human authored, milestone / minefield based system for intermediate and final execution evaluation.  A comparison between TOOLSANDBOX and other benchmarks can be found in Table 1. 
introduction
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Introduction
2408.04682
 ![2_Image_0.Png](2_Image_0.Png)  At its core, TOOLSANDBOX is a Python native LLM testing environment, with *Execution Context* as world state abstraction and Python functions as *Tools*, where User, *Agent* and *Execution Environment* communicate with each other through a Message Bus to complete a task, which is evaluated against predefined *Milestones* and *Minefields*. As shown in Figure 2, a typical test case starts with the User speaking to the *Agent*. From then on, the role being addressed gets to speak next, until the end state is reached. Upon receiving a *User* request, an Agent can decide to respond to the *User* asking for more information, or inform the *Execution Environment* to execute a *Tool*, providing intended tool name and arguments. The *Execution Environment* executes the *Tool* in an *code.InteractiveConsole*, (Foundation, 2024), which depending on the *Tool* modifies the world state stored in the *Execution* Context, and responds to the *Agent*. Once the User decides the task has been completed, it informs the *Execution Environment* to execute the end_conversation tool, which puts the system in the end state, ready to be evaluated based on the dialog's similarity to *Milestones* and *Minefields*. The remainder of this section introduces the functionality of each component in more details. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Toolsandbox **Design**
2408.04682
 Tools in TOOLSANDBOX are a set of highly composable, explicitly or implicitly dependent Python functions, creating complex reasoning challenges.  Besides python native tools, a handful of carefully selected RapidAPI tools were also included with a thin layer of python wrapper. Tools manipulate world state through the Execution Context when necessary, and raise informative exceptions when execution conditions were not met. See Appendix A.1 and A.2 for more information. As an example, in Figure 1, when send_message tool is called while cellular service is off, a ConnectionError is raised. This allows the *Agent* to reason over possible exceptions, and deduce the tool needed to resolve the exception.  To support ablation studies on the effect of tool schema representation on agent accuracy, we have implemented multiple tool augmentations, e.g.: - The agent is given distraction tools not needed to complete the task.  - Tool or argument names becomes less informative, e.g. using a tool name of settings_0 instead of set_cellular_service_status to test if the agent relies on them to infer a tool's purpose.  - Information like argument descriptions or type hints are removed.  For more details on the augmentations please refer to Appendix A.2.1. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Tools
2408.04682
 In TOOLSANDBOX there are three roles: User, Agent (Assistant) and *Execution Environment*. The Execution Environment, as a dedicated role, is responsible for executing tool-use requests from the Agent and returning the results. Interaction between the roles is enabled through a message passing system. Each message contains a sender role, recipient role, content as well as to which roles the message is visible to. A simple orchestrator determines message passing order by allowing the most recent recipient to be the next sender. Instead of representing the conversation as a single message thread, we use a collection of messages, i.e. *Message Bus*, stored within the Execution Context. The Message Bus contains a linear history of message transactions between all roles. As is shown in Appendix A.3, each role writes to the same Message Bus. However, when reading from the Message Bus, each role can only access a sub-view of the Message Bus based on which roles are allowed to "see" the individual messages. We will introduce each role in the following paragraphs.  User Role The User role represents a human interacting with an Agent, hoping to complete a task through possibly multiple rounds of conversation.  When the User role decides the task has been completed, or could not be completed, it can terminate the conversation using the end_conversation tool, which is the single tool available to the User.  The User role is implemented with an LLM (GPT4o) and carefully calibrated prompting design to make the user simulator more realistic. As related studies in user simulation (Zhang et al., 2024; Sekulic et al., 2024) suggest, one should include the user's overall goal in the simulator's system prompt. However, we found this is often insufficient for the complex interactions in TOOLSANDBOX, and can lead to two categories of failures. In some cases, it is infeasible for an LLM simulated user to judge task completion, or provide follow-up information with only access to the user goal, and not the expected result, which could lead to hallucination.  Also, with only a single system prompt, the simulated user could be derailed by the tool-use agent, failing to follow instructions. Examples of these failures can be found in Appendix A.4.  In light of this, we propose two additional components in user simulator prompts: Knowledge Boundary, which inform the user simulator what it should and should not know, providing partial access to expected result, combating hallucination.  And *Demonstration*, which provides few shot example dialogs to the user simulator. Prompt examples can be found in Appendix A.4. Note that demonstration is only visible to the user simulator and not the agent. We performed an ablation study for these components in Table 2. With both approaches combined, the LLM simulated user achieves the lowest error rate in all categories. User simulator error rate is also found to be consistent across well performing agents, shown in Table 5.  Agent Role Initially, the Agent role will receive a message from the User in natural language. The Agent could decide to prompt the User again for additional information, or decide to issue a tool call towards the Execution Environment. When issuing a tool call, the Agent selects the name of the tool from a list of available tools and provides necessary  | Hallucination ↓      | IF ↓   |      | |----------------------|--------|------| | User Goal            | 12.4   | 6.20 | | + Knowledge Boundary | 7.75   | 3.88 | | + Demonstration      | 6.97   | 0.77 |  Table 2: Percentage of user simulation failures in each failure category for each user simulator prompting setup.  IF stands for instruction following error. Statistics derived from 1032 manually annotated trajectories using GPT-4o user simulator and GPT-4o agent.  arguments, commonly expressed as JSON objects.  These JSON objects are converted to executable Python code, see Appendix A.5, and sent to the Execution Environment for execution.  Execution Environment Role The execution environment role is responsible for executing tool calls requested by the Agent and User roles in the form of Python snippets, mimicking the behavior of interactive consoles like IPython and Jupyter.  Exceptions raised while executing the code are captured through stderr, enabling the Agent to refine its tool calls through trial and error.  Some LLMs support parallel tool calling, intended to increase efficiency when multiple, independent tools need to be called. However, if an LLM uses parallel tool calls for dependent tools, it should be penalized accordingly. For example, in Figure 1 where the agent has to enable cellular service before sending a text message, parallel tool calls should not be used. Execution Environment handles race conditions in parallel tool calls by following Murphy's Law, ensuring race condition always happens if detected. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Roles And Message Bus
2408.04682
 With an interactive, stateful and conversational environment, evaluation trajectories are highly dynamic. Multiple trajectories can lead to the same outcome. A given task may be completed using different tools, the same tools in a different order, or through trial and error, and the evaluation strategy has to be flexible enough to accommodate for that. To combat this, we developed an evaluation strategy based on *Milestones* and *Minefields*, which defines key events that must or must not happen in a trajectory, allowing us to evaluate any trajectory with rich intermediate and final execution signals, providing deeper understanding of the model performance. An example can be found in Figure 10.  In specific, *Milestones* are the critical steps needed to achieve a goal. An example is shown in 4 Figure 1, where cellular service is turned off and the user asks the agent to send a text message. The milestones, in this example, would be defined as: 1. The cellular status in the settings database must be changed to True.  2. The Agent must issue a tool call using the search_contacts tool and the correct arguments, before or after milestone 1.  3. The Agent must issue a tool call using the send_message tool and the correct arguments, after milestone 1 and 2.  4. The messaging database must contain a message with a phone number matching the expected one exactly and the content loosely matching the expected text, after milestone 3.  Each milestone also defines a similarity measure which calculates a 0 to 1 similarity between each turn and the milestone. Types of available similarity measures are introduced in Appendix A.6. Milestones form a directed acyclic graph (DAG) based on temporal dependency. To evaluate a trajectory against a milestone DAG, we find the the highest averaged similarity scoreM+ among all possible mappings between turns and milestones, given that the resulting chronic milestone sequence is a topological sort of the DAG. Task efficiency is not considered by Milestones, and is instead tracked by a complementary turn count metric shown in Appendix D.2. We introduce the milestone matching process with more details in Appendix A.6.  Milestone evaluation combines the best of both worlds. As shown in Figure 1, it allows for explainable evaluation metrics like tool call AST matching and execution result exact match found in BFCL, while retaining the flexibility to evaluate any possible trajectory, similar to ToolEval.  On the other side of *Milestones*, there are *Minefields*, which define events that must NOT occur, as shown in Figure 3. This is mainly used in scenarios where we test that an agent understands that it cannot complete a task with the given tools instead of hallucinating. *Minefields* are otherwise identical to Milestones, except when the final trajectory similarity score is calculated. Assuming using Equation 2 we found the similarity score scoreM− for minefield DAG GM−(VM−, EM−), the final similarity score of the trajectory would be score = scoreM+ × I(scoreM− = 0), (1) ensuring if minefields are violated (non-zero minefield similarity), the similarity score for the whole trajectory is 0.  ![4_image_0.png](4_image_0.png) 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Evaluation
2408.04682
 | TOOLSANDBOX    | BFCL   | ToolEval   | API-Bank   |      | |----------------|--------|------------|------------|------| | Avg Turns      | 13.9   | 2.00       | 7.53       | 3.88 | | Avg Tool calls | 3.80   | 0.78       | 1.46       | 2.04 | | Test cases     | 1032   | 2000       | 1625       | 261  | | Tools          | 34     | 1193       | 3917       | 73   |  Table 3: Statistics between TOOLSANDBOX and other tool-use benchmarks. Calculation details can be found in Appendix B.1.  TOOLSANDBOX contains 1032 test cases meticulously crafted by 2 internal domain experts to capture challenging tool-use scenarios, with human authored and carefully calibrated Milestones and Minefields to support evaluation. 1 annotator is tasked to create test scenarios, while the other acts as an agent to validate milestones and minefields. We designed a rigorous annotation process to ensure coverage across realistic, complex use case scenarios, detailed in Appendix B.2. Statistics comparison between TOOLSANDBOX and other benchmarks can be found in Table 3. Tools in TOOLSANDBOX are designed to be representative, diverse and composable in conversational dialogs, while making tool count manageable for milestone annotation. As a result, TOOLSANDBOX test scenarios contain on average much higher number of tool calls and turns per dialog compared to other benchmarks. Additional details about tool domain coverage and design principles can be found in Appendix B.3.  To closely inspect the intricate challenges in LLM tool-use applications, test scenarios are organized into detailed categories, statistics can be found in Appendix B.4. A test scenario is defined by the initial world state, the initial messages, the available tools and the evaluation milestone and minefields. Variations of test scenarios are also introduced using the tool augmentations described in 2.1. Scenarios are organized by the following categories: Single / Multiple Tool Call These categories apply to scenarios where one / multiple tool calls are needed to fulfill the user task. Examples are shown in Appendix C.2. Note that this definition is different from the Berkeley Function-Calling leaderboard (Yan et al., 2024), which resembles distraction tools in TOOLSANDBOX described in Section 2.1.  Single / Multiple User Turn In the single user turn category, the first user message provides all necessary information to complete the task, whereas multiple user turn scenarios start with an ambiguous request or missing information, requiring further clarification from the user. An example is shown in Appendix C.2.  State Dependency The state dependency category describes scenarios where successful tool execution depends on the world state, e.g. settings like cellular service. The world state can be modified by the agent through the use of another tool.  Thus, an implicit dependency is formed between the tools, which can only be discovered through trial and error, as shown in Figure 1. There can even be nested state dependencies. As shown in Figure 19, sending a message would require cellular service to be turned on, but turning on cellular service requires low battery mode to be turned off.  This requires the agent to implicitly keep track of a call stack, and backtrack when necessary to fulfill the task efficiently.  Canonicalization Canonicalization refers to the process of transforming surface form representation commonly seen in a natural language query, to its corresponding canonical form, similar to INFORM dialog act in Schema Guided Dialog (Rastogi et al., 2020). This is particularly crucial when an API is less intelligent, and requires canonical form as argument. In some cases, canonicalization can be performed by the model itself, for example transforming 1B to *1_000_000_000*, or $ to corresponding ISO 4217 currency code USD. However, there are also cases where canonicalization requires the help of tools, for example transforming this Friday to *5/24/2024*, which requires knowledge about the current date, or transforming Golden Gate Bridge to the latitude longitude pair *(37.8199,* -122.4786), which requires a lookup in an external knowledge base. This scenario category captures both cases, probing the Agent's ability to perform canonicalization with or without the aid of tools.  Insufficient Information The insufficient information category is used for scenarios where the agent is not able to perform the task on purpose, by withholding a tool that would be needed for the task. This category exercises if the agent is able to identify that it cannot complete the task, as opposed to hallucinating tools or tool arguments, as shown in Figure 3. In these scenarios, minefields are defined to evaluate if tools that would imply hallucination are called or not. Comparing to relevance detection in BFCL where provided tools are often irrelevant to the task at hand, this is a much more challenging scenario, which requires the agent to reason over highly relevant tools to figure out the missing pieces. Comparing to solvability in ToolEval, which assumes full credit for any task deemed unsolvable, this is much more fine-grained, testing if the agent would hallucinate when the task is unsolvable. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Test Scenarios
2408.04682
 Open Source Models Table 4 shows the average similarity for each of the scenario categories described in Section 3 and tool augmentations described in Section 2.1. There is a significant performance gap between proprietary and open source models, with the best performing open source model Hermes (interstellarninja et al.) lagging more than 20 points behind the second to last proprietary model Claude-3-Haiku (Anthropic, 2024).  This is partly due to the fact that models like Gorilla (Patil et al., 2023) and Command-R (Cohere and for AI, 2024) are incapable of consuming tool responses, as shown in Appendix D.1. They can theoretically solve Single Tool Call test scenarios, but would fail in any scenario that requires multiple tool calls. As for Hermes and Mistral (Jiang et al., 2023), both models struggle at identifying when a tool call should be issued. Mistral for example would often mistake a tool-use scenario for a code generation task, as shown in Figure 11. These models' subpar performance unexpectedly caused them to achieve higher rating in the Insufficient Information category, which rewards the model for not generating hallucinated tool calls or arguments  Avg Score ↑ **Scenario Categories Tool Augmentations**  STC MTC SUT MUT SD C II 0 DT 3 DT 10 DT AT TNS TDS ADS ATS  GPT-4o-2024-05-13 **73.0** 87.8 80.1 84.2 **74.7** 84.0 **76.6** 42.0 75.1 75.0 74.6 72.6 **72.4** 69.3 73.0 **71.9**  Claude-3-Opus-20240229 69.2 83.5 70.0 74.5 67.2 74.5 71.1 57.3 68.3 68.6 70.0 67.5 70.8 **71.5** 65.8 71.1 GPT-3.5-Turbo-0125 65.6 **93.4** 73.9 81.8 66.6 82.6 70.4 22.3 67.3 63.2 67.0 65.4 63.9 64.3 66.7 66.9 GPT-4-0125-Preview 64.3 89.1 69.0 74.4 68.6 69.2 65.2 33.6 66.8 62.5 64.0 65.1 69.7 64.4 58.1 63.5 Claude-3-Sonnet-20240229 63.8 82.1 66.2 69.1 69.7 **84.5** 65.5 44.2 67.2 64.5 63.2 58.8 63.7 61.9 62.5 68.7  Gemini-1.5-Pro-001 60.4 82.6 49.8 63.1 37.3 70.5 51.6 76.2 63.3 63.1 60.8 59.8 62.2 60.5 58.7 54.4  Claude-3-Haiku-20240307 54.9 80.9 54.2 64.3 46.0 69.5 54.4 39.4 56.0 56.9 54.1 52.2 56.6 54.1 54.5 55.1 Gemini-1.0-Pro 38.1 68.7 21.6 36.5 14.6 39.3 18.2 65.5 38.2 39.5 41.9 37.7 40.1 35.3 36.7 34.9 Hermes-2-Pro-Mistral-7B 31.4 63.3 18.3 29.9 18.6 27.1 19.9 48.3 33.1 31.9 30.6 28.3 31.8 31.0 32.6 32.2 Mistral-7B-Instruct-v0.3 29.8 48.1 9.5 20.1 7.9 19.5 6.1 **76.8** 30.5 30.2 24.7 27.1 32.0 30.7 32.8 30.1 C4AI-Command-R-v01 26.2 52.6 12.7 23.0 12.7 3.1 18.0 47.8 24.8 27.9 25.6 23.3 25.0 25.6 28.7 28.3  Gorilla-Openfunctions-v2 25.6 36.2 8.2 15.1 9.3 0.0 8.9 69.2 25.5 27.5 26.1 18.6 24.5 27.1 26.8 28.6  C4AI-Command R+ 24.7 57.2 13.6 24.3 15.2 4.0 19.4 35.3 23.4 27.2 24.9 23.5 21.7 27.6 24.8 24.8  when provided tools are insufficient to complete the task. This should be considered a side effect instead of a positive outcome.  Proprietary Models Of the proprietary models, GPT-4o (OpenAI, 2024) achieves the highest similarity score, with Claude-3-Opus closely behind.  Both models have their own strengths. While GPT4o achieves the higher score, Claude-3-Opus maintains a lower average turn count as shown in Appendix D.2, achieving the user goal with higher efficiency. Interestingly, comparing the largest and smallest models in the GPT, Claude and Gemini families (Reid et al., 2024), Multiple Tool Call and Multiple User Turn categories deteriorate much faster than Single Tool Call and Single User Turn, showing that reasoning about complex tool call sequences and ambiguous user requests requires much more model capacity.  State Dependency The State Dependency category shows an interesting trend where, larger models like GPT-4 (Achiam et al., 2023) and Claude-3-Opus perform significantly worse than mid to smaller sized models like GPT-3.5-Turbo and Claude-3-Sonnet. This is due to erroneous parallel tool calls in face of state dependency. As mentioned in Section 2.2, the Execution Environment always surfaces race conditions when present.  Larger models like GPT-4 and Claude-3-Opus are prune to issuing parallel tool calls even for dependent tools, leading to a performance deficiency. An example is shown in Figure 17. Nested state dependency is also tricky to solve efficiently. As shown in Figure 18, models often forget about open issues and would not optimally backtrack, leading to repeated errors and as a result a much higher than 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Evaluation Results
2408.04682
 Canonicalization Canonicalization remains a challenging category for all models, especially in tool assisted canonicalization. Larger models would tend to memorize world knowledge that is unlikely to change, like latitude longitude for famous geographical location, while smaller models are more keen on using tools.  However, time related arguments in specific show to be really challenging to canonicalize and reason about. Models would frequently hallucinate timestamps (Figure 15), and incorrectly canonicalize relative date and time (Figure 14).  In addition, models could take premature decisions in face of ambiguity, also leading to canonicalization errors. In Figure 16, multiple location entities were returned in the tool response, while the model simply chose the first one, without returning to the user for disambiguation.  Insufficient Information Insufficient Information performance overall negatively correlates with other categories. The stronger the model performance on complex tasks, the worse the insufficient information performance, showing its value at evaluating model reasoning capabilities. Even with simple tasks and very little tools, top performing models like GPT-3.5-Turbo and GPT-4 could hallucinate tool name, or hallucinate arguments, as shown in Figure 3 and 20. The test scenario's difficulty positively correlates with the number of steps involved in the tasks, as the models would get lost in solving immediate errors, and forget about the main objective.  Tool Augmentations Robustness against tool augmentations seems to vary model by model.  While adding distraction tools, Claude-3-Sonnet seems to be affected the most, dropping almost 10 points between 0 distraction tools, and making all TOOLSANDBOX tools available. GPT-4o is particularly susceptible to Tool Description Scrambling, GPT-4 pays extra attention to argument descriptions, and Gemini-1.5 doesn't do well with Argument Type Scrambling. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Optimal Turn Count.
2408.04682
 Tool-use Benchmarks Various tool-use benchmarks have been developed to evaluate LLM-based agent performance in different tool-use domains.  The Berkeley Function Calling Leaderboard (Yan et al., 2024), ToolBench (Qin et al., 2023b), StableToolBench (Guo et al., 2024), NexusRaven V2 Function Calling Benchmark (team, 2023), and API-BLEND (Basu et al., 2024) assess the ability of LLM agents to plan and perform function calls. WebArena (Zhou et al., 2023), MiniWoB++ (Humphreys et al., 2022), Webshop (Yao et al., preprint), Mind2Web (Deng et al., 2023) and VisualWebArena (Koh et al., 2024) focus on the agent's ability to call search functionality to browse and leverage the web to solve the task. Apart from benchmarks specifically designed for tool-use, generalist agent benchmark suites like AgentBench (Liu et al., 2023) and AgentBoard (Ma et al., 2024) include evaluating the tool-use capability of agents as a central task to examine the general problemsolving ability of LLM-based agents.  Tool-use agent Various tool-use model have been developed to solve the complicated tool-use scenarios in real-world. Toolformer (Schick et al., 2023) first demonstrated that language models could autonomously learn to use various tools, through a self-supervised learning approach. Gorilla (Patil et al., 2023) employs a self-instruct paradigm to generate {instruction, API} pairs and is trained both with and without a retriever. ToolLLM (Qin et al., 2024) enables LLMs to use over 16,000 realworld APIs by automating the generation of diverse instructional data and leveraging a neural API retriever, showing better generalization across unseen APIs. CodeACT (Wang et al., 2024a) integrates executable code actions into training to enhance the decision-making and task-solving capabilities of LLMs, leading to more effective agents.  Dialogue State Tracking Dialogue State Tracking (DST) requires the agent to maintain and update dialogue states and actions. The MultiWOZ dataset (Budzianowski et al., 2018) offers a diverse set of dialogues requiring complex state tracking across multiple domains. Building on this, Rastogi et al.  (2020) proposed a schema-guided approach to DST, addressing scalability issues in multi-domain settings and enhancing the model's adaptability and reducing the need for extensive domain-specific annotations. However, these datasets focus on explicit state tracking on off-policy trajectories.  Our benchmark complements them by introducing world states, typically implicit and requiring to be inferred from world knowledge, and an interactive environment that offers a more diverse and extensive online evaluation.  User Simulator in Sandbox When assessing the agent's core competencies in state tracking, memorization, and long-term planning, dialogues between users and the system can span over several turns, and off-policy evaluation may not always suffice. On the other hand, human-in-the-loop online evaluation is costly and time-consuming. Some studies have investigated incorporating a built-in user simulator to facilitate the evaluation process.  DAUS (Sekulic et al., 2024) utilizes LLM finetuned on task oriented dialog trajectories. MINT (Wang et al., 2024b) utilizes GPT-4 to simulate natural language user feedback for multi-turn LLM evaluation. For medical agents, AMIE (Tu et al., 2024) integrates a built-in patient model to engage with a symptom collection agent. Zhang et al. (2024) develop a virtual environment for the agent model to predict unknown entities by interacting with a user simulator that responds with only yes or no.  Our approach resonates with these approaches. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Related Work
2408.04682
 TOOLSANDBOX presents a stateful, conversational and interactive evaluation benchmark for LLM tooluse capabilities. With stateful and state dependent tools, LLM simulated user and flexible evaluation with milestones and minefields, it showcased a significant performance gap between open source and proprietary models, and unveiled challenging scenarios even for SOTA models, including State Dependency, Canonicalization and Insufficient Information, bringing new insights to understanding tool-use capabilities. We hope TOOLSANDBOX could be a valuable addition to LLM evaluation suites, pushing the boundary of tool-use research.  8 
conclusion
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Conclusion
2408.04682
 While TOOLSANDBOX is powerful, being the first work of its kind, it still has many areas to be improved upon. In this section, we introduce some of these limitations, to motivate future research in this area.  Even though Milestone and Minefields are powerful interactive metrics that offer insights into intermediate and final outcomes, authoring them, especially mandatory intermediate milestones, requires deep knowledge around the tool capacities in TOOLSANDBOX and many iterations, hindering its scalability. A simplified, or fully automatic method for identifying Milestones and Minefields could be the key to further scale up the data volume in TOOLSANDBOX.  Despite our best effort at controlling user simulator behavior, it is still subject to non-negligible hallucination and instruction following errors. In this work we toyed with the idea of tool assisted user simulator. Only one tool end_conversation was offered to the user simulator, and we saw a noticeable improvement in instruction following in the dialog termination aspect. By expanding its tool set, a tool assisted user simulator could be a promising direction to further reduce hallucination and improve instruction following.  Mandatory confirmation and authentication is an interesting problem currently not addressed in TOOLSANDBOX. In dialog state tracking, confirmation is modeled by its corresponding dialog action before a transactional service is called. However, in most tool-use LLM designs, we are at the liberty of the model to decide when a confirmation is necessary. An orchestration level solution to enforce confirmation, similar to GoEx (Patil et al., 2024) could be a potential inspiration, and an interesting problem for models to reason over.  A challenging category of tools, namely tools that spawn a daemon process, e.g. setting a timer, is not addressed in TOOLSANDBOX currently. These tools complete and return in the main process after the daemon is spawned, and at some point in the future, the daemon would interrupt the main process, e.g. when the time is up. This kind of interruption poses a novel problem for both execution orchestration, and the model itself.  While most of TOOLSANDBOX tools are selfcontained implementations, some tools that depend on an external knowledge base, like searching for weather, are still backed by external web services, affecting its reproducibility. A cached solution similar to StableToolBench (Guo et al., 2024) could maintain tool implementation simplicity, while providing stable, reproducible results. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Limitations
2408.04682
 Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.  arXiv preprint arXiv:2303.08774.  AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. *Claude-3 Model Card*.  Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A. Lastras. 2024. Api-blend: A comprehensive corpora for training and benchmarking api llms. *ArXiv*, abs/2402.15491.  Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašic. 2018. ´ MultiWOZ - a largescale multi-domain Wizard-of-Oz dataset for taskoriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.  Cohere and Cohere for AI. 2024. Command r. https://huggingface.co/CohereForAI/ c4ai-command-r-v01.  Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su.  2023. Mind2web: Towards a generalist agent for the web. *Preprint*, arXiv:2306.06070.  Python Software Foundation. 2024. Python 3.9.19 documentation.  Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models. *Preprint*, arXiv:2403.07714.  Matthew Henderson, Blaise Thomson, and Jason D.  Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263–272, Philadelphia, PA, U.S.A. Association for Computational Linguistics.  Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. 2022. A data-driven approach for learning to control computers. In *International* Conference on Machine Learning, pages 9466–9482.  PMLR.  interstellarninja, teknium, theemozilla, karan4d, and huemin_art. Hermes-2-pro-mistral-7b.  https://huggingface.co/NousResearch/ Hermes-2-Pro-Mistral-7B.  Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. *ArXiv preprint*, abs/2310.06825.  Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649.  Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-bank: A comprehensive benchmark for tool-augmented LLMs. In *Proceedings of the 2023 Conference on Empirical Methods* in Natural Language Processing, pages 3102–3116, Singapore. Association for Computational Linguistics. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents. *arXiv preprint arXiv:2308.03688*.  Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents.  Preprint, arXiv:2401.13178.  OpenAI. 2024. Gpt-4o. https://openai.com/ index/hello-gpt-4o/.  Shishir G. Patil, Tianjun Zhang, Vivian Fang, Noppapon C., Roy Huang, Aaron Hao, Martin Casado, Joseph E. Gonzalez, Raluca Ada Popa, and Ion Stoica. 2024. Goex: Perspectives and designs towards a runtime for autonomous llm applications. *Preprint*, arXiv:2404.06921.  Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. *arXiv preprint* arXiv:2305.15334.  Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023a. Tool learning with foundation models. *Preprint*, arXiv:2304.08354.  Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. Toolllm: Facilitating large language models to master 16000+ real-world apis. In *The Twelfth International Conference on Learning Representations*.  Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023b. Toolllm: Facilitating large language models to master 16000+ real-world apis.  Preprint, arXiv:2307.16789.  Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8689–8696.  Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. *arXiv preprint arXiv:2403.05530*.  Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, volume 36, pages 68539–68551.  Curran Associates, Inc.  Ivan Sekulic, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, Andre Ferreira Manso, and Roland Mathis. 2024. Reliable LLM-based user simulator for task-oriented dialogue systems. In Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT 2024), pages 19–35, St. Julians, Malta. Association for Computational Linguistics.  Nexusflow.ai team. 2023. Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling.  Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al. 2024.  Towards conversational diagnostic ai. *arXiv preprint* arXiv:2401.05654.  Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024a. Executable code actions elicit better llm agents. In *ICML*.  Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2024b. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. In The Twelfth International Conference on Learning Representations.  Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E.  Gonzalez. 2024. Berkeley function calling leaderboard.  Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. preprint. Webshop: Towards scalable real-world web interaction with grounded language agents. In *ArXiv*.  Yizhe Zhang, Jiarui Lu, and Navdeep Jaitly. 2024. Probing the multi-turn planning capabilities of llms via 20 question games.  Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. Webarena: A realistic web environment for building autonomous agents. *arXiv preprint arXiv:2307.13854*. 
references
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
References
2408.04682
 This appendix section introduces implementation details about the TOOLSANDBOX design. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
A Implementation Details
2408.04682
 Execution Context represents the complete state of the TOOLSANDBOX. More specifically, it contains tool databases for stateful tools, referred to as *World State* in Figure 1, and the dialog history between different roles, referred to as *Message Bus*.  It maintains a snapshot of all tool databases and dialog history at any given turn, allowing for easy introspection and evaluation. The Execution Context exists as a global variable for all roles and tools to easily access, while prohibiting direct manipulation from the LLM agent. This allows us to implement stateful tools that can manipulate or access database stored in the Execution Context, without defining it as function argument. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
A.1 Execution Context
2408.04682
 Tools are implemented as type-hinted, doc-string equipped Python functions, as shown in Listing 1.  When a tool is passed to the Agent as an available tool, type hints and doc-string are converted into JSON API schema, as shown in Figure 9.  ![11_image_3.png](11_image_3.png)  When a tool is executed, the name, input and output of the tool is committed into the current Execution Context as a tool trace, which allows for automatic evaluation. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
A.2 Tools
2408.04682
 To enable ablation studies of how the tool schema affects agent accuracy we have implemented multiple augmentations.  Distraction Tools In addition to necessary tools that must be present to complete a task, 0, 3, 10 or all the rest of the tools in the sandbox are made available to the Agent in addition, to evaluate the model's ability to pick the right tool. Distraction tools are chosen from a sorted list, where tools with domain overlap and textual similarities with necessary tools are prioritized. In addition, all of the scrambling augmentations below are applied in conjunction to adding 3 distraction tools, to ensure the augmentation is challenging yet feasible.  Tool name scrambling The name of the tool is modified to a less informative form, e.g.  messages_0 instead of send_message. The agent LLM should be able to infer the purpose of the tool based on the description that is also part of the tool definition.  Tool description scrambling This removes the one-liner summary of the documentation, see Listing 2. The agent LLM still has access to the tool name, argument names as well as argument and return value documentations.  Listing 2: Tool description scrambling example def send_message(phone_number: str, content: str) -> str:  ![11_image_0.png](11_image_0.png) ![11_image_2.png](11_image_2.png)  ![11_image_1.png](11_image_1.png)  Argument description scrambling The section  ![11_image_4.png](11_image_4.png)  ![11_image_5.png](11_image_5.png)  of the documentation explaining the arguments is being removed, see Listing 3. The agent LLM can infer the arguments from the tool definition as well as discover the correct usage through trial and error.  number.  ## 1 Introduction  The study of the $\alpha$-function of the \(\  ![11_image_6.png](11_image_6.png)  ![11_image_7.png](11_image_7.png)  Listing 3: Argument description scrambling example def send_message(phone_number: str, content: str) -> str:  ![11_image_8.png](11_image_8.png)  Argument type scrambling The type hints in the tool declaration are removed. The agent LLM still has access to the argument documentation. Note that when an agent generate argument does not conform to the annotated data type, an exception will be raised, presenting the expected data type, ensuring the problem is always solvable in face of this augmentation. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
A.2.1 Tool Augmentations
2408.04682
 ![12_Image_0.Png](12_Image_0.Png)  - A *Knowledge Boundary* section describing what the user simulator should or should not know.  - A *Demonstration* section including few shot dialog examples to further improve instruction following capabilities. Note that demonstration is only visible to the user role. It does not affect the agent and execution environment roles. An example of these components can be found  ![12_image_1.png](12_image_1.png)  in Figure 7. Without Knowledge Boundary and Demonstration, hallucination and instruction following errors can happen much more frequently, as shown in Figure 5 and 6.  ![12_image_2.png](12_image_2.png) 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
A.3 Message Bus
2408.04682
 We introduce the implementation details of User Role in this section. As is shown in Figure 7, user simulator prompts consists of 3 components:  - A *User Goal* section describing the general instructions and the goal of the simulated user.  The idea of role reversal was challenging for the simulator, so we opted to refer to the agent as another user (User B), which improves instruction following. User simulation error rate are consistent across representative agent models, as shown in Table 5.  A decrease of error rate for Mistral was caused by the fact that, open source models would even struggle with prompting for user input correctly. Considering the wide performance gap between open source and proprietary models in Table 4, this does not affect our conclusions for Table 4.  ![13_image_0.png](13_image_0.png)  | GPT4-o        | Claude-3-Opus   | Gemini-1.5-Pro   | Mistral-7B-Instruct   |           | |---------------|-----------------|------------------|-----------------------|-----------| | Hallacination | 6.90±1.45       | 6.40±0.97        | 7.15±0.71             | 3.10±0.46 | | 1.11±0.84     | 1.38 ±0.69      | 0.92±0.49        | 0.63±0.37             |           | | Total Error   | 8.02±1.36       | 7.78±0.52        | 8.07±1.03             | 3.73±0.64 |  Table 5: Percentage of user simulation failures in each failure category for each agent model.  Mean and std collected from 4 repeated trials. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
A.4 User Role
2408.04682
 Prompts used for all agents is shown in Figure 8.  The prompt is meant to be consistent for all agents, and does not leak specific information about the testing environment.  Agent Prompt Don't make assumptions about what values to plug into functions Ask for clarification if a user request is ambiguous Figure 8: Prompt for Agent role.  Figure 9 shows how tool calls requested by the Agent are converted to Python code, which can then be executed by the Execution Environment .  ![13_image_1.png](13_image_1.png) 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Agent Role A.5
2408.04682
 To evaluate a trajectory against a milestone DAG, We attempt to find the best match between milestone nodes and trajectory snapshots among all possible mappings. Formally, suppose we have a milestone DAG G M +( VM +, EM +), | VM + | = m, a sequence of database snapshots at each conversation turn S n = ( s 1 , s 2 , ..., s n ) and a similarity measure sim : V M + × S → [0, 1] measuring how close a milestone is to a snapshot, we aim to find the best mapping function f + : S → V M + which achieves the highest averaged similarity score, under the constraint that mapped milestones forms a possible topological sort of G M + :  $$\begin{array}{l}{{\mathrm{avgsim}_{+}=\frac{1}{m}\sum_{i=1}^{m}s i m(v_{M+}^{i},f(v_{M+}^{i})),}}\\ {{f^{+}=\mathrm{\large~\arg\max~\~avgsim}_{+},}}\\ {{f^{+}(S_{n})\in t o p(G_{M+})}}\\ {{\mathrm{score}_{M+}=\mathrm{\max{avgsim}}_{+}.}}\end{array}\tag{2}$$  The max value is the similarity score between the trajectory and the milestone DAG.  The similarity measure sim ( v i , s j ) calculates similarity between a database snapshot and a target database defined in the milestone. The milestone defines the column wise similarity function used for each column. These function have a [0, 1] output space, and could be exact matching for cellular service, ROUGE-L F measure for message content, AST matching for tool trace similar to the AST metric found in BFCL (Yan et al., 2024 ), and many more. This allows for great flexibility when defining milestones. For a snapshot database and milestone target both containing k rows, we calculate a pairwise similarity for those rows da,b by calculating the geometric mean of column similarities. Then we solve for the best assignment problem between snapshot and milestone rows, by maximizing the geometric mean of row similarities, which will be the similarity measure sim(v i, sj).  We use the geometric mean throughout to ensure that, if any column similarity must not be violated, it could emit a 0 similarity, which would nullify the overall similarity measure.  In addition to similarities conditioning on the current milestone and snapshot, in some cases we also allow an additional "reference milestone" to be provided, enabling similarity to be conditioned on two milestones. This can unlock powerful constraints including guradrail_similarity, which checks if any changes are made to a certain database between two milestone events, and tool_trace_dependant_similarity, which allows one to extract tool trace output from a reference milestone, and ingest into the current milestone, allowing one to track the information flow of tools. An example can be found in Figure 10.  Milestone evaluation is a powerful tool that unlocks a deeper understanding into model performance, and hints at possible areas of improvement.  An example is shown in Figure 10. In the end, the task was not completed before the maximum allowed number of turns. However, intermediate milestones showcased that the model was capable of solving the state dependency challenges and requesting the current location. In order to successfully resolve this test case, we should improve the model's turn efficiency on state dependency. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Evaluation A.6
2408.04682
 The tool-use benchmark statistics shown in Table 3 are calculated as follows:  - Average turn considers any message between the user, the agent or the tools as 1 turn.  - For TOOLSANDBOX, statistics are calculated from trajectories collected on GPT-4o agent.  - BFCL only evaluates tool call generation from a single user prompt, which we consider as 2 turns.  - ToolEval was calculated from ToolLlama DFS Retriever trajectories.  - API-BANK was calculated from level 1 and 2 test set. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
B Test Scenarios B.1 Tool-Use Benchmark Comparisons
2408.04682
 Test scenarios are created by 2 internal domain experts from our institute who are familiar with the tool capacities in TOOLSANDBOX, and the field of task-oriented dialog. 1 annotator creates test scenarios including the starting world state, user task, initial message, and milestone/minefield. To ensure dataset diversity while making the annotation task feasible, the annotator followed the process below:  - The annotation process starts by creating seed scenarios. These are simple, single user turn, single tool call, self-contained requests that are supposed to cover most tools, as well as most of their arguments. For example, a seed scenario for add_reminder tool that requires timestamp, latitude, longitude would likely contain a starting user utterance saying Create a reminder to buy chocolate milk at timestamp 111222333 at latitude 37 longitude -122. Creating milestone for said scenario is trivial. - Next, starting from the seed scenario, the annotator branches off to create derived scenarios that are more involved. This could be a Multiple Tool Call scenario, which requires the agent to invoke other tools before this one, e.g. *Create a reminder to buy chocolate milk* tomorrow 5PM at Whole Foods on McKinley Ave, which requires reasoning for the relative datetime, as well as searching for location latitude longitude. Note that this will also be considered a **Canonicalization** scenario.  - It could be a **Multiple User Turn** scenario, which requires the agent to request more slots from the user. e.g. *Create a reminder*.  - It could be a **State Dependency** scenario, which requires the model to solve state dependencies, e.g. *Create a reminder to buy chocolate milk tomorrow 5PM at Whole Foods on* McKinley Ave, but wifi is set to off. Preventing access to location search capability unless the dependency is resolved.  - It could be an **Insufficient Information** scenario, which requires the model to figure out this task cannot be solved, e.g *Create a reminder to buy chocolate milk tomorrow 5PM*  ![15_image_0.png](15_image_0.png)  at Whole Foods on McKinley Ave , but the model does not have access to current timestamp.  - These branches can be combined as well, creating numerous combinatorial and complex scenarios. Building branches off of seed also makes annotating milestones an incremental process that's easier to accomplish.  - Finally, to cover the diverse and ambiguous nature of realistic user dialog, the annotator paraphrased the scenarios above into alternative phrasing, e.g. Create a reminder to buy chocolate milk tomorrow → I need to buy chocolate milk tomorrow . For these alternative phrasings, milestone definitions can be reused, reducing annotation workload. After the test scenarios were collected, the other annotator acts as an agent to validate the feasibility of the task and Milestones / Minefields.  This annotator has the same message sub-view as a model agent and is asked to try to complete the task. After the test scenarios are validated through this process, at least 4 rounds of tests are issued to multiple model-based agents, to further confirm test and milestone/minefield validity against correct and incorrect trajectories. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
B.2 Annotation Process
2408.04682
 Tool design choices in TOOLSANDBOX are driven by two major principles:  - Tools should be representative and diverse, to cover key task oriented dialog use cases as well as T OOL S ANDBOX test scenario categories.  - Tool capacities should be well-defined, and tool counts should be manageable, so that milestone / minefield annotation is feasible for annotators.  Driven by these guiding principles, we designed 34 tools in TOOLSANDBOX covering 11 domains including Contact, Messaging, Reminder, System settings, Time utilities, Math utilities, Map, Weather, Stock, Conversion, and Holiday, backed by python native implementation when possible, carefully selected RapidAPI endpoints when necessary. In more details:  - Each domain designed at least one "omnisearch" tool. All possible search fields should be present as arguments for this tool, and all relevant information for this domain should be returned in the response. As an example, if a user would like to ask for the lowest temperature or humidity for a location, the agent should invoke the search_weather tool for both requests, and the agent is expected to retrieve corresponding key values based on user intent. This ensures "search" requests within a domain have 1 single entry point.  - Stateful domains should implement at least one state manipulation tool. This could be adding a new database entry, e.g.  send_message, modifying an existing entry, e.g. set_wifi_status, or both, e.g.  add_reminder and modify_reminder.  - Utilities should be created to ensure the agent could transform necessary surface / canonical form slot types, e.g.  timestamp_to_datetime_info turns Unix timestamp into year, month, day, and weekday; and calculate expected slot values, e.g.  calculate_lat_lon_distance calculates the distances between two latitude-longitude pairs. While agents are allowed to infer these with its own inherent abilities, utility tools should be created to ensure the agents are not forced to. Specifically for State Dependencies, we designed 4 types of state dependencies. 44% of all tools are either tools that depend on these states, manipulate them, or inspect them. These include:  - Cellular service: Tools that require cellular service (e.g., send_message) depend on this state to be true.  - Wifi: All RapidAPI-backed tools (e.g., search_stock) depend on this state to be true.  - Location Service: All tools that utilize current location (e.g., get_current_location) depend on this state to be true.  - Low battery mode: All tools that turn on cellular service, wifi, and location service status depend on this state to be false, creating nested state dependency. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
B.3 Tool Design
2408.04682
 The number of test scenarios per scenario category in TOOLSANDBOX can be found in Table 6  | Test Scenario Count      |     | |--------------------------|-----| | SINGLE_TOOL_CALL         | 152 | | MULTIPLE_TOOL_CALL       | 656 | | SINGLE_USER_TURN         | 584 | | MULTIPLE_USER_TURN       | 224 | | STATE_DEPENDENCY         | 192 | | CANONICALIZATION         | 472 | | INSUFFICIENT_INFORMATION | 224 |  Table 6: Number of test scenarios per category. Note that one test scenario can be assigned with multiple scenario categories. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
B.4 Toolsandbox **Category Statistics**
2408.04682
 ![16_Image_0.Png](16_Image_0.Png) Single/Multiple Tool Call/User Turn  ![17_image_0.png](17_image_0.png)  Message Bus  ![17_image_1.png](17_image_1.png)  Figure 14: Example trajectories where GPT-4 incorrectly inferred relative time. Instead of deducing next Friday 5PM by inspecting current day and weekday, GPT-4 randomly shifted current time stamp by 6 days and 16 hours.  ![17_image_2.png](17_image_2.png)  Message Bus  ![17_image_3.png](17_image_3.png)  Message Bus  ![18_image_0.png](18_image_0.png) 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
C Example Trajectories C.1 Tool Call Detection
2408.04682
 Figure 16: Disambiguation failure from GPT-4o. The User intended to set a reminder at Whole Foods McKinley Ave in multiple turns.  However, upon receiving multiple possible entities, GPT-40 chose to set the reminder with the first location unannounced, without disambiguating with the User, leading to undesired results. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Create A Reminder To Buy Chocolate Milk Tomorrow 5Pm At Whole Foods On Mckinley Ave.
2408.04682
 ![18_image_2.png](18_image_2.png)  Message Bus  ![18_image_1.png](18_image_1.png)  Message Bus  ![19_image_0.png](19_image_0.png) 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
C.4 State Dependency
2408.04682
 ![19_image_1.png](19_image_1.png)  Figure 20: Example trajectories where GPT-3.5 failed 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Insufficient Information C.5
2408.04682
 | Generate                 | Consume       |    | |--------------------------|---------------|----| | Tool Call                | Tool Response |    | | GPT-4o-2024-05-13        | r             | s  | | GPT-4-0125-Preview       | s             | r  | | GPT-3.5-Turbo-0125       | l             | l  | | Claude-3-Opus-20240229   | A             | l  | | Claude-3-Sonnet-20240229 | r             | s  | | Claude-3-Haiku-20240307  | s             | s  | | Gemini-1.5-Pro-001       | r             | s  | | Gemini-1.0-pro           | r             | r  | | Hermes-2-Pro-Mistral-7B  | r             | r  | | r                        | r             |    | | Mistral-7B-Instruct-v0.3 |               |    | | Gorilla-Openfunctions-v2 | x             |    | | r                        |               |    | | C4AI-Command-R-v01       | 2             | x  | | C4AI-Command-R+          | x             |    | 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Additional Evaluation Results D
2408.04682
 Table 7: A comparison between model feature support.  Command R models are tested through Huggingface released weights, which, to the best of our knowledge, does not provide a prompt template for tool response consumption.  Some models tested in this work, especially open source models, do not support all features required for a conversational, interactive tool-use workflow.  We think it is useful to document these shortcomings to motivate future research, and set the right context while understanding the experiment metrics from these models.  In a conversational, interactive tool-use workflow, the agent needs to be able to accept multiple rounds of user input, decide when to generate a tool call or respond to user, and consume a tool response to determine the next step. However, as shown in 7, open source models including Gorilla and Command-R are not capable of consuming tool responses. Because of this, they can theoretically produce reasonable results for Single Tool Call test scenarios, but cannot complete any test scenario that requires multiple tool calls. 
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Model Feature Comparison D.1
2408.04682
body
Toolsandbox**: A Stateful, Conversational, Interactive Evaluation** Benchmark For Llm Tool Use Capabilities
Turn Count Comparison D.2