<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.00121] Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks</title><meta property="og:description" content="Large language models (LLMs) have recently shown tremendous promise in serving as the backbone to agentic systems, as demonstrated by their performance in multi-faceted, challenging benchmarks like SWE-Bench and Agent-…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.00121">

<!--Generated on Mon Aug  5 13:36:36 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks</h1>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id11.id1" class="ltx_p">Large language models (LLMs) have recently shown tremendous promise in serving as the backbone to agentic systems, as demonstrated by their performance in multi-faceted, challenging benchmarks like SWE-Bench and Agent-Bench.
However, to realize the true potential of LLMs as autonomous agents, they must learn to identify, call, and interact with external tools and application program interfaces (APIs) to complete complex tasks. These tasks together are termed <span id="id11.id1.1" class="ltx_text ltx_font_italic">function calling</span>.
Endowing LLMs with function calling abilities leads to a myriad of advantages, such as access to current and domain-specific information in databases and knowledge sources, and the ability to outsource tasks that can be reliably performed by tools, e.g., a Python interpreter or calculator.
While there has been significant progress in function calling with LLMs, there is still a dearth of open models that perform on par with proprietary LLMs like GPT, Claude, and Gemini.
Therefore, in this work, we introduce the <span id="id11.id1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note"><span id="footnote1.1.1.1" class="ltx_text ltx_font_upright">1</span></span><span id="footnote1.5" class="ltx_text ltx_font_upright">The model will be available soon at </span><a target="_blank" href="https://huggingface.co/ibm-granite/" title="" class="ltx_ref ltx_url ltx_font_typewriter ltx_font_upright">https://huggingface.co/ibm-granite/</a></span></span></span></span> model under an Apache 2.0 license. The model is trained using a multi-task training approach on seven fundamental tasks encompassed in function calling, those being Nested Function Calling, Function Chaining, Parallel Functions, Function Name Detection, Parameter-Value Pair Detection, Next-Best Function, and Response Generation. We present a comprehensive evaluation on multiple out-of-domain datasets comparing <span id="id11.id1.3" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> to more than 15 other best proprietary and open models.
<span id="id11.id1.4" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> provides the best performance among all open models on the Berkeley Function Calling Leaderboard and fourth overall.
As a result of the diverse tasks and datasets used for training our model, we show that <span id="id11.id1.5" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> has better generalizability on multiple tasks in seven different evaluation datasets.</p>
</div>
<div id="id10" class="ltx_logical-block">
<div id="id10.p1" class="ltx_para">
<p id="id7.7" class="ltx_p ltx_align_center"><span id="id2.2.2" class="ltx_text ltx_font_bold">Ibrahim Abdelaziz<sup id="id2.2.2.1" class="ltx_sup"><span id="id2.2.2.1.1" class="ltx_text ltx_font_medium">⋆</span></sup><sup id="id2.2.2.2" class="ltx_sup"><span id="id2.2.2.2.1" class="ltx_text ltx_font_medium">†</span></sup></span> <span id="id4.4.4" class="ltx_text ltx_font_bold">Kinjal Basu<sup id="id4.4.4.1" class="ltx_sup"><span id="id4.4.4.1.1" class="ltx_text ltx_font_medium">⋆</span></sup><sup id="id4.4.4.2" class="ltx_sup"><span id="id4.4.4.2.1" class="ltx_text ltx_font_medium">†</span></sup></span> <span id="id6.6.6" class="ltx_text ltx_font_bold">Mayank Agarwal<sup id="id6.6.6.1" class="ltx_sup"><span id="id6.6.6.1.1" class="ltx_text ltx_font_medium">⋆</span></sup><sup id="id6.6.6.2" class="ltx_sup"><span id="id6.6.6.2.1" class="ltx_text ltx_font_medium">†</span></sup></span> <span id="id7.7.8" class="ltx_text ltx_font_bold">Sadhana Kumaravel</span> <span id="id7.7.9" class="ltx_text ltx_font_bold">Matthew Stallone</span> <span id="id7.7.10" class="ltx_text ltx_font_bold">Rameswar Panda</span> <span id="id7.7.11" class="ltx_text ltx_font_bold">Yara Rizk</span> <span id="id7.7.12" class="ltx_text ltx_font_bold">GP Bhargav</span> <span id="id7.7.13" class="ltx_text ltx_font_bold">Maxwell Crouse</span> <span id="id7.7.14" class="ltx_text ltx_font_bold">Chulaka Gunasekara</span> <span id="id7.7.15" class="ltx_text ltx_font_bold">Shajith Ikbal</span> <span id="id7.7.16" class="ltx_text ltx_font_bold">Sachin Joshi</span> <span id="id7.7.17" class="ltx_text ltx_font_bold">Hima Karanam</span> <span id="id7.7.18" class="ltx_text ltx_font_bold">Vineet Kumar</span> <span id="id7.7.19" class="ltx_text ltx_font_bold">Asim Munawar</span> <span id="id7.7.20" class="ltx_text ltx_font_bold">Sumit Neelam</span> <span id="id7.7.21" class="ltx_text ltx_font_bold">Dinesh Raghu</span> <span id="id7.7.22" class="ltx_text ltx_font_bold">Udit Sharma</span> <span id="id7.7.23" class="ltx_text ltx_font_bold">Adriana Meza Soria</span> <span id="id7.7.24" class="ltx_text ltx_font_bold">Dheeraj Sreedhar</span> <span id="id7.7.25" class="ltx_text ltx_font_bold">Praveen Venkateswaran</span> <span id="id7.7.26" class="ltx_text ltx_font_bold">Merve Unuvar</span> <span id="id7.7.27" class="ltx_text ltx_font_bold">David Cox</span> <span id="id7.7.28" class="ltx_text ltx_font_bold">Salim Roukos</span> <span id="id7.7.29" class="ltx_text ltx_font_bold">Luis Lastras</span> <span id="id7.7.7" class="ltx_text ltx_font_bold">Pavan Kapanipathi<sup id="id7.7.7.1" class="ltx_sup"><span id="id7.7.7.1.1" class="ltx_text ltx_font_medium">†</span></sup></span></p>
<p id="id10.p1.1" class="ltx_p ltx_align_center">IBM Research</p>
<p id="id8.8" class="ltx_p ltx_align_center"><sup id="id8.8.1" class="ltx_sup">⋆</sup>Equal Contribution</p>
<p id="id9.9" class="ltx_p ltx_align_center"><sup id="id9.9.1" class="ltx_sup">†</sup>Corresponding Authors</p>
<p id="id10.p1.2" class="ltx_p ltx_align_center"><span id="id10.p1.2.1" class="ltx_text ltx_font_typewriter">{ibrahim.abdelaziz1, kinjal.basu, mayank.agarwal}@ibm.com, kapanipa@us.ibm.com</span></p>
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Large language models (LLMs) have garnered significant attention due to their broad applicability to an important set of challenging domains, e.g., programming <cite class="ltx_cite ltx_citemacro_citep">(Mishra et al., <a href="#bib.bib22" title="" class="ltx_ref">2024</a>; Roziere et al., <a href="#bib.bib35" title="" class="ltx_ref">2023</a>)</cite>, reasoning <cite class="ltx_cite ltx_citemacro_citep">(Reid et al., <a href="#bib.bib34" title="" class="ltx_ref">2024</a>; Jiang et al., <a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>, and multi-modal interaction <cite class="ltx_cite ltx_citemacro_citep">(Reid et al., <a href="#bib.bib34" title="" class="ltx_ref">2024</a>)</cite>. Increasingly, applying these models to solve real-world problems requires them to act as autonomous agents powering intelligent decision-making in specific environments <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib50" title="" class="ltx_ref">2022</a>; Xu et al., <a href="#bib.bib44" title="" class="ltx_ref">2023a</a>; Yang et al., <a href="#bib.bib47" title="" class="ltx_ref">2024a</a>)</cite><span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>Auto-GPT:<a target="_blank" href="https://github.com/Significant-Gravitas/AutoGPT" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/Significant-Gravitas/AutoGPT</a></span></span></span><span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>BabyAGI:<a target="_blank" href="https://github.com/yoheinakajima/babyagi" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/yoheinakajima/babyagi</a></span></span></span>. For LLMs to serve as autonomous agents, they must perform accurately on two fundamental capabilities: (a) reasoning and planning, and (b) function calling, which includes identifying, calling, and interacting with tools and APIs in external environments. In this work, we focus on improving LLMs’ function calling abilities.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Function calling provides a means for language models to leverage external tools and resources.
These tools can make available to an LLM specific, up-to-date information that would otherwise be inaccessible (e.g., stored in a dynamic knowledge base) and thus reduce its proclivity for hallucinating responses <cite class="ltx_cite ltx_citemacro_citep">(Schick et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite>. This is particularly crucial in enterprise use cases where a significant portion of relevant data is stored in a structured format accessible only via storage engines. In addition to knowledge access, function calling can allow an LLM to outsource tasks that are out of scope for a generalized language model. Most commonly, these tasks involve compute-heavy operations, e.g., program execution <cite class="ltx_cite ltx_citemacro_citep">(Shinn et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite>, numerical calculation, or retrieval <cite class="ltx_cite ltx_citemacro_citep">(Schick et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite>, and are otherwise a frequent source of LLM hallucinations <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib17" title="" class="ltx_ref">2023a</a>)</cite>.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">The importance of function calling has spurred the development of several recent
data generation efforts for fine-tuning <cite class="ltx_cite ltx_citemacro_citep">(Basu et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>; Guo et al., <a href="#bib.bib11" title="" class="ltx_ref">2024</a>; Qin et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>; Yan et al., <a href="#bib.bib46" title="" class="ltx_ref">2024</a>; Tang et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite> and evaluation of models <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib19" title="" class="ltx_ref">2023b</a>; Muennighoff et al., <a href="#bib.bib24" title="" class="ltx_ref">2023</a>)</cite>. Typically, however, the fine-tuned models from datasets like ToolLLM <cite class="ltx_cite ltx_citemacro_citep">(Qin et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite>, ToolAlpaca <cite class="ltx_cite ltx_citemacro_citep">(Tang et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite>, and Gorilla <cite class="ltx_cite ltx_citemacro_citep">(Patil et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>)</cite> underperform in one (or more) of three key dimensions: (a) <span id="S1.p3.1.1" class="ltx_text ltx_font_bold">Generalizability:</span> While the datasets are generated using diverse sets of APIs (e.g., ToolLLama uses RapidAPIs <span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span><a target="_blank" href="https://rapidapi.com/hub" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://rapidapi.com/hub</a></span></span></span>, ToolAlpaca uses public APIs<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a target="_blank" href="https://github.com/public-apis/public-apis" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/public-apis/public-apis</a></span></span></span>, and Gorilla uses TensorFlow Hub, PyTorch Hub, and Hugging Face Hub), work from <cite class="ltx_cite ltx_citemacro_citep">(Basu et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite> has shown that models trained on these datasets have difficulty generalizing to out-of-domain datasets. (b) <span id="S1.p3.1.2" class="ltx_text ltx_font_bold">Granular tasks:</span> Function calling, as an umbrella term, can encompass multiple granular sub-tasks such as function-name detection, slot filling<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span><span id="footnote6.1" class="ltx_text ltx_font_italic">Slot</span>, <span id="footnote6.2" class="ltx_text ltx_font_italic">parameter</span>, and <span id="footnote6.3" class="ltx_text ltx_font_italic">argument</span> are used interchangeably.</span></span></span> or parameter-value pair detection, and detecting the ordered sequence of functions needed to be called. Existing models trained to perform function calling lack the ability to handle these granular tasks independently, and hence, perform poorly on such sub-tasks. (c) <span id="S1.p3.1.3" class="ltx_text ltx_font_bold">Openness:</span> The best performing models are proprietary and the ones that have open licenses (e.g., Gorilla <cite class="ltx_cite ltx_citemacro_citep">(Patil et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>)</cite>) are trained using data generated from OpenAI models.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.00121/assets/x1.png" id="S1.F1.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="133" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Step-by-step building process of <span id="S1.F1.3.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span></figcaption>
</figure>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">To address the aforementioned limitations, in this work, we focus on introducing function-calling abilities to models with an inherent focus on granular tasks. Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows an overview of how <span id="S1.p4.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> was trained. The datasets used for training are API-Blend <cite class="ltx_cite ltx_citemacro_citep">(Basu et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite> that include tasks such as function name detection, slot filling, parallel functions, multiple functions, sequencing<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span><span id="footnote7.1" class="ltx_text ltx_font_italic">Sequencing</span> and <span id="footnote7.2" class="ltx_text ltx_font_italic">chaining</span> are used interchangeably.</span></span></span>, and calling APIs<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span> <span id="footnote8.1" class="ltx_text ltx_font_italic">Function</span> and <span id="footnote8.2" class="ltx_text ltx_font_italic">API</span> are used interchangeably.</span></span></span> using multiple programming languages.
We build upon Granite code models by instruction tuning them for function calling using the datasets for granular tasks with a multi-task learning approach. Granite code models are trained on data that is license permissible following IBM’s AI Ethics principles for trustworthy enterprise usage <cite class="ltx_cite ltx_citemacro_citep">(Mishra et al., <a href="#bib.bib22" title="" class="ltx_ref">2024</a>)</cite>. Being part of the Granite family, we release <span id="S1.p4.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> under Apache 2.0 license. Finally, in this work, we perform a comprehensive evaluation of the open and proprietary models using Berkeley Function Calling Leaderboard (BFCL), four Function Calling Academic Benchmarks, and Response Generation Benchmark from API-Bank <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib19" title="" class="ltx_ref">2023b</a>)</cite> to evaluate the generalizability of function-calling models. <span id="S1.p4.1.3" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> is on par with the best open model on BFCL and fourth overall. Furthermore, compared to other models based on the out-of-domain datasets, <span id="S1.p4.1.4" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> shows significant generalizability.
Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> shows how <span id="S1.p4.1.5" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> compares to the top two open models (according to BFCL) on various tasks where despite only having 20B parameters, it performs as well or better than Meta-Llama-3-70B-Instruct which has 70B parameters.</p>
</div>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2407.00121/assets/x2.png" id="S1.F2.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="380" height="225" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Evaluation of <span id="S1.F2.4.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> against the best <span id="S1.F2.5.2" class="ltx_text ltx_font_italic">open</span> function calling models (according to BFCL)</figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Instruction Tuning</h3>

<div id="S2.SS1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.p1.1" class="ltx_p">Our work is an instantiation of <span id="S2.SS1.p1.1.1" class="ltx_text ltx_font_italic">instruction tuning</span> <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib43" title="" class="ltx_ref">2021</a>)</cite>, a fine-tuning method that improves an LLM’s ability to solve natural language tasks <cite class="ltx_cite ltx_citemacro_citep">(Mishra et al., <a href="#bib.bib23" title="" class="ltx_ref">2022</a>; Wang et al., <a href="#bib.bib42" title="" class="ltx_ref">2023</a>)</cite>. It involves taking a large collection of NLP datasets, reformulating those datasets into a set of instruction-following tasks, and then fine-tuning an LLM on the modified data. While the earliest versions of instruction tuning straightforwardly combined large datasets together, the most recent iterations use more sophisticated mixtures of tasks to achieve the best results <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib18" title="" class="ltx_ref">2024</a>; Sudalairaj et al., <a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite>. Our work draws largely upon API-Blend <cite class="ltx_cite ltx_citemacro_citep">(Basu et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite> and API Pack <cite class="ltx_cite ltx_citemacro_citep">(Guo et al., <a href="#bib.bib11" title="" class="ltx_ref">2024</a>)</cite>, two recently introduced instruction-tuning datasets specifically focused on tasks related to APIs, e.g., slot filling and API intent detection.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para ltx_noindent">
<p id="S2.SS1.p2.1" class="ltx_p">Instruction-tuned models often significantly outperform their base models on a wide range of tasks, particularly in the zero-shot setting <cite class="ltx_cite ltx_citemacro_citep">(Ouyang et al., <a href="#bib.bib27" title="" class="ltx_ref">2022</a>; Muennighoff et al., <a href="#bib.bib24" title="" class="ltx_ref">2023</a>; Chung et al., <a href="#bib.bib5" title="" class="ltx_ref">2022</a>)</cite>. Further improvement has been observed through alignment of the model after instruction-fine-tuning, e.g., TP-LLaMA <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib3" title="" class="ltx_ref">2024a</a>)</cite> uses Direct Preference Optimization <cite class="ltx_cite ltx_citemacro_citep">(Rafailov et al., <a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite> in addition to fine-tuning.</p>
</div>
<div id="S2.SS1.p3" class="ltx_para ltx_noindent">
<p id="S2.SS1.p3.1" class="ltx_p">There is also growing interest in developing smaller models that match or surpass the accuracy of larger proprietary LLMs in function calling tasks <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib4" title="" class="ltx_ref">2024b</a>)</cite>. These compact models are crucial in the increasing prevalence of on-device LLMs, enabling efficient and effective performance on local devices.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Function Calling by LLMs</h3>

<div id="S2.SS2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS2.p1.1" class="ltx_p">Function calling augmentation has broadened the scope of problems addressable by LLMs to include those that cannot be solved with internal knowledge alone. For instance, prior work has demonstrated the use of API-enhanced LLMs to solve problems requiring up-to-date information retrieval <cite class="ltx_cite ltx_citemacro_citep">(Schick et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>)</cite>, intricate mathematical calculations <cite class="ltx_cite ltx_citemacro_citep">(He-Yueya et al., <a href="#bib.bib13" title="" class="ltx_ref">2023</a>; Patel et al., <a href="#bib.bib30" title="" class="ltx_ref">2021</a>)</cite>, internet use <cite class="ltx_cite ltx_citemacro_citep">(Komeili et al., <a href="#bib.bib16" title="" class="ltx_ref">2022</a>; Gur et al., <a href="#bib.bib12" title="" class="ltx_ref">2023</a>)</cite>, task orchestration <cite class="ltx_cite ltx_citemacro_citep">(Jain et al., <a href="#bib.bib14" title="" class="ltx_ref">2024</a>)</cite>, and even programming abilities <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para ltx_noindent">
<p id="S2.SS2.p2.1" class="ltx_p">Multiple strategies have been proposed for how best to enable LLM function calling. One line of prior research has investigated the design of elaborate prompting approaches, best exemplified by the popular ReACT prompting framework <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib50" title="" class="ltx_ref">2022</a>)</cite>. Such prompting methods can vary in their design, with some works optimizing for cost <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib44" title="" class="ltx_ref">2023a</a>)</cite>, raw performance <cite class="ltx_cite ltx_citemacro_citep">(Shinn et al., <a href="#bib.bib37" title="" class="ltx_ref">2023</a>; Yang et al., <a href="#bib.bib49" title="" class="ltx_ref">2023</a>)</cite>, or a blend of both <cite class="ltx_cite ltx_citemacro_citep">(Crouse et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite>. More relevant to our approach are methods that train models to directly output function calls <cite class="ltx_cite ltx_citemacro_citep">(Tang et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>; Qin et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite>. Typically, these works will use some form of self-supervision to enable scaling to the breadth of domains required for general-purpose function use <cite class="ltx_cite ltx_citemacro_citep">(Schick et al., <a href="#bib.bib36" title="" class="ltx_ref">2023</a>; Parisi et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>; Yang et al., <a href="#bib.bib48" title="" class="ltx_ref">2024b</a>)</cite>.</p>
</div>
<div id="S2.SS2.p3" class="ltx_para ltx_noindent">
<p id="S2.SS2.p3.1" class="ltx_p">Recently, many language models with function-calling capabilities have been introduced. They broadly fall into two categories: pre-trained models which are capable of function-calling <cite class="ltx_cite ltx_citemacro_citep">(Reid et al., <a href="#bib.bib34" title="" class="ltx_ref">2024</a>; CodeGemma Team et al., <a href="#bib.bib6" title="" class="ltx_ref">2024</a>; CohereForAI, <a href="#bib.bib7" title="" class="ltx_ref">2024</a>; AI@Meta, <a href="#bib.bib1" title="" class="ltx_ref">2024</a>; Jiang et al., <a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>, and models fine-tuned specifically for function-calling <cite class="ltx_cite ltx_citemacro_citep">(Qin et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>; Tang et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>; MeetKai, <a href="#bib.bib21" title="" class="ltx_ref">2024</a>; Patil et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>; Nous-Research, <a href="#bib.bib26" title="" class="ltx_ref">2023</a>; Nexusflow.ai, <a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite>. While the pre-trained models enable function-calling using a combination of supervised and preference fine-tuning, details of the datasets used to train models for these tasks are not generally available. On the other hand, specialized function-calling models mostly rely on synthetic data generated from proprietary state-of-the-art models. Models like Gorilla <cite class="ltx_cite ltx_citemacro_citep">(Patil et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>)</cite>, ToolLlama <cite class="ltx_cite ltx_citemacro_citep">(Qin et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite>, ToolAlpaca <cite class="ltx_cite ltx_citemacro_citep">(Tang et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite>, and the NousResearch Hermes series of models <cite class="ltx_cite ltx_citemacro_citep">(Nous-Research, <a href="#bib.bib26" title="" class="ltx_ref">2023</a>)</cite> utilize GPT-4 or ChatGPT to generate synthetic instruction tuning datasets and fine-tune a base model such as the Llama or Mistral model for function-calling tasks. The NexusRaven models <cite class="ltx_cite ltx_citemacro_citep">(Nexusflow.ai, <a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite> are one of the few open-source models that focus on building function-calling models for commercial purposes by avoiding using proprietary models for synthetic data generation.</p>
</div>
<div id="S2.SS2.p4" class="ltx_para ltx_noindent">
<p id="S2.SS2.p4.1" class="ltx_p">In section <a href="#S5" title="5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, we compare our model to the above models and show that <span id="S2.SS2.p4.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> provides the best or comparable performance amongst all open models across multiple tasks.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Multi-Task Training Data</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.1" class="ltx_p">In this section, we describe our detailed approach to fine-tune the
<span id="S3.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-Code-Instruct</span> model with multi-task data related to functions to build <span id="S3.p1.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span>, a robust model designed for function-calling. We use API-BLEND <cite class="ltx_cite ltx_citemacro_citep">(Basu et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite>, a diverse corpora of multiple API datasets for training LLMs. It consists of five datasets with a total of about 160K training examples: SeqSGD, SeqSNIPS, SeqTopV2, SeqATIS, and SeqMultiWOZ.</p>
</div>
<div id="S3.p2" class="ltx_para ltx_noindent">
<p id="S3.p2.1" class="ltx_p">A key contribution to the process of building <span id="S3.p2.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> is multi-task training, where we reuse the same data in different formats with distinct instructions for different function-calling related tasks. We have identified six underlying sub-tasks for function calling and divided them into two broad categories based on the difficulty levels: (A) <span id="S3.p2.1.2" class="ltx_text ltx_font_italic">High-Level Function Calling Tasks</span> which are complex tasks for an LLM and typically handle multiple functions; and (B) <span id="S3.p2.1.3" class="ltx_text ltx_font_italic">Low-Level Function Calling Tasks</span> which are simpler tasks for an LLM and relate to either function names or only parameter-value pairs. We have included <span id="S3.p2.1.4" class="ltx_text ltx_font_italic">“Response Generation”</span> as the seventh task in our training data since producing natural language responses is one of the fundamental goals of an LLM. Table <a href="#S3.T1" title="Table 1 ‣ 3 Multi-Task Training Data ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> demonstrates the task-wise mapping of each dataset. Below, we describe each task in detail.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<table id="S3.T1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T1.1.1.1" class="ltx_tr">
<td id="S3.T1.1.1.1.1" class="ltx_td ltx_border_r ltx_border_tt"></td>
<td id="S3.T1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="3"><span id="S3.T1.1.1.1.2.1" class="ltx_text ltx_font_bold">High-Level Function Calling Tasks</span></td>
<td id="S3.T1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" colspan="3"><span id="S3.T1.1.1.1.3.1" class="ltx_text ltx_font_bold">Low-Level Function Calling Tasks</span></td>
<td id="S3.T1.1.1.1.4" class="ltx_td ltx_border_tt"></td>
</tr>
<tr id="S3.T1.1.2.2" class="ltx_tr">
<td id="S3.T1.1.2.2.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><span id="S3.T1.1.2.2.1.1" class="ltx_text ltx_font_bold">Datasets</span></td>
<td id="S3.T1.1.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.2.2.2.1" class="ltx_text" style="color:#9A0000;">
<span id="S3.T1.1.2.2.2.1.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.2.2.2.1.1.1" class="ltx_tr">
<span id="S3.T1.1.2.2.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.2.1.1.1.1.1" class="ltx_text ltx_font_bold">Nested Func.</span></span></span>
<span id="S3.T1.1.2.2.2.1.1.2" class="ltx_tr">
<span id="S3.T1.1.2.2.2.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.2.1.1.2.1.1" class="ltx_text ltx_font_bold">Calling</span></span></span>
</span></span></td>
<td id="S3.T1.1.2.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.2.2.3.1" class="ltx_text" style="color:#9A0000;">
<span id="S3.T1.1.2.2.3.1.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.2.2.3.1.1.1" class="ltx_tr">
<span id="S3.T1.1.2.2.3.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.3.1.1.1.1.1" class="ltx_text ltx_font_bold">Func.</span></span></span>
<span id="S3.T1.1.2.2.3.1.1.2" class="ltx_tr">
<span id="S3.T1.1.2.2.3.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.3.1.1.2.1.1" class="ltx_text ltx_font_bold">Chaining</span></span></span>
</span></span></td>
<td id="S3.T1.1.2.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T1.1.2.2.4.1" class="ltx_text" style="color:#9A0000;">
<span id="S3.T1.1.2.2.4.1.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.2.2.4.1.1.1" class="ltx_tr">
<span id="S3.T1.1.2.2.4.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.4.1.1.1.1.1" class="ltx_text ltx_font_bold">Parallel</span></span></span>
<span id="S3.T1.1.2.2.4.1.1.2" class="ltx_tr">
<span id="S3.T1.1.2.2.4.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.4.1.1.2.1.1" class="ltx_text ltx_font_bold">Func.</span></span></span>
</span></span></td>
<td id="S3.T1.1.2.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.2.2.5.1" class="ltx_text" style="color:#303498;">
<span id="S3.T1.1.2.2.5.1.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.2.2.5.1.1.1" class="ltx_tr">
<span id="S3.T1.1.2.2.5.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.5.1.1.1.1.1" class="ltx_text ltx_font_bold">Next-Best</span></span></span>
<span id="S3.T1.1.2.2.5.1.1.2" class="ltx_tr">
<span id="S3.T1.1.2.2.5.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.5.1.1.2.1.1" class="ltx_text ltx_font_bold">Func.</span></span></span>
</span></span></td>
<td id="S3.T1.1.2.2.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.2.2.6.1" class="ltx_text" style="color:#303498;">
<span id="S3.T1.1.2.2.6.1.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.2.2.6.1.1.1" class="ltx_tr">
<span id="S3.T1.1.2.2.6.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.6.1.1.1.1.1" class="ltx_text ltx_font_bold">Func. Name</span></span></span>
<span id="S3.T1.1.2.2.6.1.1.2" class="ltx_tr">
<span id="S3.T1.1.2.2.6.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.6.1.1.2.1.1" class="ltx_text ltx_font_bold">Detection</span></span></span>
</span></span></td>
<td id="S3.T1.1.2.2.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T1.1.2.2.7.1" class="ltx_text" style="color:#303498;">
<span id="S3.T1.1.2.2.7.1.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.2.2.7.1.1.1" class="ltx_tr">
<span id="S3.T1.1.2.2.7.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.7.1.1.1.1.1" class="ltx_text ltx_font_bold">Param-Val</span></span></span>
<span id="S3.T1.1.2.2.7.1.1.2" class="ltx_tr">
<span id="S3.T1.1.2.2.7.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.7.1.1.2.1.1" class="ltx_text ltx_font_bold">Pair Detection</span></span></span>
</span></span></td>
<td id="S3.T1.1.2.2.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.2.2.8.1" class="ltx_text">
<span id="S3.T1.1.2.2.8.1.1" class="ltx_tabular ltx_align_middle">
<span id="S3.T1.1.2.2.8.1.1.1" class="ltx_tr">
<span id="S3.T1.1.2.2.8.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.8.1.1.1.1.1" class="ltx_text ltx_font_bold">Response</span></span></span>
<span id="S3.T1.1.2.2.8.1.1.2" class="ltx_tr">
<span id="S3.T1.1.2.2.8.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S3.T1.1.2.2.8.1.1.2.1.1" class="ltx_text ltx_font_bold">Generation</span></span></span>
</span></span></td>
</tr>
<tr id="S3.T1.1.3.3" class="ltx_tr">
<td id="S3.T1.1.3.3.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><span id="S3.T1.1.3.3.1.1" class="ltx_text" style="color:#343434;"><span id="S3.T1.1.3.3.1.1.1" class="ltx_text ltx_font_bold">SeqSGD<span class="ltx_rule" style="width:0.0pt;height:11.2pt;background:black;display:inline-block;"></span></span></span></td>
<td id="S3.T1.1.3.3.2" class="ltx_td ltx_border_t"></td>
<td id="S3.T1.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.3.3.3.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.3.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T1.1.3.3.4.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.3.3.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.3.3.5.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.3.3.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.3.3.6.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.3.3.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S3.T1.1.3.3.7.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.3.3.8" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="S3.T1.1.4.4" class="ltx_tr">
<td id="S3.T1.1.4.4.1" class="ltx_td ltx_align_left ltx_border_r"><span id="S3.T1.1.4.4.1.1" class="ltx_text" style="color:#343434;"><span id="S3.T1.1.4.4.1.1.1" class="ltx_text ltx_font_bold">SeqSNIPS</span></span></td>
<td id="S3.T1.1.4.4.2" class="ltx_td"></td>
<td id="S3.T1.1.4.4.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.4.4.3.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.4.4.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.4.4.4.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.4.4.5" class="ltx_td ltx_align_center"><span id="S3.T1.1.4.4.5.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.4.4.6" class="ltx_td ltx_align_center"><span id="S3.T1.1.4.4.6.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.4.4.7" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.4.4.7.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.4.4.8" class="ltx_td"></td>
</tr>
<tr id="S3.T1.1.5.5" class="ltx_tr">
<td id="S3.T1.1.5.5.1" class="ltx_td ltx_align_left ltx_border_r"><span id="S3.T1.1.5.5.1.1" class="ltx_text" style="color:#343434;"><span id="S3.T1.1.5.5.1.1.1" class="ltx_text ltx_font_bold">SeqTopV2</span></span></td>
<td id="S3.T1.1.5.5.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.5.5.2.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.5.5.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.5.5.3.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.5.5.4" class="ltx_td ltx_border_r"></td>
<td id="S3.T1.1.5.5.5" class="ltx_td ltx_align_center"><span id="S3.T1.1.5.5.5.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.5.5.6" class="ltx_td ltx_align_center"><span id="S3.T1.1.5.5.6.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.5.5.7" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.5.5.7.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.5.5.8" class="ltx_td"></td>
</tr>
<tr id="S3.T1.1.6.6" class="ltx_tr">
<td id="S3.T1.1.6.6.1" class="ltx_td ltx_align_left ltx_border_r"><span id="S3.T1.1.6.6.1.1" class="ltx_text" style="color:#343434;"><span id="S3.T1.1.6.6.1.1.1" class="ltx_text ltx_font_bold">SeqATIS</span></span></td>
<td id="S3.T1.1.6.6.2" class="ltx_td"></td>
<td id="S3.T1.1.6.6.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.6.6.3.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.6.6.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.6.6.4.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.6.6.5" class="ltx_td ltx_align_center"><span id="S3.T1.1.6.6.5.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.6.6.6" class="ltx_td ltx_align_center"><span id="S3.T1.1.6.6.6.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.6.6.7" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.6.6.7.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.6.6.8" class="ltx_td"></td>
</tr>
<tr id="S3.T1.1.7.7" class="ltx_tr">
<td id="S3.T1.1.7.7.1" class="ltx_td ltx_align_left ltx_border_r"><span id="S3.T1.1.7.7.1.1" class="ltx_text" style="color:#343434;"><span id="S3.T1.1.7.7.1.1.1" class="ltx_text ltx_font_bold">SeqMultiWOZ</span></span></td>
<td id="S3.T1.1.7.7.2" class="ltx_td"></td>
<td id="S3.T1.1.7.7.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.7.7.3.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.7.7.4" class="ltx_td ltx_border_r"></td>
<td id="S3.T1.1.7.7.5" class="ltx_td ltx_align_center"><span id="S3.T1.1.7.7.5.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.7.7.6" class="ltx_td ltx_align_center"><span id="S3.T1.1.7.7.6.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.7.7.7" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T1.1.7.7.7.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.7.7.8" class="ltx_td"></td>
</tr>
<tr id="S3.T1.1.8.8" class="ltx_tr">
<td id="S3.T1.1.8.8.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r"><span id="S3.T1.1.8.8.1.1" class="ltx_text" style="color:#343434;"><span id="S3.T1.1.8.8.1.1.1" class="ltx_text ltx_font_bold">Glaive-V2</span></span></td>
<td id="S3.T1.1.8.8.2" class="ltx_td ltx_border_bb"></td>
<td id="S3.T1.1.8.8.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T1.1.8.8.3.1" class="ltx_text" style="color:#00E000;">✔</span></td>
<td id="S3.T1.1.8.8.4" class="ltx_td ltx_border_bb ltx_border_r"></td>
<td id="S3.T1.1.8.8.5" class="ltx_td ltx_border_bb"></td>
<td id="S3.T1.1.8.8.6" class="ltx_td ltx_border_bb"></td>
<td id="S3.T1.1.8.8.7" class="ltx_td ltx_border_bb ltx_border_r"></td>
<td id="S3.T1.1.8.8.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T1.1.8.8.8.1" class="ltx_text" style="color:#00E000;">✔</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 1: </span>Training Datasets with Task mapping</figcaption>
</figure>
<div id="S3.p3" class="ltx_para ltx_noindent">
<p id="S3.p3.1" class="ltx_p">In the rest of the section, we describe how to unify the data of different datasets in the same format for model training and then describe each of these training tasks.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Data Unification</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.1" class="ltx_p">In addition to the datasets in API-BLEND <cite class="ltx_cite ltx_citemacro_citep">(Basu et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite> (i.e., SeqSGD, SeqSNIPS, SeqTopV2, SeqATIS, SeqMultiWOZ), we also use Glaive-V2<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span><a target="_blank" href="https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2</a></span></span></span> to prepare the training data mixture for <span id="S3.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span>, where each dataset is multi-purposed for different function calling related tasks with different instructions. These datasets come from different sources and have various function formats that require unification to an identical representation for better usability.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.1" class="ltx_p">In this unification process, we convert all the APIs, tools, and functions from the data into a JSON format representation. We choose this format because (a) JSON is a language-independent, human-readable, and widely used data format for code-related tasks; (b) it is easily parsable to insert/extract information; and (c) many web services, APIs, and tools accept JSON objects and generate responses in JSON format.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.1" class="ltx_p">In <span id="S3.SS1.p3.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span>, we unify the model <span id="S3.SS1.p3.1.2" class="ltx_text ltx_font_italic">output</span> representation of function calls to the following format:</p>
</div>
<div id="S3.SS1.p4" class="ltx_para ltx_noindent">
<div id="S3.SS1.p4.1" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogIm5hbWUiOiAiPEZVTkNUSU9OLU5BTUU+IiwKICJhcmd1bWVudHMiOiB7IjxQQVJBTUVURVItMT4iOiAiVkFMVUUtMSIsICI8UEFSQU1FVEVSLTI+IjogLi4ufQp9" download="">⬇</a></div>
<div id="lstnumberx1" class="ltx_listingline">
<span id="lstnumberx1.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx2" class="ltx_listingline">
<span id="lstnumberx2.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx2.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx2.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx2.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx2.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.7" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx2.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">FUNCTION</span><span id="lstnumberx2.9" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx2.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">NAME</span><span id="lstnumberx2.11" class="ltx_text ltx_font_typewriter">&gt;"</span><span id="lstnumberx2.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx3" class="ltx_listingline">
<span id="lstnumberx3.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx3.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx3.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx3.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx3.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.7" class="ltx_text ltx_font_typewriter">{"&lt;</span><span id="lstnumberx3.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">PARAMETER</span><span id="lstnumberx3.9" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx3.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx3.11" class="ltx_text ltx_font_typewriter">&gt;"</span><span id="lstnumberx3.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx3.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.14" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx3.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">VALUE</span><span id="lstnumberx3.16" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx3.17" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx3.18" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx3.19" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx3.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.21" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx3.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">PARAMETER</span><span id="lstnumberx3.23" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx3.24" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">2</span><span id="lstnumberx3.25" class="ltx_text ltx_font_typewriter">&gt;"</span><span id="lstnumberx3.26" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx3.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.28" class="ltx_text ltx_font_typewriter">...}</span>
</div>
<div id="lstnumberx4" class="ltx_listingline">
<span id="lstnumberx4.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
</div>
<div id="S3.SS1.p5" class="ltx_para ltx_noindent">
<p id="S3.SS1.p5.1" class="ltx_p">With a similar JSON representation, we express the functions in the library to be passed as input to the model:</p>
<div id="S3.SS1.p5.2" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogIm5hbWUiOiAiPEZVTkNUSU9OLU5BTUU+IiwKICJkZXNjcmlwdGlvbjogIjxGVU5DVElPTi1ERVNDUklQVElPTj4iCiAiYXJndW1lbnRzIjogeyI8UEFSQU1FVEVSLTE+IjogeyJkZXNjcmlwdGlvbiI6ICIuLi4iLCAuLi59IC4uLiB9Cn0=" download="">⬇</a></div>
<div id="lstnumberx5" class="ltx_listingline">
<span id="lstnumberx5.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx6" class="ltx_listingline">
<span id="lstnumberx6.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx6.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx6.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx6.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx6.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx6.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx6.7" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx6.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">FUNCTION</span><span id="lstnumberx6.9" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx6.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">NAME</span><span id="lstnumberx6.11" class="ltx_text ltx_font_typewriter">&gt;"</span><span id="lstnumberx6.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx7" class="ltx_listingline">
<span id="lstnumberx7.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx7.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx7.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">description</span><span id="lstnumberx7.4" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx7.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx7.6" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx7.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">FUNCTION</span><span id="lstnumberx7.8" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx7.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">DESCRIPTION</span><span id="lstnumberx7.10" class="ltx_text ltx_font_typewriter">&gt;"</span>
</div>
<div id="lstnumberx8" class="ltx_listingline">
<span id="lstnumberx8.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx8.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx8.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx8.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx8.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.7" class="ltx_text ltx_font_typewriter">{"&lt;</span><span id="lstnumberx8.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">PARAMETER</span><span id="lstnumberx8.9" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx8.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx8.11" class="ltx_text ltx_font_typewriter">&gt;"</span><span id="lstnumberx8.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx8.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.14" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx8.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">description</span><span id="lstnumberx8.16" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx8.17" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx8.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.19" class="ltx_text ltx_font_typewriter">"..."</span><span id="lstnumberx8.20" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx8.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.22" class="ltx_text ltx_font_typewriter">...}</span><span id="lstnumberx8.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.24" class="ltx_text ltx_font_typewriter">...</span><span id="lstnumberx8.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.26" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx9" class="ltx_listingline">
<span id="lstnumberx9.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>High-Level Function Calling Tasks</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.1" class="ltx_p">In general, these tasks are challenging for LLMs to accomplish since they require the LLM to generate multiple function calls with the parameters and their values. In the following sections, we describe different types of high-level function-calling tasks with examples.</p>
</div>
<section id="S3.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Nested Function Calling</h5>

<div id="S3.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS0.Px1.p1.1" class="ltx_p">The main characteristic of this task is in the output function sequence, where the current function’s output becomes an input to the next function. So, the answer to a user query is a sequence of nested function calls selected from the function library. Furthermore, the parameters of these function calls need to be filled by extracting the values from the user query. The example below lists two functions in the output under <span id="S3.SS2.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_typewriter">&lt;function_call&gt;</span> tag <span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span>In all of our examples, we use <span id="footnote10.1" class="ltx_text ltx_font_typewriter">&lt;function_call&gt;</span> tag to represent a function in JSON form.</span></span></span> and the output of the first function will populate the <span id="S3.SS2.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_typewriter">destination</span> parameter in the second function.</p>
</div>
<div id="S3.SS2.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<div id="S3.SS2.SSS0.Px1.p2.1" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogImlucHV0IjogIldoYXQncyB0aGUgdHlwaWNhbCBkcml2aW5nIHRpbWUgYmV0d2VlbiBMYXMgVmVnYXMgYW5kIHRoZSBHcmFuZCBDYW55b24/IiwKICJvdXRwdXQ6ICI8ZnVuY3Rpb25fY2FsbD4gewogICAgICAgICAgICAgIm5hbWUiOiAiZ2V0X2xvY2F0aW9uIiwKICAgICAgICAgICAgICJhcmd1bWVudHMiOgogICAgICAgICAgICAgICAgeyJwb2ludF9vbl9tYXAiOiAidGhlIEdyYW5kIENhbnlvbiJ9CiAgICAgICAgICAgICB9CiAgICAgICAgICAgPGZ1bmN0aW9uX2NhbGw+IHsKICAgICAgICAgICAgICJuYW1lIjogImdldF9lc3RpbWF0ZWRfZHVyYXRpb24iLAogICAgICAgICAgICAgImFyZ3VtZW50cyI6CiAgICAgICAgICAgICAgICB7InNvdXJjZSI6ICJMYXMgVmVnYXMiLAogICAgICAgICAgICAgICAgICJtZXRob2RfdHJhdmVsIjogImRyaXZpbmciLAogICAgICAgICAgICAgICAgICJkZXN0aW5hdGlvbiI6ICI8ZnVuY3Rpb25fcmVzcG9uc2U+Z2V0X2xvY2F0aW9uIn0KICAgICAgICAgICAgIH0iCn0=" download="">⬇</a></div>
<div id="lstnumberx10" class="ltx_listingline">
<span id="lstnumberx10.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx11" class="ltx_listingline">
<span id="lstnumberx11.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx11.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">input</span><span id="lstnumberx11.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx11.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx11.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx11.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">What</span><span id="lstnumberx11.9" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx11.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx11.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx11.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">typical</span><span id="lstnumberx11.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">driving</span><span id="lstnumberx11.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">time</span><span id="lstnumberx11.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">between</span><span id="lstnumberx11.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Las</span><span id="lstnumberx11.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Vegas</span><span id="lstnumberx11.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx11.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx11.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Grand</span><span id="lstnumberx11.31" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.32" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Canyon</span><span id="lstnumberx11.33" class="ltx_text ltx_font_typewriter">?"</span><span id="lstnumberx11.34" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx12" class="ltx_listingline">
<span id="lstnumberx12.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx12.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx12.4" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx12.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.6" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx12.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx12.8" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx12.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.10" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx13" class="ltx_listingline">
<span id="lstnumberx13.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx13.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx13.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx13.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx13.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx13.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx13.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">get_location</span><span id="lstnumberx13.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx13.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx14" class="ltx_listingline">
<span id="lstnumberx14.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx14.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx14.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx14.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx14.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</div>
<div id="lstnumberx15" class="ltx_listingline">
<span id="lstnumberx15.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                </span><span id="lstnumberx15.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx15.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">point_on_map</span><span id="lstnumberx15.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx15.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx15.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx15.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx15.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx15.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx15.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Grand</span><span id="lstnumberx15.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx15.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Canyon</span><span id="lstnumberx15.13" class="ltx_text ltx_font_typewriter">"}</span>
</div>
<div id="lstnumberx16" class="ltx_listingline">
<span id="lstnumberx16.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx16.2" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx17" class="ltx_listingline">
<span id="lstnumberx17.1" class="ltx_text ltx_lst_space ltx_font_typewriter">           </span><span id="lstnumberx17.2" class="ltx_text ltx_font_typewriter">&lt;</span><span id="lstnumberx17.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx17.4" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx17.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx17.6" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx18" class="ltx_listingline">
<span id="lstnumberx18.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx18.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx18.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx18.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx18.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx18.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx18.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">get_estimated_duration</span><span id="lstnumberx18.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx18.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx19" class="ltx_listingline">
<span id="lstnumberx19.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx19.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx19.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx19.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx19.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</div>
<div id="lstnumberx20" class="ltx_listingline">
<span id="lstnumberx20.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                </span><span id="lstnumberx20.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx20.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">source</span><span id="lstnumberx20.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx20.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx20.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx20.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx20.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Las</span><span id="lstnumberx20.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx20.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Vegas</span><span id="lstnumberx20.11" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx20.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx21" class="ltx_listingline">
<span id="lstnumberx21.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx21.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx21.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">method_travel</span><span id="lstnumberx21.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx21.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx21.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx21.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx21.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">driving</span><span id="lstnumberx21.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx21.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx22" class="ltx_listingline">
<span id="lstnumberx22.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx22.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx22.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">destination</span><span id="lstnumberx22.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx22.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx22.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx22.7" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx22.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_response</span><span id="lstnumberx22.9" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx22.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">get_location</span><span id="lstnumberx22.11" class="ltx_text ltx_font_typewriter">"}</span>
</div>
<div id="lstnumberx23" class="ltx_listingline">
<span id="lstnumberx23.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx23.2" class="ltx_text ltx_font_typewriter">}"</span>
</div>
<div id="lstnumberx24" class="ltx_listingline">
<span id="lstnumberx24.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
</div>
</section>
<section id="S3.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Function Chaining</h5>

<div id="S3.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS0.Px2.p1.1" class="ltx_p">In this task, a model needs to call multiple functions in a sequence to answer a user query. However, unlike Nested Function Calling, these functions do not have to be nested. Also, for each function, the parameters whose values are present in the user query must be passed as arguments. The following example demonstrates how three functions must be called in sequence to answer the query.</p>
</div>
<div id="S3.SS2.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<div id="S3.SS2.SSS0.Px2.p2.1" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogImlucHV0IjogIkZpbmQgYSBwcm92aWRlciBpbiBTRi4gR2V0IHRoZSB3ZWF0aGVyIGluIFNGIG9uIE1hcmNoIDNyZC4gQm9vayBhbiBhcHBvaW50bWVudCB3aXRoIEEgYnVzaW5lc3NtYW4ncyBoYWlyY3V0IG9uIE1hcmNoIDNyZCBhdCAxODoxNS4iLAogIm91dHB1dDogIjxmdW5jdGlvbl9jYWxsPiB7CiAgICAgICAgICAgICAibmFtZSI6ICJmaW5kX3Byb3ZpZGVyIiwKICAgICAgICAgICAgICJhcmd1bWVudHMiOgogICAgICAgICAgICAgICAgIHsiY2l0eSI6ICJTRiJ9CiAgICAgICAgICAgICB9CiAgICAgICAgICAgPGZ1bmN0aW9uX2NhbGw+IHsKICAgICAgICAgICAgICJuYW1lIjogImdldF93ZWF0aGVyIiwKICAgICAgICAgICAgICJhcmd1bWVudHMiOgogICAgICAgICAgICAgICAgIHsiY2l0eSI6ICJTRiIsCiAgICAgICAgICAgICAgICAgICJkYXRlIjogIm1hcmNoIDNyZCJ9CiAgICAgICAgICAgICB9CiAgICAgICAgICAgPGZ1bmN0aW9uX2NhbGw+IHsKICAgICAgICAgICAgICJuYW1lIjogImJvb2tfYXBwb2ludG1lbnQiLAogICAgICAgICAgICAgImFyZ3VtZW50cyI6CiAgICAgICAgICAgICAgICAgeyJhcHBvaW50bWVudF9kYXRlIjogIm1hcmNoIDNyZCIsCiAgICAgICAgICAgICAgICAgICJhcHBvaW50bWVudF90aW1lIjogIjE4OjE1IiwKICAgICAgICAgICAgICAgICAgInN0eWxpc3RfbmFtZSI6ICJBIGJ1c2luZXNzbWFuJ3MgaGFpcmN1dCJ9CiAgICAgICAgICAgICB9Igp9" download="">⬇</a></div>
<div id="lstnumberx25" class="ltx_listingline">
<span id="lstnumberx25.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx26" class="ltx_listingline">
<span id="lstnumberx26.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx26.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">input</span><span id="lstnumberx26.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx26.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx26.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx26.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Find</span><span id="lstnumberx26.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx26.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">provider</span><span id="lstnumberx26.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx26.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SF</span><span id="lstnumberx26.17" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx26.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Get</span><span id="lstnumberx26.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx26.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">weather</span><span id="lstnumberx26.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx26.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SF</span><span id="lstnumberx26.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">on</span><span id="lstnumberx26.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">March</span><span id="lstnumberx26.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.33" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span><span id="lstnumberx26.34" class="ltx_text ltx_lst_identifier ltx_font_typewriter">rd</span><span id="lstnumberx26.35" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx26.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Book</span><span id="lstnumberx26.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.39" class="ltx_text ltx_lst_identifier ltx_font_typewriter">an</span><span id="lstnumberx26.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">appointment</span><span id="lstnumberx26.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx26.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">A</span><span id="lstnumberx26.46" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.47" class="ltx_text ltx_lst_identifier ltx_font_typewriter">businessman</span><span id="lstnumberx26.48" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx26.49" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx26.50" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.51" class="ltx_text ltx_lst_identifier ltx_font_typewriter">haircut</span><span id="lstnumberx26.52" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.53" class="ltx_text ltx_lst_identifier ltx_font_typewriter">on</span><span id="lstnumberx26.54" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.55" class="ltx_text ltx_lst_identifier ltx_font_typewriter">March</span><span id="lstnumberx26.56" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.57" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span><span id="lstnumberx26.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter">rd</span><span id="lstnumberx26.59" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter">at</span><span id="lstnumberx26.61" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx26.62" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx26.63" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">8</span><span id="lstnumberx26.64" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx26.65" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx26.66" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">5</span><span id="lstnumberx26.67" class="ltx_text ltx_font_typewriter">."</span><span id="lstnumberx26.68" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx27" class="ltx_listingline">
<span id="lstnumberx27.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx27.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx27.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx27.4" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx27.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx27.6" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx27.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx27.8" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx27.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx27.10" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx28" class="ltx_listingline">
<span id="lstnumberx28.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx28.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx28.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx28.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx28.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx28.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx28.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx28.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">find_provider</span><span id="lstnumberx28.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx28.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx29" class="ltx_listingline">
<span id="lstnumberx29.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx29.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx29.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx29.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx29.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</div>
<div id="lstnumberx30" class="ltx_listingline">
<span id="lstnumberx30.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx30.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx30.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">city</span><span id="lstnumberx30.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx30.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx30.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx30.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx30.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SF</span><span id="lstnumberx30.9" class="ltx_text ltx_font_typewriter">"}</span>
</div>
<div id="lstnumberx31" class="ltx_listingline">
<span id="lstnumberx31.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx31.2" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx32" class="ltx_listingline">
<span id="lstnumberx32.1" class="ltx_text ltx_lst_space ltx_font_typewriter">           </span><span id="lstnumberx32.2" class="ltx_text ltx_font_typewriter">&lt;</span><span id="lstnumberx32.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx32.4" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx32.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.6" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx33" class="ltx_listingline">
<span id="lstnumberx33.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx33.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx33.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx33.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx33.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx33.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx33.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx33.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">get_weather</span><span id="lstnumberx33.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx33.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx34" class="ltx_listingline">
<span id="lstnumberx34.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx34.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx34.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx34.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx34.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</div>
<div id="lstnumberx35" class="ltx_listingline">
<span id="lstnumberx35.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx35.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx35.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">city</span><span id="lstnumberx35.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx35.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx35.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx35.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx35.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SF</span><span id="lstnumberx35.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx35.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx36" class="ltx_listingline">
<span id="lstnumberx36.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                  </span><span id="lstnumberx36.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx36.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">date</span><span id="lstnumberx36.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx36.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx36.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx36.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">march</span><span id="lstnumberx36.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx36.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span><span id="lstnumberx36.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">rd</span><span id="lstnumberx36.12" class="ltx_text ltx_font_typewriter">"}</span>
</div>
<div id="lstnumberx37" class="ltx_listingline">
<span id="lstnumberx37.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx37.2" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx38" class="ltx_listingline">
<span id="lstnumberx38.1" class="ltx_text ltx_lst_space ltx_font_typewriter">           </span><span id="lstnumberx38.2" class="ltx_text ltx_font_typewriter">&lt;</span><span id="lstnumberx38.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx38.4" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx38.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx38.6" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx39" class="ltx_listingline">
<span id="lstnumberx39.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx39.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx39.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx39.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx39.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx39.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx39.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx39.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">book_appointment</span><span id="lstnumberx39.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx39.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx40" class="ltx_listingline">
<span id="lstnumberx40.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx40.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx40.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx40.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx40.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</div>
<div id="lstnumberx41" class="ltx_listingline">
<span id="lstnumberx41.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx41.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx41.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">appointment_date</span><span id="lstnumberx41.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx41.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx41.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx41.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">march</span><span id="lstnumberx41.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span><span id="lstnumberx41.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">rd</span><span id="lstnumberx41.12" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx41.13" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx42" class="ltx_listingline">
<span id="lstnumberx42.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                  </span><span id="lstnumberx42.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx42.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">appointment_time</span><span id="lstnumberx42.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx42.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx42.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx42.8" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx42.9" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">8</span><span id="lstnumberx42.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx42.11" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx42.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">5</span><span id="lstnumberx42.13" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx42.14" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx43" class="ltx_listingline">
<span id="lstnumberx43.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                  </span><span id="lstnumberx43.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx43.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stylist_name</span><span id="lstnumberx43.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx43.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx43.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx43.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">A</span><span id="lstnumberx43.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">businessman</span><span id="lstnumberx43.11" class="ltx_text ltx_font_typewriter">’</span><span id="lstnumberx43.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">s</span><span id="lstnumberx43.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">haircut</span><span id="lstnumberx43.15" class="ltx_text ltx_font_typewriter">"}</span>
</div>
<div id="lstnumberx44" class="ltx_listingline">
<span id="lstnumberx44.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx44.2" class="ltx_text ltx_font_typewriter">}"</span>
</div>
<div id="lstnumberx45" class="ltx_listingline">
<span id="lstnumberx45.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
</div>
</section>
<section id="S3.SS2.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Parallel Functions</h5>

<div id="S3.SS2.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS0.Px3.p1.1" class="ltx_p">Similar to the Function Chaining task, here, the answer to a user query requires the same function to be called multiple times (in parallel). Also, the parameters should be mapped with the values extracted from the user query. In the below example, the <span id="S3.SS2.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_typewriter">get_weather</span> function has been called twice with different parameter-value pairs to answer the user query.</p>
</div>
<div id="S3.SS2.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<div id="S3.SS2.SSS0.Px3.p2.1" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogImlucHV0IjogIldoYXQgaXMgdGhlIHdlYXRoZXIgbGlrZSBpbiB0aGUgbmV4dCAzIGRheXMgaW4gTmV3IFlvcmsgYW5kIGluIHRoZSBuZXh0IDUgZGF5cyBpbiBMb3MgQW5nZWxlcz8iLAogIm91dHB1dDogIjxmdW5jdGlvbl9jYWxsPiB7CiAgICAgICAgICAgICAibmFtZSI6ICJnZXRfd2VhdGhlciIsCiAgICAgICAgICAgICAiYXJndW1lbnRzIjoKICAgICAgICAgICAgICAgICB7ImN1cnJlbnRfbG9jYXRpb24iOiAiTmV3IFlvcmsiLAogICAgICAgICAgICAgICAgICAidGltZV9yYW5nZSI6ICJuZXh0IDMgZGF5cyJ9CiAgICAgICAgICAgICB9CiAgICAgICAgICAgPGZ1bmN0aW9uX2NhbGw+IHsKICAgICAgICAgICAgICJuYW1lIjogImdldF93ZWF0aGVyIiwKICAgICAgICAgICAgICJhcmd1bWVudHMiOgogICAgICAgICAgICAgICAgIHsiY3VycmVudF9sb2NhdGlvbiI6ICJMb3MgQW5nZWxlcyIsCiAgICAgICAgICAgICAgICAgICJ0aW1lX3JhbmdlIjogIm5leHQgNSBkYXlzIn0KICAgICAgICAgICAgIH0iCn0=" download="">⬇</a></div>
<div id="lstnumberx46" class="ltx_listingline">
<span id="lstnumberx46.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx47" class="ltx_listingline">
<span id="lstnumberx47.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx47.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">input</span><span id="lstnumberx47.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx47.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx47.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx47.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">What</span><span id="lstnumberx47.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx47.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx47.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">weather</span><span id="lstnumberx47.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">like</span><span id="lstnumberx47.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx47.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx47.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx47.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.24" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span><span id="lstnumberx47.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">days</span><span id="lstnumberx47.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx47.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter">New</span><span id="lstnumberx47.31" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.32" class="ltx_text ltx_lst_identifier ltx_font_typewriter">York</span><span id="lstnumberx47.33" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.34" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx47.35" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.36" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx47.37" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx47.39" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx47.41" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.42" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">5</span><span id="lstnumberx47.43" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter">days</span><span id="lstnumberx47.45" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx47.47" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Los</span><span id="lstnumberx47.49" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx47.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Angeles</span><span id="lstnumberx47.51" class="ltx_text ltx_font_typewriter">?"</span><span id="lstnumberx47.52" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx48" class="ltx_listingline">
<span id="lstnumberx48.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx48.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx48.4" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx48.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.6" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx48.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx48.8" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx48.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.10" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx49" class="ltx_listingline">
<span id="lstnumberx49.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx49.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx49.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx49.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx49.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx49.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx49.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx49.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">get_weather</span><span id="lstnumberx49.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx49.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx50" class="ltx_listingline">
<span id="lstnumberx50.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx50.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx50.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx50.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx50.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</div>
<div id="lstnumberx51" class="ltx_listingline">
<span id="lstnumberx51.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx51.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx51.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">current_location</span><span id="lstnumberx51.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx51.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx51.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx51.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">New</span><span id="lstnumberx51.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx51.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">York</span><span id="lstnumberx51.11" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx51.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx52" class="ltx_listingline">
<span id="lstnumberx52.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                  </span><span id="lstnumberx52.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx52.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">time_range</span><span id="lstnumberx52.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx52.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx52.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx52.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx52.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span><span id="lstnumberx52.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx52.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">days</span><span id="lstnumberx52.13" class="ltx_text ltx_font_typewriter">"}</span>
</div>
<div id="lstnumberx53" class="ltx_listingline">
<span id="lstnumberx53.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx53.2" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx54" class="ltx_listingline">
<span id="lstnumberx54.1" class="ltx_text ltx_lst_space ltx_font_typewriter">           </span><span id="lstnumberx54.2" class="ltx_text ltx_font_typewriter">&lt;</span><span id="lstnumberx54.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx54.4" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx54.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.6" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx55" class="ltx_listingline">
<span id="lstnumberx55.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx55.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx55.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx55.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx55.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx55.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx55.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">get_weather</span><span id="lstnumberx55.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx55.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx56" class="ltx_listingline">
<span id="lstnumberx56.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx56.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx56.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx56.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx56.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</div>
<div id="lstnumberx57" class="ltx_listingline">
<span id="lstnumberx57.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx57.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx57.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">current_location</span><span id="lstnumberx57.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx57.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx57.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx57.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Los</span><span id="lstnumberx57.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Angeles</span><span id="lstnumberx57.11" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx57.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx58" class="ltx_listingline">
<span id="lstnumberx58.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                  </span><span id="lstnumberx58.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx58.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">time_range</span><span id="lstnumberx58.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx58.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx58.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx58.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx58.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">5</span><span id="lstnumberx58.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">days</span><span id="lstnumberx58.13" class="ltx_text ltx_font_typewriter">"}</span>
</div>
<div id="lstnumberx59" class="ltx_listingline">
<span id="lstnumberx59.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx59.2" class="ltx_text ltx_font_typewriter">}"</span>
</div>
<div id="lstnumberx60" class="ltx_listingline">
<span id="lstnumberx60.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
</div>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Low-Level Function Calling Tasks</h3>

<div id="S3.SS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.p1.1" class="ltx_p">Compared to High-Level tasks, these tasks are simpler because they either request function names (without parameters) or parameter-value pairs for a function. To excel in High-Level function calling tasks, it is crucial for any LLM to master these foundational sub-tasks. In what follows, we demonstrate three types of low-level function-calling tasks.</p>
</div>
<section id="S3.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Next-Best Function</h5>

<div id="S3.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px1.p1.1" class="ltx_p">In this task, given the function library along with the user query and the partial function sequence, the models are supposed to select the next most suitable function from the function library. It only requires the model to choose one function name without any parameters. The following example demonstrates how the partial function sequence (provided under <span id="S3.SS3.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_typewriter">&lt;|partial_function_call_list|&gt;</span>) can be passed with the query and the model is supposed to predict the next best function.</p>
</div>
<div id="S3.SS3.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<div id="S3.SS3.SSS0.Px1.p2.1" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogImlucHV0IjogIkdldCB0aGUgZGF0ZXMgZm9yIFJhaWRlcnMgVnMgTGlvbnMgaW4gT2FrbGFuZC4gQnV5IDQgdGlja2V0cyBmb3IgUmFpZGVycyBWcyBMaW9ucyBpbiBPYWtsYW5kIHRvbW9ycm93LgogICAgICA8fHBhcnRpYWxfZnVuY3Rpb25fY2FsbF9saXN0fD4KICAgICAgICAgICAgPGZ1bmN0aW9uX2NhbGw+IHsibmFtZSI6ICJnZXRfZXZlbnRfZGF0ZXMiLCAiYXJndW1lbnRzIjoge319IiwKICJvdXRwdXQ6ICI8ZnVuY3Rpb25fY2FsbD4geyJuYW1lIjogImJ1eV9ldmVudF90aWNrZXRzIiwgImFyZ3VtZW50cyI6IHt9fSIKfQ==" download="">⬇</a></div>
<div id="lstnumberx61" class="ltx_listingline">
<span id="lstnumberx61.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx62" class="ltx_listingline">
<span id="lstnumberx62.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx62.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">input</span><span id="lstnumberx62.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx62.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx62.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx62.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Get</span><span id="lstnumberx62.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx62.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dates</span><span id="lstnumberx62.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">for</span><span id="lstnumberx62.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Raiders</span><span id="lstnumberx62.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Vs</span><span id="lstnumberx62.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lions</span><span id="lstnumberx62.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx62.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Oakland</span><span id="lstnumberx62.25" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx62.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Buy</span><span id="lstnumberx62.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.29" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">4</span><span id="lstnumberx62.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">tickets</span><span id="lstnumberx62.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">for</span><span id="lstnumberx62.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Raiders</span><span id="lstnumberx62.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Vs</span><span id="lstnumberx62.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.39" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Lions</span><span id="lstnumberx62.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx62.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Oakland</span><span id="lstnumberx62.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx62.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">tomorrow</span><span id="lstnumberx62.46" class="ltx_text ltx_font_typewriter">.</span>
</div>
<div id="lstnumberx63" class="ltx_listingline">
<span id="lstnumberx63.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx63.2" class="ltx_text ltx_font_typewriter">&lt;|</span><span id="lstnumberx63.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">partial_function_call_list</span><span id="lstnumberx63.4" class="ltx_text ltx_font_typewriter">|&gt;</span>
</div>
<div id="lstnumberx64" class="ltx_listingline">
<span id="lstnumberx64.1" class="ltx_text ltx_lst_space ltx_font_typewriter">            </span><span id="lstnumberx64.2" class="ltx_text ltx_font_typewriter">&lt;</span><span id="lstnumberx64.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx64.4" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx64.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.6" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx64.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx64.8" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx64.9" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx64.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.11" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx64.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">get_event_dates</span><span id="lstnumberx64.13" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx64.14" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx64.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.16" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx64.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx64.18" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx64.19" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx64.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.21" class="ltx_text ltx_font_typewriter">{}}"</span><span id="lstnumberx64.22" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx65" class="ltx_listingline">
<span id="lstnumberx65.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx65.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx65.4" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx65.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.6" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx65.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx65.8" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx65.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.10" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx65.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx65.12" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx65.13" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx65.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.15" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx65.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">buy_event_tickets</span><span id="lstnumberx65.17" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx65.18" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx65.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.20" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx65.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx65.22" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx65.23" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx65.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.25" class="ltx_text ltx_font_typewriter">{}}"</span>
</div>
<div id="lstnumberx66" class="ltx_listingline">
<span id="lstnumberx66.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
</div>
</section>
<section id="S3.SS3.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Function Name Detection</h5>

<div id="S3.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px2.p1.1" class="ltx_p">This task expects the model to produce only the sequence of function names (without parameters) from the function library that are required to answer the user query. This task closely resembles Function Chaining (a High-Level task), with the sole distinction being it does not necessitate the model to populate the function’s arguments. Below is an example of this task.</p>
</div>
<div id="S3.SS3.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<div id="S3.SS3.SSS0.Px2.p2.1" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogImlucHV0IjogIkZpbmQgdGhlIGhvdGVsIG5hbWVkIGF1dHVtbiBob3VzZS4gQm9vayBhIGhvdGVsIG5hbWVkIEF1dHVtbiBIb3VzZSBmb3IgMiBwZW9wbGUgZm9yIDQgZGF5cyBvbiBTdW5kYXkuIEZpbmQgYSByZXN0YXVyYW50IGluIHNvdXRoIHdpdGggbmFtZSB0YWogdGFuZG9vcmkgYW5kIHByaWNlIHJhbmdlIGV4cGVuc2l2ZS4iLAogIm91dHB1dDogIjxmdW5jdGlvbl9jYWxsPiB7Im5hbWUiOiAiZmluZF9ob3RlbCIsICJhcmd1bWVudHMiOiB7fX0KICAgICAgICAgICA8ZnVuY3Rpb25fY2FsbD4geyJuYW1lIjogImJvb2tfaG90ZWwiLCAiYXJndW1lbnRzIjoge319CiAgICAgICAgICAgPGZ1bmN0aW9uX2NhbGw+IHsibmFtZSI6ICJmaW5kX3Jlc3RhdXJhbnQiLCAiYXJndW1lbnRzIjoge319Igp9" download="">⬇</a></div>
<div id="lstnumberx67" class="ltx_listingline">
<span id="lstnumberx67.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx68" class="ltx_listingline">
<span id="lstnumberx68.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx68.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">input</span><span id="lstnumberx68.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx68.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx68.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx68.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Find</span><span id="lstnumberx68.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx68.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">hotel</span><span id="lstnumberx68.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">named</span><span id="lstnumberx68.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">autumn</span><span id="lstnumberx68.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">house</span><span id="lstnumberx68.19" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx68.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Book</span><span id="lstnumberx68.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx68.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">hotel</span><span id="lstnumberx68.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">named</span><span id="lstnumberx68.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Autumn</span><span id="lstnumberx68.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">House</span><span id="lstnumberx68.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">for</span><span id="lstnumberx68.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.35" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">2</span><span id="lstnumberx68.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">people</span><span id="lstnumberx68.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.39" class="ltx_text ltx_lst_identifier ltx_font_typewriter">for</span><span id="lstnumberx68.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.41" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">4</span><span id="lstnumberx68.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">days</span><span id="lstnumberx68.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">on</span><span id="lstnumberx68.46" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.47" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Sunday</span><span id="lstnumberx68.48" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx68.49" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Find</span><span id="lstnumberx68.51" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx68.53" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter">restaurant</span><span id="lstnumberx68.55" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx68.57" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter">south</span><span id="lstnumberx68.59" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx68.61" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx68.63" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.64" class="ltx_text ltx_lst_identifier ltx_font_typewriter">taj</span><span id="lstnumberx68.65" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.66" class="ltx_text ltx_lst_identifier ltx_font_typewriter">tandoori</span><span id="lstnumberx68.67" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx68.69" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter">price</span><span id="lstnumberx68.71" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.72" class="ltx_text ltx_lst_identifier ltx_font_typewriter">range</span><span id="lstnumberx68.73" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.74" class="ltx_text ltx_lst_identifier ltx_font_typewriter">expensive</span><span id="lstnumberx68.75" class="ltx_text ltx_font_typewriter">."</span><span id="lstnumberx68.76" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx69" class="ltx_listingline">
<span id="lstnumberx69.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx69.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx69.4" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx69.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.6" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx69.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx69.8" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx69.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.10" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx69.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx69.12" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx69.13" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx69.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.15" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx69.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">find_hotel</span><span id="lstnumberx69.17" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx69.18" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx69.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.20" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx69.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx69.22" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx69.23" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx69.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.25" class="ltx_text ltx_font_typewriter">{}}</span>
</div>
<div id="lstnumberx70" class="ltx_listingline">
<span id="lstnumberx70.1" class="ltx_text ltx_lst_space ltx_font_typewriter">           </span><span id="lstnumberx70.2" class="ltx_text ltx_font_typewriter">&lt;</span><span id="lstnumberx70.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx70.4" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx70.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.6" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx70.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx70.8" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx70.9" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx70.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.11" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx70.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">book_hotel</span><span id="lstnumberx70.13" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx70.14" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx70.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.16" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx70.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx70.18" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx70.19" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx70.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.21" class="ltx_text ltx_font_typewriter">{}}</span>
</div>
<div id="lstnumberx71" class="ltx_listingline">
<span id="lstnumberx71.1" class="ltx_text ltx_lst_space ltx_font_typewriter">           </span><span id="lstnumberx71.2" class="ltx_text ltx_font_typewriter">&lt;</span><span id="lstnumberx71.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx71.4" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx71.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.6" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx71.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx71.8" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx71.9" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx71.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.11" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx71.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">find_restaurant</span><span id="lstnumberx71.13" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx71.14" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx71.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.16" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx71.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx71.18" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx71.19" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx71.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.21" class="ltx_text ltx_font_typewriter">{}}"</span>
</div>
<div id="lstnumberx72" class="ltx_listingline">
<span id="lstnumberx72.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
</div>
</section>
<section id="S3.SS3.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Parameter-Value Pair Detection</h5>

<div id="S3.SS3.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px3.p1.1" class="ltx_p">In this task, when provided with a user query or a user-agent conversation along with a list of parameters and their descriptions, the model must identify all the parameters for which the values are present in the query or conversation. See an example of this task below. To keep the data format consistent and JSON parsable, we use the function name - <span id="S3.SS3.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_italic">“dummy”</span> for all the data under this task.</p>
</div>
<div id="S3.SS3.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<div id="S3.SS3.SSS0.Px3.p2.1" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogImlucHV0IjogIkdldCBhIGx1eHVyeSByaWRlIHRvIEdyZWVudHJlZSBBcGFydG1lbnRzIHdpdGggMSBzZWF0LiIsCiAib3V0cHV0OiAiPGZ1bmN0aW9uX2NhbGw+IHsKICAgICAgICAgICAgICJuYW1lIjogImR1bW15IiwKICAgICAgICAgICAgICJhcmd1bWVudHMiOgogICAgICAgICAgICAgICAgIHsiZGVzdGluYXRpb24iOiAiR3JlZW50cmVlIEFwYXJ0bWVudHMiLAogICAgICAgICAgICAgICAgICAibnVtYmVyX29mX3NlYXRzIjogIjEiLAogICAgICAgICAgICAgICAgICAicmlkZV90eXBlIjogIkx1eHVyeSJ9CiAgICAgICAgICAgICB9Igp9" download="">⬇</a></div>
<div id="lstnumberx73" class="ltx_listingline">
<span id="lstnumberx73.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx74" class="ltx_listingline">
<span id="lstnumberx74.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx74.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">input</span><span id="lstnumberx74.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx74.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx74.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx74.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Get</span><span id="lstnumberx74.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx74.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">luxury</span><span id="lstnumberx74.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ride</span><span id="lstnumberx74.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx74.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Greentree</span><span id="lstnumberx74.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Apartments</span><span id="lstnumberx74.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx74.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.24" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx74.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">seat</span><span id="lstnumberx74.27" class="ltx_text ltx_font_typewriter">."</span><span id="lstnumberx74.28" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx75" class="ltx_listingline">
<span id="lstnumberx75.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx75.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">output</span><span id="lstnumberx75.4" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx75.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.6" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx75.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx75.8" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx75.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.10" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx76" class="ltx_listingline">
<span id="lstnumberx76.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx76.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx76.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx76.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx76.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx76.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx76.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx76.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dummy</span><span id="lstnumberx76.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx76.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx77" class="ltx_listingline">
<span id="lstnumberx77.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx77.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx77.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx77.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx77.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</div>
<div id="lstnumberx78" class="ltx_listingline">
<span id="lstnumberx78.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx78.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx78.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">destination</span><span id="lstnumberx78.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx78.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx78.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx78.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Greentree</span><span id="lstnumberx78.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Apartments</span><span id="lstnumberx78.11" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx78.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx79" class="ltx_listingline">
<span id="lstnumberx79.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                  </span><span id="lstnumberx79.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx79.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">number_of_seats</span><span id="lstnumberx79.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx79.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx79.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx79.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx79.8" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx79.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx79.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx80" class="ltx_listingline">
<span id="lstnumberx80.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                  </span><span id="lstnumberx80.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx80.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ride_type</span><span id="lstnumberx80.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx80.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx80.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx80.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx80.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Luxury</span><span id="lstnumberx80.9" class="ltx_text ltx_font_typewriter">"}</span>
</div>
<div id="lstnumberx81" class="ltx_listingline">
<span id="lstnumberx81.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx81.2" class="ltx_text ltx_font_typewriter">}"</span>
</div>
<div id="lstnumberx82" class="ltx_listingline">
<span id="lstnumberx82.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
</div>
</section>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Response Generation</h3>

<div id="S3.SS4.p1" class="ltx_para ltx_noindent">
<p id="S3.SS4.p1.1" class="ltx_p">Natural language response generation is a crucial feature of any LLM. In this task, the model must comprehend an ongoing conversation between a user and an AI assistant. Then, it generates a natural language response, answering the most recent user utterance. Such responses are needed to chit-chat with the user, ask clarifying questions, or synthesize a function call’s output into a natural language response.</p>
</div>
<div id="S3.SS4.p2" class="ltx_para ltx_noindent">
<div id="S3.SS4.p2.1" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ewogIlVTRVIiOiAiSSBoYXZlIHNvbWUgY2hpY2tlbiwgYnJvY2NvbGksIGFuZCBjaGVlc2UuIFdoYXQgY2FuIEkgbWFrZSB3aXRoIHRoZXNlPyIsCiAiQUdFTlQiOiAiPGZ1bmN0aW9uX2NhbGw+IHsKICAgICAgICAgICAgICJuYW1lIjogInNlYXJjaF9yZWNpcGVzIiwKICAgICAgICAgICAgICJhcmd1bWVudHMiOgogICAgICAgICAgICAgICAgIHsiaW5ncmVkaWVudHMiOiBbImNoaWNrZW4iLCAiYnJvY2NvbGkiLCAiY2hlZXNlIl19CiAgICAgICAgICAgICB9CiAgICAgICAgICA8ZnVuY3Rpb25fcmVzcG9uc2U+IHsKICAgICAgICAgICAgICJyZWNpcGVzIjogWwogICAgICAgICAgICAgICAgIHsibmFtZSI6ICJDaGlja2VuIGFuZCBCcm9jY29saSBDYXNzZXJvbGUiLAogICAgICAgICAgICAgICAgICJpbmdyZWRpZW50cyI6IFsiY2hpY2tlbiIsICJicm9jY29saSIsICJjaGVlc2UiXSwKICAgICAgICAgICAgICAgICAiaW5zdHJ1Y3Rpb25zIjogIlByZWhlYXQgb3ZlbiB0byAzNTAgZGVncmVlcyBGICgxNzUgZGVncmVlcyBDKS4gSW4gYSBsYXJnZSBib3dsLCBtaXggdG9nZXRoZXIgY2hpY2tlbiwgYnJvY2NvbGksIGFuZCBjaGVlc2UuIEJha2UgaW4gcHJlaGVhdGVkIG92ZW4gZm9yIDI1IHRvIDMwIG1pbnV0ZXMsIG9yIHVudGlsIGNoZWVzZSBpcyBtZWx0ZWQgYW5kIGJ1YmJseS4ifSwKICAgICAgICAgICAgICAgICB7Im5hbWUiOiAiQ2hlZXN5IENoaWNrZW4gYW5kIEJyb2Njb2xpIFBhc3RhIiwKICAgICAgICAgICAgICAgICAiaW5ncmVkaWVudHMiOiBbImNoaWNrZW4iLCAiYnJvY2NvbGkiLCAiY2hlZXNlIl0sCiAgICAgICAgICAgICAgICAgImluc3RydWN0aW9ucyI6ICJDb29rIHBhc3RhIGFjY29yZGluZyB0byBwYWNrYWdlIGluc3RydWN0aW9ucy4gSW4gYSBsYXJnZSBza2lsbGV0LCBjb29rIGNoaWNrZW4gdW50aWwgbm8gbG9uZ2VyIHBpbmsuIEFkZCBicm9jY29saSBhbmQgY2hlZXNlLCBjb29rIHVudGlsIGNoZWVzZSBpcyBtZWx0ZWQuIn1dCiAgICAgICAgICAgICB9IgogIkFTU0lTVEFOVCI6ICJIZXJlIGFyZSBhIGNvdXBsZSBvZiByZWNpcGVzIHlvdSBjYW4gbWFrZSB3aXRoIGNoaWNrZW4sIGJyb2Njb2xpLCBhbmQgY2hlZXNlOi4uLiIKfQ==" download="">⬇</a></div>
<div id="lstnumberx83" class="ltx_listingline">
<span id="lstnumberx83.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx84" class="ltx_listingline">
<span id="lstnumberx84.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx84.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">USER</span><span id="lstnumberx84.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx84.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx84.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx84.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">I</span><span id="lstnumberx84.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">have</span><span id="lstnumberx84.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">some</span><span id="lstnumberx84.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">chicken</span><span id="lstnumberx84.15" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx84.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">broccoli</span><span id="lstnumberx84.18" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx84.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx84.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cheese</span><span id="lstnumberx84.23" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx84.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">What</span><span id="lstnumberx84.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">can</span><span id="lstnumberx84.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">I</span><span id="lstnumberx84.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">make</span><span id="lstnumberx84.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx84.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">these</span><span id="lstnumberx84.36" class="ltx_text ltx_font_typewriter">?"</span><span id="lstnumberx84.37" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx85" class="ltx_listingline">
<span id="lstnumberx85.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx85.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx85.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">AGENT</span><span id="lstnumberx85.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx85.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx85.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx85.7" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx85.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call</span><span id="lstnumberx85.9" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx85.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx85.11" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx86" class="ltx_listingline">
<span id="lstnumberx86.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx86.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx86.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx86.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx86.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx86.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx86.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx86.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">search_recipes</span><span id="lstnumberx86.9" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx86.10" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx87" class="ltx_listingline">
<span id="lstnumberx87.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx87.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx87.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx87.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx87.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</div>
<div id="lstnumberx88" class="ltx_listingline">
<span id="lstnumberx88.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx88.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx88.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ingredients</span><span id="lstnumberx88.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx88.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx88.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx88.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">[</span><span id="lstnumberx88.8" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx88.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">chicken</span><span id="lstnumberx88.10" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx88.11" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx88.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx88.13" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx88.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">broccoli</span><span id="lstnumberx88.15" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx88.16" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx88.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx88.18" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx88.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cheese</span><span id="lstnumberx88.20" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx88.21" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">]</span><span id="lstnumberx88.22" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx89" class="ltx_listingline">
<span id="lstnumberx89.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx89.2" class="ltx_text ltx_font_typewriter">}</span>
</div>
<div id="lstnumberx90" class="ltx_listingline">
<span id="lstnumberx90.1" class="ltx_text ltx_lst_space ltx_font_typewriter">          </span><span id="lstnumberx90.2" class="ltx_text ltx_font_typewriter">&lt;</span><span id="lstnumberx90.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_response</span><span id="lstnumberx90.4" class="ltx_text ltx_font_typewriter">&gt;</span><span id="lstnumberx90.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.6" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx91" class="ltx_listingline">
<span id="lstnumberx91.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx91.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx91.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">recipes</span><span id="lstnumberx91.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx91.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx91.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx91.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">[</span>
</div>
<div id="lstnumberx92" class="ltx_listingline">
<span id="lstnumberx92.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx92.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx92.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx92.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx92.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx92.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx92.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx92.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Chicken</span><span id="lstnumberx92.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx92.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx92.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx92.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Broccoli</span><span id="lstnumberx92.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx92.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Casserole</span><span id="lstnumberx92.15" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx92.16" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx93" class="ltx_listingline">
<span id="lstnumberx93.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx93.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx93.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ingredients</span><span id="lstnumberx93.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx93.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx93.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx93.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">[</span><span id="lstnumberx93.8" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx93.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">chicken</span><span id="lstnumberx93.10" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx93.11" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx93.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx93.13" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx93.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">broccoli</span><span id="lstnumberx93.15" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx93.16" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx93.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx93.18" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx93.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cheese</span><span id="lstnumberx93.20" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx93.21" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">]</span><span id="lstnumberx93.22" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx94" class="ltx_listingline">
<span id="lstnumberx94.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx94.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx94.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">instructions</span><span id="lstnumberx94.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx94.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx94.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx94.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Preheat</span><span id="lstnumberx94.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">oven</span><span id="lstnumberx94.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx94.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.14" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span><span id="lstnumberx94.15" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">5</span><span id="lstnumberx94.16" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">0</span><span id="lstnumberx94.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">degrees</span><span id="lstnumberx94.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">F</span><span id="lstnumberx94.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.22" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx94.23" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span><span id="lstnumberx94.24" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">7</span><span id="lstnumberx94.25" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">5</span><span id="lstnumberx94.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">degrees</span><span id="lstnumberx94.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">C</span><span id="lstnumberx94.30" class="ltx_text ltx_font_typewriter">).</span><span id="lstnumberx94.31" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.32" class="ltx_text ltx_lst_identifier ltx_font_typewriter">In</span><span id="lstnumberx94.33" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.34" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx94.35" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.36" class="ltx_text ltx_lst_identifier ltx_font_typewriter">large</span><span id="lstnumberx94.37" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter">bowl</span><span id="lstnumberx94.39" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx94.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">mix</span><span id="lstnumberx94.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">together</span><span id="lstnumberx94.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">chicken</span><span id="lstnumberx94.46" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx94.47" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter">broccoli</span><span id="lstnumberx94.49" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx94.50" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.51" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx94.52" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.53" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cheese</span><span id="lstnumberx94.54" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx94.55" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Bake</span><span id="lstnumberx94.57" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx94.59" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter">preheated</span><span id="lstnumberx94.61" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter">oven</span><span id="lstnumberx94.63" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.64" class="ltx_text ltx_lst_identifier ltx_font_typewriter">for</span><span id="lstnumberx94.65" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.66" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">2</span><span id="lstnumberx94.67" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">5</span><span id="lstnumberx94.68" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.69" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx94.70" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.71" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span><span id="lstnumberx94.72" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">0</span><span id="lstnumberx94.73" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.74" class="ltx_text ltx_lst_identifier ltx_font_typewriter">minutes</span><span id="lstnumberx94.75" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx94.76" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.77" class="ltx_text ltx_lst_identifier ltx_font_typewriter">or</span><span id="lstnumberx94.78" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.79" class="ltx_text ltx_lst_identifier ltx_font_typewriter">until</span><span id="lstnumberx94.80" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.81" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cheese</span><span id="lstnumberx94.82" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.83" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx94.84" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.85" class="ltx_text ltx_lst_identifier ltx_font_typewriter">melted</span><span id="lstnumberx94.86" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.87" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx94.88" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.89" class="ltx_text ltx_lst_identifier ltx_font_typewriter">bubbly</span><span id="lstnumberx94.90" class="ltx_text ltx_font_typewriter">."}</span><span id="lstnumberx94.91" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx95" class="ltx_listingline">
<span id="lstnumberx95.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx95.2" class="ltx_text ltx_font_typewriter">{"</span><span id="lstnumberx95.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx95.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx95.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx95.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx95.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Cheesy</span><span id="lstnumberx95.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Chicken</span><span id="lstnumberx95.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx95.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Broccoli</span><span id="lstnumberx95.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Pasta</span><span id="lstnumberx95.17" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx95.18" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx96" class="ltx_listingline">
<span id="lstnumberx96.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx96.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx96.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ingredients</span><span id="lstnumberx96.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx96.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx96.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">[</span><span id="lstnumberx96.8" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx96.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">chicken</span><span id="lstnumberx96.10" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx96.11" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx96.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.13" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx96.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">broccoli</span><span id="lstnumberx96.15" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx96.16" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx96.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx96.18" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx96.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cheese</span><span id="lstnumberx96.20" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx96.21" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">]</span><span id="lstnumberx96.22" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx97" class="ltx_listingline">
<span id="lstnumberx97.1" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx97.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx97.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">instructions</span><span id="lstnumberx97.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx97.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx97.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx97.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Cook</span><span id="lstnumberx97.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">pasta</span><span id="lstnumberx97.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">according</span><span id="lstnumberx97.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx97.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">package</span><span id="lstnumberx97.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">instructions</span><span id="lstnumberx97.19" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx97.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">In</span><span id="lstnumberx97.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx97.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">large</span><span id="lstnumberx97.26" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">skillet</span><span id="lstnumberx97.28" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx97.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cook</span><span id="lstnumberx97.31" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.32" class="ltx_text ltx_lst_identifier ltx_font_typewriter">chicken</span><span id="lstnumberx97.33" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.34" class="ltx_text ltx_lst_identifier ltx_font_typewriter">until</span><span id="lstnumberx97.35" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.36" class="ltx_text ltx_lst_identifier ltx_font_typewriter">no</span><span id="lstnumberx97.37" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter">longer</span><span id="lstnumberx97.39" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter">pink</span><span id="lstnumberx97.41" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx97.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Add</span><span id="lstnumberx97.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">broccoli</span><span id="lstnumberx97.46" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.47" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx97.48" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.49" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cheese</span><span id="lstnumberx97.50" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx97.51" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cook</span><span id="lstnumberx97.53" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter">until</span><span id="lstnumberx97.55" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cheese</span><span id="lstnumberx97.57" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx97.59" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter">melted</span><span id="lstnumberx97.61" class="ltx_text ltx_font_typewriter">."}</span><span id="lstnumberx97.62" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">]</span>
</div>
<div id="lstnumberx98" class="ltx_listingline">
<span id="lstnumberx98.1" class="ltx_text ltx_lst_space ltx_font_typewriter">             </span><span id="lstnumberx98.2" class="ltx_text ltx_font_typewriter">}"</span>
</div>
<div id="lstnumberx99" class="ltx_listingline">
<span id="lstnumberx99.1" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx99.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ASSISTANT</span><span id="lstnumberx99.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx99.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx99.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx99.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Here</span><span id="lstnumberx99.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx99.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx99.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">couple</span><span id="lstnumberx99.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx99.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">recipes</span><span id="lstnumberx99.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">you</span><span id="lstnumberx99.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">can</span><span id="lstnumberx99.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">make</span><span id="lstnumberx99.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx99.27" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter">chicken</span><span id="lstnumberx99.29" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx99.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">broccoli</span><span id="lstnumberx99.32" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx99.33" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.34" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx99.35" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.36" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cheese</span><span id="lstnumberx99.37" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx99.38" class="ltx_text ltx_font_typewriter">..."</span>
</div>
<div id="lstnumberx100" class="ltx_listingline">
<span id="lstnumberx100.1" class="ltx_text ltx_font_typewriter">}</span>
</div>
</div>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Instruct Tuning</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Training Data Mixture Creation</h3>

<div id="S4.SS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.p1.1" class="ltx_p">After generating the data for various tasks, the next step is to create a training data mixture including all the data. We programmatically generate the mixture of data by following a weighted configuration for datasets and tasks. Following is an <span id="S4.SS1.p1.1.1" class="ltx_text ltx_font_italic">example</span> of the weighted configuration, where the total mixture samples will be divided between Function Chaining and Next-Best Function in a 3:5 ratio. Within the Function Chaining portion, the allocation is split between SeqSGD and Glaive-V2 in a 2:3 ratio. Similarly, the Next-Best Function chunk will be divided in a 2:1 ratio between SeqTopV2 and SeqSNIPS.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para ltx_noindent">
<div id="S4.SS1.p2.1" class="ltx_listing ltx_lst_language_json ltx_lstlisting ltx_framed ltx_framed_topbottom ltx_listing" style="background-color:#EEEEEE;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,W3sKICAgICJpbnN0cnVjdGlvbl9uYW1lIjogIkZ1bmN0aW9uIENoYWluaW5nIiwKICAgICJkYXRhc2V0cyI6IHsKICAgICAgICAiU2VxU0dEIjogMiwKICAgICAgICAiR2xhaXZlLVYyIjogMwogICAgfSwKICAgICJ3ZWlnaHQiOiAzCn0sCnsKICAgICJpbnN0cnVjdGlvbl9uYW1lIjogIk5leHQtQmVzdCBGdW5jdGlvbiIsCiAgICAiZGF0YXNldHMiOiB7CiAgICAgICAgIlNlcVRvcFYyIjogMiwKICAgICAgICAiU2VxU05JUFMiOiAxCiAgICB9LAogICAgIndlaWdodCI6IDUKfV0=" download="">⬇</a></div>
<div id="lstnumberx101" class="ltx_listingline">
<span id="lstnumberx101.1" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">[</span><span id="lstnumberx101.2" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx102" class="ltx_listingline">
<span id="lstnumberx102.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx102.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx102.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">instruction_name</span><span id="lstnumberx102.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx102.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx102.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx102.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx102.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Function</span><span id="lstnumberx102.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx102.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Chaining</span><span id="lstnumberx102.11" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx102.12" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx103" class="ltx_listingline">
<span id="lstnumberx103.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx103.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx103.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">datasets</span><span id="lstnumberx103.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx103.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx103.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.7" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx104" class="ltx_listingline">
<span id="lstnumberx104.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx104.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx104.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SeqSGD</span><span id="lstnumberx104.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx104.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx104.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">2</span><span id="lstnumberx104.8" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx105" class="ltx_listingline">
<span id="lstnumberx105.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx105.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx105.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Glaive</span><span id="lstnumberx105.4" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx105.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">V2</span><span id="lstnumberx105.6" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx105.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx105.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx105.9" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span>
</div>
<div id="lstnumberx106" class="ltx_listingline">
<span id="lstnumberx106.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx106.2" class="ltx_text ltx_font_typewriter">}</span><span id="lstnumberx106.3" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx107" class="ltx_listingline">
<span id="lstnumberx107.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx107.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx107.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">weight</span><span id="lstnumberx107.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx107.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx107.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">3</span>
</div>
<div id="lstnumberx108" class="ltx_listingline">
<span id="lstnumberx108.1" class="ltx_text ltx_font_typewriter">}</span><span id="lstnumberx108.2" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx109" class="ltx_listingline">
<span id="lstnumberx109.1" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx110" class="ltx_listingline">
<span id="lstnumberx110.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx110.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx110.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">instruction_name</span><span id="lstnumberx110.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx110.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx110.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.7" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx110.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Next</span><span id="lstnumberx110.9" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx110.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Best</span><span id="lstnumberx110.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Function</span><span id="lstnumberx110.13" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx110.14" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx111" class="ltx_listingline">
<span id="lstnumberx111.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx111.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx111.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">datasets</span><span id="lstnumberx111.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx111.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx111.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx111.7" class="ltx_text ltx_font_typewriter">{</span>
</div>
<div id="lstnumberx112" class="ltx_listingline">
<span id="lstnumberx112.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx112.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx112.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SeqTopV2</span><span id="lstnumberx112.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx112.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx112.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">2</span><span id="lstnumberx112.8" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx113" class="ltx_listingline">
<span id="lstnumberx113.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx113.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx113.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SeqSNIPS</span><span id="lstnumberx113.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx113.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx113.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx113.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">1</span>
</div>
<div id="lstnumberx114" class="ltx_listingline">
<span id="lstnumberx114.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx114.2" class="ltx_text ltx_font_typewriter">}</span><span id="lstnumberx114.3" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span>
</div>
<div id="lstnumberx115" class="ltx_listingline">
<span id="lstnumberx115.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx115.2" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx115.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">weight</span><span id="lstnumberx115.4" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx115.5" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx115.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx115.7" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990099;">5</span>
</div>
<div id="lstnumberx116" class="ltx_listingline">
<span id="lstnumberx116.1" class="ltx_text ltx_font_typewriter">}</span><span id="lstnumberx116.2" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#1469B0;">]</span>
</div>
</div>
</div>
<figure id="S4.SS1.tab1" class="ltx_table">
<div class="ltx_flex_figure ltx_flex_table">
<div class="ltx_flex_cell ltx_flex_size_2">
<table id="S4.SS1.tab1.1" class="ltx_tabular ltx_figure_panel ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.SS1.tab1.1.1.1" class="ltx_tr">
<th id="S4.SS1.tab1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S4.SS1.tab1.1.1.1.1.1" class="ltx_text ltx_font_bold">Task</span></th>
<th id="S4.SS1.tab1.1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_tt">
<span id="S4.SS1.tab1.1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.SS1.tab1.1.1.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="S4.SS1.tab1.1.1.1.2.1.1.1" class="ltx_text ltx_font_bold">Instruction</span></span>
</span>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.SS1.tab1.1.2.1" class="ltx_tr">
<th id="S4.SS1.tab1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="S4.SS1.tab1.1.2.1.1.1" class="ltx_text ltx_font_bold">Nested Function Calling</span></th>
<td id="S4.SS1.tab1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t"></td>
</tr>
<tr id="S4.SS1.tab1.1.3.2" class="ltx_tr">
<th id="S4.SS1.tab1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.SS1.tab1.1.3.2.1.1" class="ltx_text ltx_font_bold">Function Chaining</span></th>
<td id="S4.SS1.tab1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top"></td>
</tr>
<tr id="S4.SS1.tab1.1.4.3" class="ltx_tr">
<th id="S4.SS1.tab1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.SS1.tab1.1.4.3.1.1" class="ltx_text ltx_font_bold">Parallel Functions</span></th>
<td id="S4.SS1.tab1.1.4.3.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="S4.SS1.tab1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.SS1.tab1.1.4.3.2.1.1" class="ltx_p" style="width:256.1pt;">
<span id="S4.SS1.tab1.1.4.3.2.1.1.1" class="ltx_listing ltx_lst_language_text ltx_lstlisting ltx_listing" style="background-color:#ECECEC;"><span class="ltx_listing_data"><a href="data:text/plain;base64,U1lTVEVNOiBZb3UgYXJlIGEgaGVscGZ1bCBhc3Npc3RhbnQgd2l0aCBhY2Nlc3MgdG8gdGhlIGZvbGxvd2luZyBmdW5jdGlvbiBjYWxscy4gWW91ciB0YXNrIGlzIHRvIHByb2R1Y2UgYSBzZXF1ZW5jZSBvZiBmdW5jdGlvbiBjYWxscyBuZWNlc3NhcnkgdG8gZ2VuZXJhdGUgcmVzcG9uc2UgdG8gdGhlIHVzZXIgdXR0ZXJhbmNlLiBVc2UgdGhlIGZvbGxvd2luZyBmdW5jdGlvbiBjYWxscyBhcyByZXF1aXJlZC5cbjx8ZnVuY3Rpb25fY2FsbF9saWJyYXJ5fD5cbntBUElfU1BFQ19JTlNUUlVDVElPTn1cblxuVVNFUjoge1FVRVJZfVxuQVNTSVNUQU5UOg==" download="">⬇</a></span>
<span id="lstnumberx117" class="ltx_listingline"><span id="lstnumberx117.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SYSTEM</span><span id="lstnumberx117.2" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx117.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">You</span><span id="lstnumberx117.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx117.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx117.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">helpful</span><span id="lstnumberx117.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">assistant</span><span id="lstnumberx117.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx117.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">access</span><span id="lstnumberx117.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx117.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx117.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx117.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx117.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx117.27" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx117.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Your</span><span id="lstnumberx117.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">task</span><span id="lstnumberx117.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx117.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx117.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">produce</span><span id="lstnumberx117.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.39" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx117.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sequence</span><span id="lstnumberx117.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx117.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx117.46" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.47" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx117.48" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.49" class="ltx_text ltx_lst_identifier ltx_font_typewriter">necessary</span><span id="lstnumberx117.50" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.51" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx117.52" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.53" class="ltx_text ltx_lst_identifier ltx_font_typewriter">generate</span><span id="lstnumberx117.54" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.55" class="ltx_text ltx_lst_identifier ltx_font_typewriter">response</span><span id="lstnumberx117.56" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.57" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx117.58" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.59" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx117.60" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.61" class="ltx_text ltx_lst_identifier ltx_font_typewriter">user</span><span id="lstnumberx117.62" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.63" class="ltx_text ltx_lst_identifier ltx_font_typewriter">utterance</span><span id="lstnumberx117.64" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx117.65" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.66" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Use</span><span id="lstnumberx117.67" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx117.69" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx117.71" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.72" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx117.73" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.74" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx117.75" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.76" class="ltx_text ltx_lst_identifier ltx_font_typewriter">as</span><span id="lstnumberx117.77" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.78" class="ltx_text ltx_lst_identifier ltx_font_typewriter">required</span><span id="lstnumberx117.79" class="ltx_text ltx_font_typewriter">.\</span><span id="lstnumberx117.80" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx117.81" class="ltx_text ltx_font_typewriter">&lt;|</span><span id="lstnumberx117.82" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call_library</span><span id="lstnumberx117.83" class="ltx_text ltx_font_typewriter">|&gt;\</span><span id="lstnumberx117.84" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx117.85" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx117.86" class="ltx_text ltx_lst_identifier ltx_font_typewriter">API_SPEC_INSTRUCTION</span><span id="lstnumberx117.87" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx117.88" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx117.89" class="ltx_text ltx_font_typewriter">\</span><span id="lstnumberx117.90" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nUSER</span><span id="lstnumberx117.91" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx117.92" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.93" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx117.94" class="ltx_text ltx_lst_identifier ltx_font_typewriter">QUERY</span><span id="lstnumberx117.95" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx117.96" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nASSISTANT</span><span id="lstnumberx117.97" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</span>
</span></span>
</span>
</td>
</tr>
<tr id="S4.SS1.tab1.1.5.4" class="ltx_tr">
<th id="S4.SS1.tab1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.SS1.tab1.1.5.4.1.1" class="ltx_text ltx_font_bold">Next-Best Function</span></th>
<td id="S4.SS1.tab1.1.5.4.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="S4.SS1.tab1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.SS1.tab1.1.5.4.2.1.1" class="ltx_p" style="width:256.1pt;">
<span id="S4.SS1.tab1.1.5.4.2.1.1.1" class="ltx_listing ltx_lst_language_text ltx_lstlisting ltx_listing" style="background-color:#ECECEC;"><span class="ltx_listing_data"><a href="data:text/plain;base64,U1lTVEVNOiBZb3UgYXJlIGEgaGVscGZ1bCBhc3Npc3RhbnQgd2l0aCBhY2Nlc3MgdG8gdGhlIGZvbGxvd2luZyBmdW5jdGlvbiBjYWxscy4gWW91ciB0YXNrIGlzIHRvIHByb2R1Y2UgdGhlIG5leHQgZnVuY3Rpb24gY2FsbCBuZWNlc3NhcnkgdG8gZ2VuZXJhdGUgcmVzcG9uc2UgdG8gdGhlIHVzZXIgdXR0ZXJhbmNlIGdpdmVuIHRoZSBwYXJ0aWFsIGZ1bmN0aW9uIGxpc3QuIFVzZSB0aGUgZm9sbG93aW5nIGZ1bmN0aW9uIGNhbGxzIGFzIHJlcXVpcmVkIGFuZCByZXR1cm4gb25seSBmdW5jdGlvbiAibmFtZSIgd2l0aCBlbXB0eSAiYXJndW1lbnRzIiBkaWN0aW9uYXJ5IGluIHlvdXIgcmVzcG9uc2UuIE9uY2UgYWxsIHRoZSBuZWNlc3NhcnkgZnVuY3Rpb25zIGFyZSBjYWxsZWQsIHBsZWFzZSByZXR1cm4gIjx8ZW5kb2Z0ZXh0fD4iLlxuPHxmdW5jdGlvbl9jYWxsX2xpYnJhcnl8Plxue0FQSV9TUEVDX0lOU1RSVUNUSU9OfVxuXG5VU0VSOiB7UVVFUll9XG5BU1NJU1RBTlQ6" download="">⬇</a></span>
<span id="lstnumberx118" class="ltx_listingline"><span id="lstnumberx118.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SYSTEM</span><span id="lstnumberx118.2" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx118.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">You</span><span id="lstnumberx118.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx118.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx118.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">helpful</span><span id="lstnumberx118.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">assistant</span><span id="lstnumberx118.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx118.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">access</span><span id="lstnumberx118.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx118.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx118.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx118.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx118.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx118.27" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx118.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Your</span><span id="lstnumberx118.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">task</span><span id="lstnumberx118.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx118.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx118.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">produce</span><span id="lstnumberx118.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.39" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx118.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">next</span><span id="lstnumberx118.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx118.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">call</span><span id="lstnumberx118.46" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.47" class="ltx_text ltx_lst_identifier ltx_font_typewriter">necessary</span><span id="lstnumberx118.48" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.49" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx118.50" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.51" class="ltx_text ltx_lst_identifier ltx_font_typewriter">generate</span><span id="lstnumberx118.52" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.53" class="ltx_text ltx_lst_identifier ltx_font_typewriter">response</span><span id="lstnumberx118.54" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.55" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx118.56" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.57" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx118.58" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.59" class="ltx_text ltx_lst_identifier ltx_font_typewriter">user</span><span id="lstnumberx118.60" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.61" class="ltx_text ltx_lst_identifier ltx_font_typewriter">utterance</span><span id="lstnumberx118.62" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.63" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx118.64" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx118.66" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.67" class="ltx_text ltx_lst_identifier ltx_font_typewriter">partial</span><span id="lstnumberx118.68" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.69" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx118.70" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.71" class="ltx_text ltx_lst_identifier ltx_font_typewriter">list</span><span id="lstnumberx118.72" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx118.73" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.74" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Use</span><span id="lstnumberx118.75" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.76" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx118.77" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.78" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx118.79" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.80" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx118.81" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.82" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx118.83" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.84" class="ltx_text ltx_lst_identifier ltx_font_typewriter">as</span><span id="lstnumberx118.85" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.86" class="ltx_text ltx_lst_identifier ltx_font_typewriter">required</span><span id="lstnumberx118.87" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.88" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx118.89" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.90" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx118.91" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.92" class="ltx_text ltx_lst_identifier ltx_font_typewriter">only</span><span id="lstnumberx118.93" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.94" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx118.95" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.96" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx118.97" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx118.98" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx118.99" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.100" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx118.101" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.102" class="ltx_text ltx_lst_identifier ltx_font_typewriter">empty</span><span id="lstnumberx118.103" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.104" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx118.105" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx118.106" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx118.107" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.108" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dictionary</span><span id="lstnumberx118.109" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.110" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx118.111" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.112" class="ltx_text ltx_lst_identifier ltx_font_typewriter">your</span><span id="lstnumberx118.113" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.114" class="ltx_text ltx_lst_identifier ltx_font_typewriter">response</span><span id="lstnumberx118.115" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx118.116" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.117" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Once</span><span id="lstnumberx118.118" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.119" class="ltx_text ltx_lst_identifier ltx_font_typewriter">all</span><span id="lstnumberx118.120" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.121" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx118.122" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.123" class="ltx_text ltx_lst_identifier ltx_font_typewriter">necessary</span><span id="lstnumberx118.124" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.125" class="ltx_text ltx_lst_identifier ltx_font_typewriter">functions</span><span id="lstnumberx118.126" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.127" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx118.128" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.129" class="ltx_text ltx_lst_identifier ltx_font_typewriter">called</span><span id="lstnumberx118.130" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx118.131" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.132" class="ltx_text ltx_lst_identifier ltx_font_typewriter">please</span><span id="lstnumberx118.133" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.134" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx118.135" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.136" class="ltx_text ltx_font_typewriter">"&lt;|</span><span id="lstnumberx118.137" class="ltx_text ltx_lst_identifier ltx_font_typewriter">endoftext</span><span id="lstnumberx118.138" class="ltx_text ltx_font_typewriter">|&gt;".\</span><span id="lstnumberx118.139" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx118.140" class="ltx_text ltx_font_typewriter">&lt;|</span><span id="lstnumberx118.141" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call_library</span><span id="lstnumberx118.142" class="ltx_text ltx_font_typewriter">|&gt;\</span><span id="lstnumberx118.143" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx118.144" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx118.145" class="ltx_text ltx_lst_identifier ltx_font_typewriter">API_SPEC_INSTRUCTION</span><span id="lstnumberx118.146" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx118.147" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx118.148" class="ltx_text ltx_font_typewriter">\</span><span id="lstnumberx118.149" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nUSER</span><span id="lstnumberx118.150" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx118.151" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.152" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx118.153" class="ltx_text ltx_lst_identifier ltx_font_typewriter">QUERY</span><span id="lstnumberx118.154" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx118.155" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nASSISTANT</span><span id="lstnumberx118.156" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</span>
</span></span>
</span>
</td>
</tr>
<tr id="S4.SS1.tab1.1.6.5" class="ltx_tr">
<th id="S4.SS1.tab1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.SS1.tab1.1.6.5.1.1" class="ltx_text ltx_font_bold">Function Name Detection</span></th>
<td id="S4.SS1.tab1.1.6.5.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="S4.SS1.tab1.1.6.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.SS1.tab1.1.6.5.2.1.1" class="ltx_p" style="width:256.1pt;">
<span id="S4.SS1.tab1.1.6.5.2.1.1.1" class="ltx_listing ltx_lst_language_text ltx_lstlisting ltx_listing" style="background-color:#ECECEC;"><span class="ltx_listing_data"><a href="data:text/plain;base64,U1lTVEVNOiBZb3UgYXJlIGEgaGVscGZ1bCBhc3Npc3RhbnQgd2l0aCBhY2Nlc3MgdG8gdGhlIGZvbGxvd2luZyBmdW5jdGlvbiBjYWxscy4gWW91ciB0YXNrIGlzIHRvIHByb2R1Y2UgYSBzZXF1ZW5jZSBvZiBmdW5jdGlvbiBjYWxscyBuZWNlc3NhcnkgdG8gZ2VuZXJhdGUgcmVzcG9uc2UgdG8gdGhlIHVzZXIgdXR0ZXJhbmNlLiBVc2UgdGhlIGZvbGxvd2luZyBmdW5jdGlvbiBjYWxscyBhcyByZXF1aXJlZCBhbmQgcmV0dXJuIG9ubHkgZnVuY3Rpb24gIm5hbWUiIHdpdGggZW1wdHkgImFyZ3VtZW50cyIgZGljdGlvbmFyeSBpbiB5b3VyIHJlc3BvbnNlLiBJZiBubyBmdW5jdGlvbiBpcyByZWxldmFudCwgcGxlYXNlIHJldHVybiAiPG5vX2Z1bmN0aW9uX2NhbGw+IiBmb2xsb3dlZCBieSAiPHxlbmRvZnRleHR8PiIuXG48fGZ1bmN0aW9uX2NhbGxfbGlicmFyeXw+XG57QVBJX1NQRUNfSU5TVFJVQ1RJT059XG5cblVTRVI6IHtRVUVSWX1cbkFTU0lTVEFOVDo=" download="">⬇</a></span>
<span id="lstnumberx119" class="ltx_listingline"><span id="lstnumberx119.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SYSTEM</span><span id="lstnumberx119.2" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx119.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">You</span><span id="lstnumberx119.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx119.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx119.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">helpful</span><span id="lstnumberx119.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">assistant</span><span id="lstnumberx119.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx119.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">access</span><span id="lstnumberx119.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx119.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx119.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx119.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx119.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx119.27" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx119.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Your</span><span id="lstnumberx119.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">task</span><span id="lstnumberx119.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx119.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx119.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">produce</span><span id="lstnumberx119.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.39" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx119.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sequence</span><span id="lstnumberx119.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">of</span><span id="lstnumberx119.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx119.46" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.47" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx119.48" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.49" class="ltx_text ltx_lst_identifier ltx_font_typewriter">necessary</span><span id="lstnumberx119.50" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.51" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx119.52" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.53" class="ltx_text ltx_lst_identifier ltx_font_typewriter">generate</span><span id="lstnumberx119.54" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.55" class="ltx_text ltx_lst_identifier ltx_font_typewriter">response</span><span id="lstnumberx119.56" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.57" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx119.58" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.59" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx119.60" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.61" class="ltx_text ltx_lst_identifier ltx_font_typewriter">user</span><span id="lstnumberx119.62" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.63" class="ltx_text ltx_lst_identifier ltx_font_typewriter">utterance</span><span id="lstnumberx119.64" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx119.65" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.66" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Use</span><span id="lstnumberx119.67" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx119.69" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx119.71" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.72" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx119.73" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.74" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx119.75" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.76" class="ltx_text ltx_lst_identifier ltx_font_typewriter">as</span><span id="lstnumberx119.77" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.78" class="ltx_text ltx_lst_identifier ltx_font_typewriter">required</span><span id="lstnumberx119.79" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.80" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx119.81" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.82" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx119.83" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.84" class="ltx_text ltx_lst_identifier ltx_font_typewriter">only</span><span id="lstnumberx119.85" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.86" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx119.87" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.88" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx119.89" class="ltx_text ltx_lst_identifier ltx_font_typewriter">name</span><span id="lstnumberx119.90" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx119.91" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.92" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx119.93" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.94" class="ltx_text ltx_lst_identifier ltx_font_typewriter">empty</span><span id="lstnumberx119.95" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.96" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx119.97" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx119.98" class="ltx_text ltx_font_typewriter">"</span><span id="lstnumberx119.99" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.100" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dictionary</span><span id="lstnumberx119.101" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.102" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx119.103" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.104" class="ltx_text ltx_lst_identifier ltx_font_typewriter">your</span><span id="lstnumberx119.105" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.106" class="ltx_text ltx_lst_identifier ltx_font_typewriter">response</span><span id="lstnumberx119.107" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx119.108" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.109" class="ltx_text ltx_lst_identifier ltx_font_typewriter">If</span><span id="lstnumberx119.110" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.111" class="ltx_text ltx_lst_identifier ltx_font_typewriter">no</span><span id="lstnumberx119.112" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.113" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx119.114" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.115" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx119.116" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.117" class="ltx_text ltx_lst_identifier ltx_font_typewriter">relevant</span><span id="lstnumberx119.118" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">,</span><span id="lstnumberx119.119" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.120" class="ltx_text ltx_lst_identifier ltx_font_typewriter">please</span><span id="lstnumberx119.121" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.122" class="ltx_text ltx_lst_identifier ltx_font_typewriter">return</span><span id="lstnumberx119.123" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.124" class="ltx_text ltx_font_typewriter">"&lt;</span><span id="lstnumberx119.125" class="ltx_text ltx_lst_identifier ltx_font_typewriter">no_function_call</span><span id="lstnumberx119.126" class="ltx_text ltx_font_typewriter">&gt;"</span><span id="lstnumberx119.127" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.128" class="ltx_text ltx_lst_identifier ltx_font_typewriter">followed</span><span id="lstnumberx119.129" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.130" class="ltx_text ltx_lst_identifier ltx_font_typewriter">by</span><span id="lstnumberx119.131" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.132" class="ltx_text ltx_font_typewriter">"&lt;|</span><span id="lstnumberx119.133" class="ltx_text ltx_lst_identifier ltx_font_typewriter">endoftext</span><span id="lstnumberx119.134" class="ltx_text ltx_font_typewriter">|&gt;".\</span><span id="lstnumberx119.135" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx119.136" class="ltx_text ltx_font_typewriter">&lt;|</span><span id="lstnumberx119.137" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call_library</span><span id="lstnumberx119.138" class="ltx_text ltx_font_typewriter">|&gt;\</span><span id="lstnumberx119.139" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx119.140" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx119.141" class="ltx_text ltx_lst_identifier ltx_font_typewriter">API_SPEC_INSTRUCTION</span><span id="lstnumberx119.142" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx119.143" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx119.144" class="ltx_text ltx_font_typewriter">\</span><span id="lstnumberx119.145" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nUSER</span><span id="lstnumberx119.146" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx119.147" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.148" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx119.149" class="ltx_text ltx_lst_identifier ltx_font_typewriter">QUERY</span><span id="lstnumberx119.150" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx119.151" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nASSISTANT</span><span id="lstnumberx119.152" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</span>
</span></span>
</span>
</td>
</tr>
<tr id="S4.SS1.tab1.1.7.6" class="ltx_tr">
<th id="S4.SS1.tab1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.SS1.tab1.1.7.6.1.1" class="ltx_text ltx_font_bold">Parameter-Value Pair Detection</span></th>
<td id="S4.SS1.tab1.1.7.6.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top">
<span id="S4.SS1.tab1.1.7.6.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.SS1.tab1.1.7.6.2.1.1" class="ltx_p" style="width:256.1pt;">
<span id="S4.SS1.tab1.1.7.6.2.1.1.1" class="ltx_listing ltx_lst_language_text ltx_lstlisting ltx_listing" style="background-color:#ECECEC;"><span class="ltx_listing_data"><a href="data:text/plain;base64,U1lTVEVNOiBZb3UgYXJlIGEgaGVscGZ1bCBhc3Npc3RhbnQgd2l0aCBhY2Nlc3MgdG8gdGhlIGZvbGxvd2luZyBmdW5jdGlvbiBjYWxscy4gWW91ciB0YXNrIGlzIHRvIGZpbmQgYWxsIHRoZSBuZWNlc3NhcnkgYXJndW1lbnRzIGFuZCB0aGVpciB2YWx1ZXMgZnJvbSB0aGUgdXNlciB1dHRlcmFuY2UgdG8gZ2VuZXJhdGUgcmVzcG9uc2UuIFVzZSB0aGUgZm9sbG93aW5nIGZ1bmN0aW9uIGNhbGxzIGFzIHJlcXVpcmVkIGFuZCBmaWxsIG9ubHkgdGhlIGFyZ3VtZW50cyB3aG9zZSB2YWx1ZXMgYXJlIHByZXNlbnQgaW4gdGhlIHVzZXIgdXR0ZXJhbmNlLlxuPHxmdW5jdGlvbl9jYWxsX2xpYnJhcnl8Plxue0FQSV9TUEVDX0lOU1RSVUNUSU9OfVxuXG5VU0VSOiB7UVVFUll9XG5BU1NJU1RBTlQ6" download="">⬇</a></span>
<span id="lstnumberx120" class="ltx_listingline"><span id="lstnumberx120.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SYSTEM</span><span id="lstnumberx120.2" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx120.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">You</span><span id="lstnumberx120.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx120.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx120.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">helpful</span><span id="lstnumberx120.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">assistant</span><span id="lstnumberx120.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx120.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">access</span><span id="lstnumberx120.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx120.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx120.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx120.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx120.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx120.27" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx120.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Your</span><span id="lstnumberx120.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">task</span><span id="lstnumberx120.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx120.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx120.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">find</span><span id="lstnumberx120.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.39" class="ltx_text ltx_lst_identifier ltx_font_typewriter">all</span><span id="lstnumberx120.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx120.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">necessary</span><span id="lstnumberx120.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx120.46" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.47" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx120.48" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.49" class="ltx_text ltx_lst_identifier ltx_font_typewriter">their</span><span id="lstnumberx120.50" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.51" class="ltx_text ltx_lst_identifier ltx_font_typewriter">values</span><span id="lstnumberx120.52" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.53" class="ltx_text ltx_lst_identifier ltx_font_typewriter">from</span><span id="lstnumberx120.54" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.55" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx120.56" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.57" class="ltx_text ltx_lst_identifier ltx_font_typewriter">user</span><span id="lstnumberx120.58" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.59" class="ltx_text ltx_lst_identifier ltx_font_typewriter">utterance</span><span id="lstnumberx120.60" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.61" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx120.62" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.63" class="ltx_text ltx_lst_identifier ltx_font_typewriter">generate</span><span id="lstnumberx120.64" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter">response</span><span id="lstnumberx120.66" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx120.67" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Use</span><span id="lstnumberx120.69" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx120.71" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.72" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx120.73" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.74" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx120.75" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.76" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx120.77" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.78" class="ltx_text ltx_lst_identifier ltx_font_typewriter">as</span><span id="lstnumberx120.79" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.80" class="ltx_text ltx_lst_identifier ltx_font_typewriter">required</span><span id="lstnumberx120.81" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.82" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx120.83" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.84" class="ltx_text ltx_lst_identifier ltx_font_typewriter">fill</span><span id="lstnumberx120.85" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.86" class="ltx_text ltx_lst_identifier ltx_font_typewriter">only</span><span id="lstnumberx120.87" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.88" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx120.89" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.90" class="ltx_text ltx_lst_identifier ltx_font_typewriter">arguments</span><span id="lstnumberx120.91" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.92" class="ltx_text ltx_lst_identifier ltx_font_typewriter">whose</span><span id="lstnumberx120.93" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.94" class="ltx_text ltx_lst_identifier ltx_font_typewriter">values</span><span id="lstnumberx120.95" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.96" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx120.97" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.98" class="ltx_text ltx_lst_identifier ltx_font_typewriter">present</span><span id="lstnumberx120.99" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.100" class="ltx_text ltx_lst_identifier ltx_font_typewriter">in</span><span id="lstnumberx120.101" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.102" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx120.103" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.104" class="ltx_text ltx_lst_identifier ltx_font_typewriter">user</span><span id="lstnumberx120.105" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.106" class="ltx_text ltx_lst_identifier ltx_font_typewriter">utterance</span><span id="lstnumberx120.107" class="ltx_text ltx_font_typewriter">.\</span><span id="lstnumberx120.108" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx120.109" class="ltx_text ltx_font_typewriter">&lt;|</span><span id="lstnumberx120.110" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call_library</span><span id="lstnumberx120.111" class="ltx_text ltx_font_typewriter">|&gt;\</span><span id="lstnumberx120.112" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx120.113" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx120.114" class="ltx_text ltx_lst_identifier ltx_font_typewriter">API_SPEC_INSTRUCTION</span><span id="lstnumberx120.115" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx120.116" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx120.117" class="ltx_text ltx_font_typewriter">\</span><span id="lstnumberx120.118" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nUSER</span><span id="lstnumberx120.119" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx120.120" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.121" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx120.122" class="ltx_text ltx_lst_identifier ltx_font_typewriter">QUERY</span><span id="lstnumberx120.123" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx120.124" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nASSISTANT</span><span id="lstnumberx120.125" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</span>
</span></span>
</span>
</td>
</tr>
<tr id="S4.SS1.tab1.1.8.7" class="ltx_tr">
<th id="S4.SS1.tab1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S4.SS1.tab1.1.8.7.1.1" class="ltx_text ltx_font_bold">Response Generation</span></th>
<td id="S4.SS1.tab1.1.8.7.2" class="ltx_td ltx_nopad_r ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S4.SS1.tab1.1.8.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.SS1.tab1.1.8.7.2.1.1" class="ltx_p" style="width:256.1pt;">
<span id="S4.SS1.tab1.1.8.7.2.1.1.1" class="ltx_listing ltx_lst_language_text ltx_lstlisting ltx_listing" style="background-color:#ECECEC;"><span class="ltx_listing_data"><a href="data:text/plain;base64,U1lTVEVNOiBZb3UgYXJlIGEgaGVscGZ1bCBhc3Npc3RhbnQgd2l0aCBhY2Nlc3MgdG8gdGhlIGZvbGxvd2luZyBmdW5jdGlvbiBjYWxscy4gWW91ciB0YXNrIGlzIHRvIHVuZGVyc3RhbmQgdGhlIGdpdmVuIGNvbnZlcnNhdGlvbiB3aXRoIGZ1bmN0aW9uIGNhbGxzIGFuZCByZXNwb25zZXMgYW5kIGdlbmVyYXRlIG5hdHVyYWwgbGFuZ3VhZ2UgcmVzcG9uc2UgYXMgdGhlIEFTU0lTVEFOVCB0byBjb250aW51ZSB0aGUgY29udmVyc2F0aW9uLiBZb3UgbWF5IHVzZSB0aGUgZm9sbG93aW5nIGZ1bmN0aW9uIGNhbGxzIHRvIHVuZGVyc3RhbmQgaG93IHRvIHJlc3BvbmQgdG8gdGhlIHVzZXIgcXVlcnkuXG48fGZ1bmN0aW9uX2NhbGxfbGlicmFyeXw+XG57QVBJX1NQRUNfSU5TVFJVQ1RJT059XG5cbntDT05WfVxuQVNTSVNUQU5UOg==" download="">⬇</a></span>
<span id="lstnumberx121" class="ltx_listingline"><span id="lstnumberx121.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">SYSTEM</span><span id="lstnumberx121.2" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span><span id="lstnumberx121.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">You</span><span id="lstnumberx121.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">are</span><span id="lstnumberx121.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">a</span><span id="lstnumberx121.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">helpful</span><span id="lstnumberx121.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">assistant</span><span id="lstnumberx121.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx121.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">access</span><span id="lstnumberx121.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx121.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx121.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx121.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx121.25" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx121.27" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx121.28" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter">Your</span><span id="lstnumberx121.30" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.31" class="ltx_text ltx_lst_identifier ltx_font_typewriter">task</span><span id="lstnumberx121.32" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is</span><span id="lstnumberx121.34" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx121.36" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.37" class="ltx_text ltx_lst_identifier ltx_font_typewriter">understand</span><span id="lstnumberx121.38" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.39" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx121.40" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.41" class="ltx_text ltx_lst_identifier ltx_font_typewriter">given</span><span id="lstnumberx121.42" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.43" class="ltx_text ltx_lst_identifier ltx_font_typewriter">conversation</span><span id="lstnumberx121.44" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.45" class="ltx_text ltx_lst_identifier ltx_font_typewriter">with</span><span id="lstnumberx121.46" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.47" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx121.48" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.49" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx121.50" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.51" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx121.52" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.53" class="ltx_text ltx_lst_identifier ltx_font_typewriter">responses</span><span id="lstnumberx121.54" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.55" class="ltx_text ltx_lst_identifier ltx_font_typewriter">and</span><span id="lstnumberx121.56" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.57" class="ltx_text ltx_lst_identifier ltx_font_typewriter">generate</span><span id="lstnumberx121.58" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.59" class="ltx_text ltx_lst_identifier ltx_font_typewriter">natural</span><span id="lstnumberx121.60" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.61" class="ltx_text ltx_lst_identifier ltx_font_typewriter">language</span><span id="lstnumberx121.62" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.63" class="ltx_text ltx_lst_identifier ltx_font_typewriter">response</span><span id="lstnumberx121.64" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter">as</span><span id="lstnumberx121.66" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.67" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx121.68" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.69" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ASSISTANT</span><span id="lstnumberx121.70" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.71" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx121.72" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.73" class="ltx_text ltx_lst_identifier ltx_font_typewriter">continue</span><span id="lstnumberx121.74" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.75" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx121.76" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.77" class="ltx_text ltx_lst_identifier ltx_font_typewriter">conversation</span><span id="lstnumberx121.78" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx121.79" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.80" class="ltx_text ltx_lst_identifier ltx_font_typewriter">You</span><span id="lstnumberx121.81" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.82" class="ltx_text ltx_lst_identifier ltx_font_typewriter">may</span><span id="lstnumberx121.83" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.84" class="ltx_text ltx_lst_identifier ltx_font_typewriter">use</span><span id="lstnumberx121.85" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.86" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx121.87" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.88" class="ltx_text ltx_lst_identifier ltx_font_typewriter">following</span><span id="lstnumberx121.89" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.90" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function</span><span id="lstnumberx121.91" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.92" class="ltx_text ltx_lst_identifier ltx_font_typewriter">calls</span><span id="lstnumberx121.93" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.94" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx121.95" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.96" class="ltx_text ltx_lst_identifier ltx_font_typewriter">understand</span><span id="lstnumberx121.97" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.98" class="ltx_text ltx_lst_identifier ltx_font_typewriter">how</span><span id="lstnumberx121.99" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.100" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx121.101" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.102" class="ltx_text ltx_lst_identifier ltx_font_typewriter">respond</span><span id="lstnumberx121.103" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.104" class="ltx_text ltx_lst_identifier ltx_font_typewriter">to</span><span id="lstnumberx121.105" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.106" class="ltx_text ltx_lst_identifier ltx_font_typewriter">the</span><span id="lstnumberx121.107" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.108" class="ltx_text ltx_lst_identifier ltx_font_typewriter">user</span><span id="lstnumberx121.109" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.110" class="ltx_text ltx_lst_identifier ltx_font_typewriter">query</span><span id="lstnumberx121.111" class="ltx_text ltx_font_typewriter">.\</span><span id="lstnumberx121.112" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx121.113" class="ltx_text ltx_font_typewriter">&lt;|</span><span id="lstnumberx121.114" class="ltx_text ltx_lst_identifier ltx_font_typewriter">function_call_library</span><span id="lstnumberx121.115" class="ltx_text ltx_font_typewriter">|&gt;\</span><span id="lstnumberx121.116" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx121.117" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx121.118" class="ltx_text ltx_lst_identifier ltx_font_typewriter">API_SPEC_INSTRUCTION</span><span id="lstnumberx121.119" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx121.120" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx121.121" class="ltx_text ltx_font_typewriter">\</span><span id="lstnumberx121.122" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n</span><span id="lstnumberx121.123" class="ltx_text ltx_font_typewriter">{</span><span id="lstnumberx121.124" class="ltx_text ltx_lst_identifier ltx_font_typewriter">CONV</span><span id="lstnumberx121.125" class="ltx_text ltx_font_typewriter">}\</span><span id="lstnumberx121.126" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nASSISTANT</span><span id="lstnumberx121.127" class="ltx_text ltx_lst_literate ltx_font_typewriter" style="color:#990000;">:</span>
</span>
</span></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 2: </span>Task specific instructions</figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="S4.SS1.tab1.2" class="ltx_p ltx_figure_panel">Also, in this step, the training data is embedded with the instructions. These instructions are based on the tasks associated with the data. Table <a href="#S4.SS1" title="4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a> showcases all the instructions (task-wise) we have used in our training. The “<code id="S4.SS1.tab1.2.1" class="ltx_verbatim ltx_font_typewriter">&lt;|function_call_library|&gt;</code>” tag has been used for the function library demonstrated in the prompt with the placeholder named - <code id="S4.SS1.tab1.2.2" class="ltx_verbatim ltx_font_typewriter">{API_SPEC_INSTRUCTION}</code>. As the name suggests, the <code id="S4.SS1.tab1.2.3" class="ltx_verbatim ltx_font_typewriter">{QUERY}</code> and <code id="S4.SS1.tab1.2.4" class="ltx_verbatim ltx_font_typewriter">{CONVERSATION}</code> serve as placeholders for user queries or a user-agent conversation, respectively.</p>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<section id="S4.SS2" class="ltx_subsection ltx_figure_panel">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Training</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p"><span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> is instruct-tuned version of <span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-Code-Instruct</span> <cite class="ltx_cite ltx_citemacro_citep">(Mishra et al., <a href="#bib.bib22" title="" class="ltx_ref">2024</a>)</cite><span id="footnote11" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">11</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">11</sup><span class="ltx_tag ltx_tag_note">11</span><a target="_blank" href="https://huggingface.co/ibm-granite/granite-20b-code-instruct" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/ibm-granite/granite-20b-code-instruct</a></span></span></span>. For training data, we created a mixture of 142K examples spanning all the tasks’ datasets discussed above. We then trained our model using QLoRA fine-tuning <cite class="ltx_cite ltx_citemacro_citep">(Dettmers et al., <a href="#bib.bib9" title="" class="ltx_ref">2023</a>)</cite> based on our multi-task training mixture discussed above. In particular, we trained <span id="S4.SS2.p1.1.3" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> a QLoRA rank of 8, alpha of 32 and a dropout of 0.1. We also used a learning rate of 5e-5 and ApexFusedAdam as our optimizer with a linear learning rate scheduler. Training was done using a single node of 8 A100_80GB GPUs with 800GB of RAM for a total of 3 epochs.</p>
</div>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Experimental Setup and Evaluation</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">In the section below, we detail our extensive evaluation on various evaluation datasets and public leaderboard. We provide a comprehensive comparison of our <span id="S5.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span>, open sourced with Apache 2.0 license, to other open and proprietary function calling models.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Datasets</h3>

<div id="S5.SS1.p1" class="ltx_para ltx_noindent">
<p id="S5.SS1.p1.1" class="ltx_p">The evaluation datasets and leaderboards for function calling are gaining a lot of traction in the recent past. In particular, to evaluate the models’ generalizability, we evaluated <span id="S5.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> on a variety of function calling benchmarks, all of which are out-of-domain evaluation for our model. It is worth noting that some of these datasets; e.g. ToolAlpaca and ToolLLM, have training data releases. However, we <span id="S5.SS1.p1.1.2" class="ltx_text ltx_font_italic">did not use</span> any of these benchmarks to train <span id="S5.SS1.p1.1.3" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> and we only used the datasets in <a href="#S3.T1" title="Table 1 ‣ 3 Multi-Task Training Data ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. <span id="footnote12" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">12</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">12</sup><span class="ltx_tag ltx_tag_note">12</span>We could not verify whether some (or all) of the out-of-domain datasets were used in other models’ training sets.</span></span></span> Table <a href="#S5.T3" title="Table 3 ‣ 5.1 Datasets ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> depicts the details of the evaluation datasets we used. We list the details of each of these evaluation datasets below.</p>
</div>
<figure id="S5.T3" class="ltx_table">
<div id="S5.T3.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:318.0pt;height:143.5pt;vertical-align:-0.8pt;"><span class="ltx_transformed_inner" style="transform:translate(-41.6pt,18.6pt) scale(0.792797082348602,0.792797082348602) ;">
<table id="S5.T3.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T3.1.1.1.1" class="ltx_tr">
<th id="S5.T3.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Dataset</th>
<th id="S5.T3.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">Test Instances</th>
<th id="S5.T3.1.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Testing tasks</th>
<th id="S5.T3.1.1.1.1.4" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Metrics</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T3.1.1.2.1" class="ltx_tr">
<th id="S5.T3.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">BFCL</th>
<th id="S5.T3.1.1.2.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">1,700</th>
<td id="S5.T3.1.1.2.1.3" class="ltx_td ltx_align_left ltx_border_t">Function Calling</td>
<td id="S5.T3.1.1.2.1.4" class="ltx_td ltx_align_left ltx_border_t">AST, Execution Accuracy</td>
</tr>
<tr id="S5.T3.1.1.3.2" class="ltx_tr">
<th id="S5.T3.1.1.3.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S5.T3.1.1.3.2.2" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S5.T3.1.1.3.2.3" class="ltx_td ltx_align_left">Relevancy</td>
<td id="S5.T3.1.1.3.2.4" class="ltx_td ltx_align_left">Accuracy</td>
</tr>
<tr id="S5.T3.1.1.4.3" class="ltx_tr">
<th id="S5.T3.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ToolLLM</th>
<th id="S5.T3.1.1.4.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">491</th>
<td id="S5.T3.1.1.4.3.3" class="ltx_td ltx_align_left">Function Calling</td>
<td id="S5.T3.1.1.4.3.4" class="ltx_td ltx_align_left">Func. matching (F1)</td>
</tr>
<tr id="S5.T3.1.1.5.4" class="ltx_tr">
<th id="S5.T3.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">RestGPT</th>
<th id="S5.T3.1.1.5.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">157</th>
<td id="S5.T3.1.1.5.4.3" class="ltx_td ltx_align_left">Function Calling</td>
<td id="S5.T3.1.1.5.4.4" class="ltx_td ltx_align_left">Func. matching (F1)</td>
</tr>
<tr id="S5.T3.1.1.6.5" class="ltx_tr">
<th id="S5.T3.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">API-Bank</th>
<th id="S5.T3.1.1.6.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">473</th>
<td id="S5.T3.1.1.6.5.3" class="ltx_td ltx_align_left">Function Calling</td>
<td id="S5.T3.1.1.6.5.4" class="ltx_td ltx_align_left">Func. and Param. matching (F1)</td>
</tr>
<tr id="S5.T3.1.1.7.6" class="ltx_tr">
<th id="S5.T3.1.1.7.6.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S5.T3.1.1.7.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">478</th>
<td id="S5.T3.1.1.7.6.3" class="ltx_td ltx_align_left">Response Generation</td>
<td id="S5.T3.1.1.7.6.4" class="ltx_td ltx_align_left">BERTscore, ROUGE, BLEU</td>
</tr>
<tr id="S5.T3.1.1.8.7" class="ltx_tr">
<th id="S5.T3.1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ToolBench</th>
<th id="S5.T3.1.1.8.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">214</th>
<td id="S5.T3.1.1.8.7.3" class="ltx_td ltx_align_left">Function Calling</td>
<td id="S5.T3.1.1.8.7.4" class="ltx_td ltx_align_left">Func. and Param. matching (F1)</td>
</tr>
<tr id="S5.T3.1.1.9.8" class="ltx_tr">
<th id="S5.T3.1.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ToolAlpaca</th>
<th id="S5.T3.1.1.9.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">100</th>
<td id="S5.T3.1.1.9.8.3" class="ltx_td ltx_align_left">Function Calling</td>
<td id="S5.T3.1.1.9.8.4" class="ltx_td ltx_align_left">Func. and Param. matching (F1)</td>
</tr>
<tr id="S5.T3.1.1.10.9" class="ltx_tr">
<th id="S5.T3.1.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">NexusRaven</th>
<th id="S5.T3.1.1.10.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb">318</th>
<td id="S5.T3.1.1.10.9.3" class="ltx_td ltx_align_left ltx_border_bb">Function Calling</td>
<td id="S5.T3.1.1.10.9.4" class="ltx_td ltx_align_left ltx_border_bb">Func. and Param. matching (F1)</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Evaluation Datasets</figcaption>
</figure>
<div id="S5.SS1.p2" class="ltx_para ltx_noindent">
<p id="S5.SS1.p2.1" class="ltx_p">(1) <span id="S5.SS1.p2.1.1" class="ltx_text ltx_font_bold">Berkeley Function-Calling Leaderboard (BFCL) <span id="footnote13" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">13</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">13</sup><span class="ltx_tag ltx_tag_note"><span id="footnote13.1.1.1" class="ltx_text ltx_font_medium">13</span></span><a href="BFCL:https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">BFCL:https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html</a></span></span></span> </span>
is a comprehensive function calling leaderboard that includes a dataset of over 1,700 instances. The leaderboard evaluates tasks that include (a) Simple Function, Multiple Function, Parallel Function, and Parallel Multiple Function for Python Language; and (b) for non-Python, they evaluate function relevance detection, REST API, JavaScript, and Java. 
(2) <span id="S5.SS1.p2.1.2" class="ltx_text ltx_font_bold">ToolBench <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib45" title="" class="ltx_ref">2023b</a>)</cite></span> is a subset of the data in ToolBench (as released by the authors) focused on HomeSearch and Booking domains. 
(3) <span id="S5.SS1.p2.1.3" class="ltx_text ltx_font_bold">ToolLLM</span> <cite class="ltx_cite ltx_citemacro_citep">(Qin et al., <a href="#bib.bib32" title="" class="ltx_ref">2023</a>)</cite><span id="footnote14" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">14</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">14</sup><span class="ltx_tag ltx_tag_note">14</span>ToolLLM also calls their benchmark ToolBench. To disambiguate, in this paper we use the term ToolLLM to refer to their benchmark dataset.</span></span></span> is synthetically generated using ChatGPT. The approach uses an initial collection of 16,000 APIs from RapidAPI<span id="footnote15" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">15</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">15</sup><span class="ltx_tag ltx_tag_note">15</span><a target="_blank" href="https://rapidapi.com/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://rapidapi.com/</a></span></span></span> for synthetic data generation. The evaluation is done on the three test sets categorized based on complexity; G1 – single-tool, G2 – intra-category multi-tool, G3 – intra-collection multi-tool. 
(4) <span id="S5.SS1.p2.1.4" class="ltx_text ltx_font_bold">RestGPT</span> <cite class="ltx_cite ltx_citemacro_cite">Song et al. (<a href="#bib.bib38" title="" class="ltx_ref">2023</a>)</cite> is a function calling dataset that has 157 test examples with 85 APIs from Spotify and TMDB. This dataset focuses only on testing model’s ability to detect function names. 
(5) <span id="S5.SS1.p2.1.5" class="ltx_text ltx_font_bold">API-Bank</span> <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib19" title="" class="ltx_ref">2023b</a>)</cite> has 314 tool-use dialogues with 753 API calls to assess LLMs’ capabilities in planning, retrieving, and calling APIs. 
(6) <span id="S5.SS1.p2.1.6" class="ltx_text ltx_font_bold">ToolAlpaca</span> <cite class="ltx_cite ltx_citemacro_citep">(Tang et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>)</cite> is a synthetic data generation approach that has both training and evaluation benchmarks. It contains 271 tool-use instances spanning 50 distinct categories. Similar to <cite class="ltx_cite ltx_citemacro_cite">Nexusflow.ai (<a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite>, we used the simulated part of ToolAlpaca which has a total of 100 test examples. 
(7) <span id="S5.SS1.p2.1.7" class="ltx_text ltx_font_bold">NexusRaven API Evaluation<span id="footnote16" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">16</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">16</sup><span class="ltx_tag ltx_tag_note"><span id="footnote16.1.1.1" class="ltx_text ltx_font_medium">16</span></span><a target="_blank" href="https://huggingface.co/datasets/Nexusflow/NexusRaven_API_evaluation" title="" class="ltx_ref ltx_url ltx_font_typewriter ltx_font_medium">https://huggingface.co/datasets/Nexusflow/NexusRaven_API_evaluation</a></span></span></span></span> is another function calling dataset with 318 test examples covering a total of 65 different APIs.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Evaluation Metrics</h3>

<div id="S5.SS2.p1" class="ltx_para ltx_noindent">
<p id="S5.SS2.p1.1" class="ltx_p">Below, we define the metrics we adopted for specific tasks in function calling.</p>
</div>
<div id="S5.SS2.p2" class="ltx_para ltx_noindent">
<p id="S5.SS2.p2.1" class="ltx_p"><span id="S5.SS2.p2.1.1" class="ltx_text ltx_font_bold">BFCL Metrics<span id="footnote17" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">17</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">17</sup><span class="ltx_tag ltx_tag_note"><span id="footnote17.1.1.1" class="ltx_text ltx_font_medium">17</span></span><a target="_blank" href="https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html#metrics" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html#metrics</a></span></span></span>: </span> BFCL evaluates multiple tasks using the following four metrics. 
(1) <span id="S5.SS2.p2.1.2" class="ltx_text ltx_font_italic">AST summary</span> compares the abstract syntax tree of the function output to the ground truth and the function definition. It captures the correctness of the functions called, their parameters (required or not), and the parameter types. 
(2) <span id="S5.SS2.p2.1.3" class="ltx_text ltx_font_italic">Execution Summary</span> compares the execution output from generated and ground-truth function calls. This metric is used to evaluate REST APIs and non-REST data samples. 
(3) <span id="S5.SS2.p2.1.4" class="ltx_text ltx_font_italic">Relevance</span> evaluates the model’s ability to detect no function calls when the given list of functions is irrelevant to the user query. This inversely captures the hallucination rate of models. 
(4) <span id="S5.SS2.p2.1.5" class="ltx_text ltx_font_italic">Overall Accuracy</span> is the weighted average of all individual data splits in BFCL.</p>
</div>
<div id="S5.SS2.p3" class="ltx_para ltx_noindent">
<p id="S5.SS2.p3.1" class="ltx_p">The same metrics described above cannot be used for our out-of-domain datasets because of missing information, varied formats, and response generation task. For example, ToolLLM datasets has missing arguments, ToolAlpaca has missing argument types, and API-Bank has response generation task. Therefore, we use the following metrics to evaluate the models on other datasets:</p>
</div>
<div id="S5.SS2.p4" class="ltx_para ltx_noindent">
<p id="S5.SS2.p4.1" class="ltx_p"><span id="S5.SS2.p4.1.1" class="ltx_text ltx_font_bold">F1 measure</span>: Based on <cite class="ltx_cite ltx_citemacro_cite">Basu et al. (<a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite>, we opted for standard metrics like precision, recall, and F1 scores which focus on exactly matching API and parameters’ names. The reason behind this is that APIs are very specific and unless everything (e.g., name, parameters, input/output format, etc.) matches the API specifications, executing such APIs will not be possible. We report F1 for matching function names as well as parameter names and values.</p>
</div>
<div id="S5.SS2.p5" class="ltx_para ltx_noindent">
<p id="S5.SS2.p5.1" class="ltx_p"><span id="S5.SS2.p5.1.1" class="ltx_text ltx_font_bold">Longest Common Subsequence (LCS) and Exact match</span>: We also used LCS from <cite class="ltx_cite ltx_citemacro_citep">(Basu et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite> to capture the overlap between the gold and predicted sequences of APIs. This allows us to compute models’ ability to predict APIs in the correct sequence as required by the user. Similarly, exact match score  <cite class="ltx_cite ltx_citemacro_citep">(Basu et al., <a href="#bib.bib2" title="" class="ltx_ref">2024</a>)</cite> checks if all APIs are predicted by the model and are in the same order.</p>
</div>
<div id="S5.SS2.p6" class="ltx_para ltx_noindent">
<p id="S5.SS2.p6.1" class="ltx_p"><span id="S5.SS2.p6.1.1" class="ltx_text ltx_font_bold">BERTScore, ROUGE-L and BLEU</span>: We follow the evaluation in API-Bank <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib19" title="" class="ltx_ref">2023b</a>)</cite>, a dialog dataset that also evaluates model responses based on language generation metrics such as Rouge-L <cite class="ltx_cite ltx_citemacro_citep">(Lin, <a href="#bib.bib20" title="" class="ltx_ref">2004</a>)</cite>, BertScore <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib51" title="" class="ltx_ref">2019</a>)</cite>, and BLEU <cite class="ltx_cite ltx_citemacro_citep">(Papineni et al., <a href="#bib.bib28" title="" class="ltx_ref">2002</a>)</cite>.</p>
</div>
<div id="S5.SS2.p7" class="ltx_para ltx_noindent">
<p id="S5.SS2.p7.1" class="ltx_p"><span id="S5.SS2.p7.1.1" class="ltx_text ltx_font_bold">Hallucination Rate</span>: We compute the hallucination rate as the number of samples where the model predicted an API not provided in the function library.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Evaluation Results</h3>

<div id="S5.SS3.p1" class="ltx_para ltx_noindent">
<p id="S5.SS3.p1.1" class="ltx_p">Tables <a href="#S5.T4" title="Table 4 ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, <a href="#S5.T5" title="Table 5 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, <a href="#S5.T6" title="Table 6 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, <a href="#S5.T7" title="Table 7 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, and Figure <a href="#S5.F3" title="Figure 3 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> depicts an extensive evaluation of <span id="S5.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> model in comparison to other state of the art function calling models. In order to detail this evaluation and analyses, below we categorize the results into (a) Berkeley Function Calling Leaderboard Evaluation, and (b) Function calling academic benchmarks.</p>
</div>
<figure id="S5.T4" class="ltx_table">
<div id="S5.T4.1" class="ltx_inline-block ltx_transformed_outer" style="width:397.5pt;height:191.9pt;vertical-align:-0.7pt;"><span class="ltx_transformed_inner" style="transform:translate(-100.7pt,48.4pt) scale(0.663733553599637,0.663733553599637) ;">
<table id="S5.T4.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T4.1.1.1.1" class="ltx_tr">
<th id="S5.T4.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Model</th>
<th id="S5.T4.1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Organization</th>
<th id="S5.T4.1.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">License</th>
<th id="S5.T4.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">AST Summary</th>
<th id="S5.T4.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Exec. Summary</th>
<th id="S5.T4.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Relevance</th>
<th id="S5.T4.1.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Overall Acc.</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T4.1.1.2.1" class="ltx_tr">
<td id="S5.T4.1.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t">Claude-3.5-Sonnet-20240620 (Prompt)</td>
<td id="S5.T4.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_t">Anthropic</td>
<td id="S5.T4.1.1.2.1.3" class="ltx_td ltx_align_left ltx_border_t">Proprietary</td>
<td id="S5.T4.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">91.31</td>
<td id="S5.T4.1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t">89.50</td>
<td id="S5.T4.1.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t">85.42</td>
<td id="S5.T4.1.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t">90.00</td>
</tr>
<tr id="S5.T4.1.1.3.2" class="ltx_tr">
<td id="S5.T4.1.1.3.2.1" class="ltx_td ltx_align_left">GPT-4-0125-Preview (Prompt)</td>
<td id="S5.T4.1.1.3.2.2" class="ltx_td ltx_align_left">OpenAI</td>
<td id="S5.T4.1.1.3.2.3" class="ltx_td ltx_align_left">Proprietary</td>
<td id="S5.T4.1.1.3.2.4" class="ltx_td ltx_align_center">91.22</td>
<td id="S5.T4.1.1.3.2.5" class="ltx_td ltx_align_center">88.10</td>
<td id="S5.T4.1.1.3.2.6" class="ltx_td ltx_align_center">70.42</td>
<td id="S5.T4.1.1.3.2.7" class="ltx_td ltx_align_center">88.00</td>
</tr>
<tr id="S5.T4.1.1.4.3" class="ltx_tr">
<td id="S5.T4.1.1.4.3.1" class="ltx_td ltx_align_left">Gemini-1.5-Pro-Preview-0514 (FC)</td>
<td id="S5.T4.1.1.4.3.2" class="ltx_td ltx_align_left">Google</td>
<td id="S5.T4.1.1.4.3.3" class="ltx_td ltx_align_left">Proprietary</td>
<td id="S5.T4.1.1.4.3.4" class="ltx_td ltx_align_center">87.92</td>
<td id="S5.T4.1.1.4.3.5" class="ltx_td ltx_align_center">83.32</td>
<td id="S5.T4.1.1.4.3.6" class="ltx_td ltx_align_center">89.58</td>
<td id="S5.T4.1.1.4.3.7" class="ltx_td ltx_align_center">86.35</td>
</tr>
<tr id="S5.T4.1.1.5.4" class="ltx_tr">
<td id="S5.T4.1.1.5.4.1" class="ltx_td ltx_align_left"><span id="S5.T4.1.1.5.4.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Granite-20B-FunctionCalling</span></td>
<td id="S5.T4.1.1.5.4.2" class="ltx_td ltx_align_left">IBM</td>
<td id="S5.T4.1.1.5.4.3" class="ltx_td ltx_align_left">Apache 2.0</td>
<td id="S5.T4.1.1.5.4.4" class="ltx_td ltx_align_center">84.11</td>
<td id="S5.T4.1.1.5.4.5" class="ltx_td ltx_align_center">86.50</td>
<td id="S5.T4.1.1.5.4.6" class="ltx_td ltx_align_center">87.08</td>
<td id="S5.T4.1.1.5.4.7" class="ltx_td ltx_align_center">84.71</td>
</tr>
<tr id="S5.T4.1.1.6.5" class="ltx_tr">
<td id="S5.T4.1.1.6.5.1" class="ltx_td ltx_align_left">Gorilla-OpenFunctions-v2 (FC)</td>
<td id="S5.T4.1.1.6.5.2" class="ltx_td ltx_align_left">Gorilla</td>
<td id="S5.T4.1.1.6.5.3" class="ltx_td ltx_align_left">Apache 2.0</td>
<td id="S5.T4.1.1.6.5.4" class="ltx_td ltx_align_center">89.38</td>
<td id="S5.T4.1.1.6.5.5" class="ltx_td ltx_align_center">81.55</td>
<td id="S5.T4.1.1.6.5.6" class="ltx_td ltx_align_center">61.25</td>
<td id="S5.T4.1.1.6.5.7" class="ltx_td ltx_align_center">84.71</td>
</tr>
<tr id="S5.T4.1.1.7.6" class="ltx_tr">
<td id="S5.T4.1.1.7.6.1" class="ltx_td ltx_align_left">Meta-Llama-3-70B-Instruct (Prompt)</td>
<td id="S5.T4.1.1.7.6.2" class="ltx_td ltx_align_left">Meta</td>
<td id="S5.T4.1.1.7.6.3" class="ltx_td ltx_align_left">MetaLlama 3</td>
<td id="S5.T4.1.1.7.6.4" class="ltx_td ltx_align_center">87.74</td>
<td id="S5.T4.1.1.7.6.5" class="ltx_td ltx_align_center">85.32</td>
<td id="S5.T4.1.1.7.6.6" class="ltx_td ltx_align_center">69.17</td>
<td id="S5.T4.1.1.7.6.7" class="ltx_td ltx_align_center">83.88</td>
</tr>
<tr id="S5.T4.1.1.8.7" class="ltx_tr">
<td id="S5.T4.1.1.8.7.1" class="ltx_td ltx_align_left">FireFunction-v2</td>
<td id="S5.T4.1.1.8.7.2" class="ltx_td ltx_align_left">Fireworks</td>
<td id="S5.T4.1.1.8.7.3" class="ltx_td ltx_align_left">Apache 2.0</td>
<td id="S5.T4.1.1.8.7.4" class="ltx_td ltx_align_center">86.44</td>
<td id="S5.T4.1.1.8.7.5" class="ltx_td ltx_align_center">80.26</td>
<td id="S5.T4.1.1.8.7.6" class="ltx_td ltx_align_center">56.67</td>
<td id="S5.T4.1.1.8.7.7" class="ltx_td ltx_align_center">81.88</td>
</tr>
<tr id="S5.T4.1.1.9.8" class="ltx_tr">
<td id="S5.T4.1.1.9.8.1" class="ltx_td ltx_align_left">Mistral-Medium-2312 (Prompt)</td>
<td id="S5.T4.1.1.9.8.2" class="ltx_td ltx_align_left">Mistral AI</td>
<td id="S5.T4.1.1.9.8.3" class="ltx_td ltx_align_left">Proprietary</td>
<td id="S5.T4.1.1.9.8.4" class="ltx_td ltx_align_center">83.76</td>
<td id="S5.T4.1.1.9.8.5" class="ltx_td ltx_align_center">73.47</td>
<td id="S5.T4.1.1.9.8.6" class="ltx_td ltx_align_center">88.33</td>
<td id="S5.T4.1.1.9.8.7" class="ltx_td ltx_align_center">81.35</td>
</tr>
<tr id="S5.T4.1.1.10.9" class="ltx_tr">
<td id="S5.T4.1.1.10.9.1" class="ltx_td ltx_align_left">Functionary-Medium-v2.4 (FC)</td>
<td id="S5.T4.1.1.10.9.2" class="ltx_td ltx_align_left">MeetKai</td>
<td id="S5.T4.1.1.10.9.3" class="ltx_td ltx_align_left">MIT</td>
<td id="S5.T4.1.1.10.9.4" class="ltx_td ltx_align_center">85.61</td>
<td id="S5.T4.1.1.10.9.5" class="ltx_td ltx_align_center">75.71</td>
<td id="S5.T4.1.1.10.9.6" class="ltx_td ltx_align_center">74.17</td>
<td id="S5.T4.1.1.10.9.7" class="ltx_td ltx_align_center">80.47</td>
</tr>
<tr id="S5.T4.1.1.11.10" class="ltx_tr">
<td id="S5.T4.1.1.11.10.1" class="ltx_td ltx_align_left">Command-R-Plus (Prompt) (Opt.)</td>
<td id="S5.T4.1.1.11.10.2" class="ltx_td ltx_align_left">Cohere</td>
<td id="S5.T4.1.1.11.10.3" class="ltx_td ltx_align_left">cc-by-nc-4.0</td>
<td id="S5.T4.1.1.11.10.4" class="ltx_td ltx_align_center">83.60</td>
<td id="S5.T4.1.1.11.10.5" class="ltx_td ltx_align_center">86.74</td>
<td id="S5.T4.1.1.11.10.6" class="ltx_td ltx_align_center">54.17</td>
<td id="S5.T4.1.1.11.10.7" class="ltx_td ltx_align_center">80.35</td>
</tr>
<tr id="S5.T4.1.1.12.11" class="ltx_tr">
<td id="S5.T4.1.1.12.11.1" class="ltx_td ltx_align_left">Functionary-Small-v2.4 (FC)</td>
<td id="S5.T4.1.1.12.11.2" class="ltx_td ltx_align_left">MeetKai</td>
<td id="S5.T4.1.1.12.11.3" class="ltx_td ltx_align_left">MIT</td>
<td id="S5.T4.1.1.12.11.4" class="ltx_td ltx_align_center">83.55</td>
<td id="S5.T4.1.1.12.11.5" class="ltx_td ltx_align_center">76.31</td>
<td id="S5.T4.1.1.12.11.6" class="ltx_td ltx_align_center">67.92</td>
<td id="S5.T4.1.1.12.11.7" class="ltx_td ltx_align_center">79.94</td>
</tr>
<tr id="S5.T4.1.1.13.12" class="ltx_tr">
<td id="S5.T4.1.1.13.12.1" class="ltx_td ltx_align_left">Mistral-large-2402 (FC Auto)</td>
<td id="S5.T4.1.1.13.12.2" class="ltx_td ltx_align_left">Mistral AI</td>
<td id="S5.T4.1.1.13.12.3" class="ltx_td ltx_align_left">Proprietary</td>
<td id="S5.T4.1.1.13.12.4" class="ltx_td ltx_align_center">64.73</td>
<td id="S5.T4.1.1.13.12.5" class="ltx_td ltx_align_center">60.01</td>
<td id="S5.T4.1.1.13.12.6" class="ltx_td ltx_align_center">84.17</td>
<td id="S5.T4.1.1.13.12.7" class="ltx_td ltx_align_center">68.76</td>
</tr>
<tr id="S5.T4.1.1.14.13" class="ltx_tr">
<td id="S5.T4.1.1.14.13.1" class="ltx_td ltx_align_left">Nexusflow-Raven-v2 (FC)</td>
<td id="S5.T4.1.1.14.13.2" class="ltx_td ltx_align_left">Nexusflow</td>
<td id="S5.T4.1.1.14.13.3" class="ltx_td ltx_align_left">Apache 2.0</td>
<td id="S5.T4.1.1.14.13.4" class="ltx_td ltx_align_center">65.19</td>
<td id="S5.T4.1.1.14.13.5" class="ltx_td ltx_align_center">73.89</td>
<td id="S5.T4.1.1.14.13.6" class="ltx_td ltx_align_center">57.5</td>
<td id="S5.T4.1.1.14.13.7" class="ltx_td ltx_align_center">67.35</td>
</tr>
<tr id="S5.T4.1.1.15.14" class="ltx_tr">
<td id="S5.T4.1.1.15.14.1" class="ltx_td ltx_align_left">DBRX-Instruct (Prompt)</td>
<td id="S5.T4.1.1.15.14.2" class="ltx_td ltx_align_left">Databricks</td>
<td id="S5.T4.1.1.15.14.3" class="ltx_td ltx_align_left">Databricks</td>
<td id="S5.T4.1.1.15.14.4" class="ltx_td ltx_align_center">66.62</td>
<td id="S5.T4.1.1.15.14.5" class="ltx_td ltx_align_center">74.92</td>
<td id="S5.T4.1.1.15.14.6" class="ltx_td ltx_align_center">55.83</td>
<td id="S5.T4.1.1.15.14.7" class="ltx_td ltx_align_center">65.88</td>
</tr>
<tr id="S5.T4.1.1.16.15" class="ltx_tr">
<td id="S5.T4.1.1.16.15.1" class="ltx_td ltx_align_left ltx_border_bb">Snowflake-arctic-Instruct (Prompt)</td>
<td id="S5.T4.1.1.16.15.2" class="ltx_td ltx_align_left ltx_border_bb">Snowflake</td>
<td id="S5.T4.1.1.16.15.3" class="ltx_td ltx_align_left ltx_border_bb">Apache 2.0</td>
<td id="S5.T4.1.1.16.15.4" class="ltx_td ltx_align_center ltx_border_bb">61.09</td>
<td id="S5.T4.1.1.16.15.5" class="ltx_td ltx_align_center ltx_border_bb">80.04</td>
<td id="S5.T4.1.1.16.15.6" class="ltx_td ltx_align_center ltx_border_bb">59.58</td>
<td id="S5.T4.1.1.16.15.7" class="ltx_td ltx_align_center ltx_border_bb">65.18</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 4: </span>Berkeley Function Calling Benchmark: Top 15 models by Overall Accuracy (as of 06/25/2024). All evaluations are done in a <span id="S5.T4.3.1" class="ltx_text ltx_font_italic">zero-shot</span> manner.</figcaption>
</figure>
<section id="S5.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.3.1 </span>BFCL Leaderboard Evaluation Results</h4>

<div id="S5.SS3.SSS1.p1" class="ltx_para ltx_noindent">
<p id="S5.SS3.SSS1.p1.1" class="ltx_p">Table <a href="#S5.T4" title="Table 4 ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows that <span id="S5.SS3.SSS1.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> is ranked fourth on the overall accuracy metric among the top 15 models on BFCL and is highest among models with open licenses<span id="footnote18" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">18</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">18</sup><span class="ltx_tag ltx_tag_note">18</span>We have picked the best performing version of each model. For example, Gemini-1.5-Pro-Preview-0514 (FC) and Gemini-1.5-Pro-Preview-0409 (FC) are both part of the leaderboard but for our evaluation, we consider the best of Gemini-1.5-Pro.</span></span></span>. While it is tied with the Gorilla <cite class="ltx_cite ltx_citemacro_citep">(Patil et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>)</cite> model, it is important to note that the latter was finetuned on data that are (a) generated from ChatGPT, and (b) similar data to the test set and hasn’t generalized well to other datasets as shown in Table <a href="#S5.T5" title="Table 5 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> and Figure <a href="#S5.F3" title="Figure 3 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. In the context of model sizes, <span id="S5.SS3.SSS1.p1.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> is one of the smallest models in the list. Specifically, the ones better than <span id="S5.SS3.SSS1.p1.1.3" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> in the ranking are all significantly larger in size.</p>
</div>
<div id="S5.SS3.SSS1.p2" class="ltx_para ltx_noindent">
<p id="S5.SS3.SSS1.p2.1" class="ltx_p">For the BFCL evaluation dataset, we highlight concerns in certain categories, particularly the Java, JavaScript, and REST API evaluations. We are concerned with how the Java and JavaScript categories evaluate a function-calling model’s capabilities to follow language-specific syntax, for instance how objects are instantiated and called in Java and JavaScript utilizing language-specific context and norms. For the REST API category, we observed significant brittleness in the evaluation due to issues with API availability and API call limits.</p>
</div>
</section>
<section id="S5.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.3.2 </span>Function Calling Academic Benchmarks</h4>

<div id="S5.SS3.SSS2.p1" class="ltx_para ltx_noindent">
<p id="S5.SS3.SSS2.p1.1" class="ltx_p">Tables <a href="#S5.T5" title="Table 5 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> and  <a href="#S5.T6" title="Table 6 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> focus on evaluating the models’ performance on Function Matching using F1-measure, LCS, and Exact Match. In this experiment, we reuse the model handlers from the BFCL code base, including the optimized prompts for each model. However, since the Cohere Command-R-v01 and Mistral-Instruct-v0.3 handlers available in BFCL use the REST API interface for inference, we reimplement handlers for these models, utilizing local models using prompts suggested by the respective model developers for function calling.</p>
</div>
<div id="S5.SS3.SSS2.p2" class="ltx_para ltx_noindent">
<p id="S5.SS3.SSS2.p2.1" class="ltx_p"><span id="S5.SS3.SSS2.p2.1.1" class="ltx_text ltx_font_bold">Function Name Detection:</span> On ToolLLM datasets (G1, G2, and G3) and RestGPT, <span id="S5.SS3.SSS2.p2.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> performs the best on detecting function names given a natural language utterance with <span id="S5.SS3.SSS2.p2.1.3" class="ltx_text ltx_font_bold">8%</span> better F1 score than the next best function calling model, as shown in Table <a href="#S5.T5" title="Table 5 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>. Since these datasets have multiple functions in sequence, we also compute sequencing metrics; exact score and LCS. On this front, <span id="S5.SS3.SSS2.p2.1.4" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> model also outperforms other function calling models by <span id="S5.SS3.SSS2.p2.1.5" class="ltx_text ltx_font_bold">7%</span> on LCS and <span id="S5.SS3.SSS2.p2.1.6" class="ltx_text ltx_font_bold">11%</span> on Exact Match scores.</p>
</div>
<figure id="S5.T5" class="ltx_table">
<div id="S5.T5.1" class="ltx_inline-block ltx_transformed_outer" style="width:397.5pt;height:116.1pt;vertical-align:-0.5pt;"><span class="ltx_transformed_inner" style="transform:translate(-182.0pt,52.9pt) scale(0.522051754909045,0.522051754909045) ;">
<table id="S5.T5.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T5.1.1.1.1" class="ltx_tr">
<th id="S5.T5.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_r ltx_border_tt" rowspan="2"></th>
<th id="S5.T5.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="3">ToolLLM-G1</th>
<th id="S5.T5.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="3">ToolLLM-G2</th>
<th id="S5.T5.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="3">ToolLLM-G3</th>
<th id="S5.T5.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_rr ltx_border_tt" colspan="3">RestGPT</th>
<th id="S5.T5.1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="3">Average</th>
</tr>
<tr id="S5.T5.1.1.2.2" class="ltx_tr">
<th id="S5.T5.1.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T5.1.1.2.2.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.1.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Func.</td>
</tr>
<tr id="S5.T5.1.1.2.2.1.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Match</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">LCS</th>
<th id="S5.T5.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T5.1.1.2.2.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.3.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Exact</td>
</tr>
<tr id="S5.T5.1.1.2.2.3.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Score</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T5.1.1.2.2.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.4.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Func.</td>
</tr>
<tr id="S5.T5.1.1.2.2.4.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Match</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">LCS</th>
<th id="S5.T5.1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T5.1.1.2.2.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.6.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Exact</td>
</tr>
<tr id="S5.T5.1.1.2.2.6.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.6.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Score</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T5.1.1.2.2.7.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.7.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.7.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Func.</td>
</tr>
<tr id="S5.T5.1.1.2.2.7.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.7.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Match</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">LCS</th>
<th id="S5.T5.1.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T5.1.1.2.2.9.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.9.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.9.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Exact</td>
</tr>
<tr id="S5.T5.1.1.2.2.9.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.9.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Score</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T5.1.1.2.2.10.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.10.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.10.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Func.</td>
</tr>
<tr id="S5.T5.1.1.2.2.10.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.10.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Match</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">LCS</th>
<th id="S5.T5.1.1.2.2.12" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_rr ltx_border_t">
<table id="S5.T5.1.1.2.2.12.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.12.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.12.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Exact</td>
</tr>
<tr id="S5.T5.1.1.2.2.12.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.12.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Score</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.2.2.13" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T5.1.1.2.2.13.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.13.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.13.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Func.</td>
</tr>
<tr id="S5.T5.1.1.2.2.13.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.13.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Match</td>
</tr>
</table>
</th>
<th id="S5.T5.1.1.2.2.14" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">LCS</th>
<th id="S5.T5.1.1.2.2.15" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T5.1.1.2.2.15.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T5.1.1.2.2.15.1.1" class="ltx_tr">
<td id="S5.T5.1.1.2.2.15.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Exact</td>
</tr>
<tr id="S5.T5.1.1.2.2.15.1.2" class="ltx_tr">
<td id="S5.T5.1.1.2.2.15.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Score</td>
</tr>
</table>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T5.1.1.3.1" class="ltx_tr">
<td id="S5.T5.1.1.3.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Functionary-small-v2.4 (7B)</td>
<td id="S5.T5.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">0.00</td>
<td id="S5.T5.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">0.00</td>
<td id="S5.T5.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.00</td>
<td id="S5.T5.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">0.00</td>
<td id="S5.T5.1.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t">0.00</td>
<td id="S5.T5.1.1.3.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.00</td>
<td id="S5.T5.1.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t">0.00</td>
<td id="S5.T5.1.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t">0.00</td>
<td id="S5.T5.1.1.3.1.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.00</td>
<td id="S5.T5.1.1.3.1.11" class="ltx_td ltx_align_center ltx_border_t">0.29</td>
<td id="S5.T5.1.1.3.1.12" class="ltx_td ltx_align_center ltx_border_t">0.30</td>
<td id="S5.T5.1.1.3.1.13" class="ltx_td ltx_align_center ltx_border_rr ltx_border_t">0.06</td>
<td id="S5.T5.1.1.3.1.14" class="ltx_td ltx_align_center ltx_border_t">0.07</td>
<td id="S5.T5.1.1.3.1.15" class="ltx_td ltx_align_center ltx_border_t">0.07</td>
<td id="S5.T5.1.1.3.1.16" class="ltx_td ltx_align_center ltx_border_t">0.02</td>
</tr>
<tr id="S5.T5.1.1.4.2" class="ltx_tr">
<td id="S5.T5.1.1.4.2.1" class="ltx_td ltx_align_left ltx_border_r">Gorilla-openfunctions-v2 (7B)</td>
<td id="S5.T5.1.1.4.2.2" class="ltx_td ltx_align_center">0.59</td>
<td id="S5.T5.1.1.4.2.3" class="ltx_td ltx_align_center">0.59</td>
<td id="S5.T5.1.1.4.2.4" class="ltx_td ltx_align_center ltx_border_r">0.28</td>
<td id="S5.T5.1.1.4.2.5" class="ltx_td ltx_align_center">0.48</td>
<td id="S5.T5.1.1.4.2.6" class="ltx_td ltx_align_center">0.48</td>
<td id="S5.T5.1.1.4.2.7" class="ltx_td ltx_align_center ltx_border_r">0.22</td>
<td id="S5.T5.1.1.4.2.8" class="ltx_td ltx_align_center">0.51</td>
<td id="S5.T5.1.1.4.2.9" class="ltx_td ltx_align_center">0.52</td>
<td id="S5.T5.1.1.4.2.10" class="ltx_td ltx_align_center ltx_border_r">0.24</td>
<td id="S5.T5.1.1.4.2.11" class="ltx_td ltx_align_center">0.21</td>
<td id="S5.T5.1.1.4.2.12" class="ltx_td ltx_align_center">0.21</td>
<td id="S5.T5.1.1.4.2.13" class="ltx_td ltx_align_center ltx_border_rr">0.01</td>
<td id="S5.T5.1.1.4.2.14" class="ltx_td ltx_align_center">0.44</td>
<td id="S5.T5.1.1.4.2.15" class="ltx_td ltx_align_center">0.45</td>
<td id="S5.T5.1.1.4.2.16" class="ltx_td ltx_align_center">0.19</td>
</tr>
<tr id="S5.T5.1.1.5.3" class="ltx_tr">
<td id="S5.T5.1.1.5.3.1" class="ltx_td ltx_align_left ltx_border_r">Hermes-2-Pro-Mistral (7B)</td>
<td id="S5.T5.1.1.5.3.2" class="ltx_td ltx_align_center">0.00</td>
<td id="S5.T5.1.1.5.3.3" class="ltx_td ltx_align_center">0.00</td>
<td id="S5.T5.1.1.5.3.4" class="ltx_td ltx_align_center ltx_border_r">0.00</td>
<td id="S5.T5.1.1.5.3.5" class="ltx_td ltx_align_center">0.00</td>
<td id="S5.T5.1.1.5.3.6" class="ltx_td ltx_align_center">0.00</td>
<td id="S5.T5.1.1.5.3.7" class="ltx_td ltx_align_center ltx_border_r">0.00</td>
<td id="S5.T5.1.1.5.3.8" class="ltx_td ltx_align_center">0.00</td>
<td id="S5.T5.1.1.5.3.9" class="ltx_td ltx_align_center">0.00</td>
<td id="S5.T5.1.1.5.3.10" class="ltx_td ltx_align_center ltx_border_r">0.00</td>
<td id="S5.T5.1.1.5.3.11" class="ltx_td ltx_align_center">0.03</td>
<td id="S5.T5.1.1.5.3.12" class="ltx_td ltx_align_center">0.03</td>
<td id="S5.T5.1.1.5.3.13" class="ltx_td ltx_align_center ltx_border_rr">0.01</td>
<td id="S5.T5.1.1.5.3.14" class="ltx_td ltx_align_center">0.01</td>
<td id="S5.T5.1.1.5.3.15" class="ltx_td ltx_align_center">0.01</td>
<td id="S5.T5.1.1.5.3.16" class="ltx_td ltx_align_center">0.00</td>
</tr>
<tr id="S5.T5.1.1.6.4" class="ltx_tr">
<td id="S5.T5.1.1.6.4.1" class="ltx_td ltx_align_left ltx_border_r">Mistral-Instruct-v0.3 (7B)</td>
<td id="S5.T5.1.1.6.4.2" class="ltx_td ltx_align_center">0.49</td>
<td id="S5.T5.1.1.6.4.3" class="ltx_td ltx_align_center">0.49</td>
<td id="S5.T5.1.1.6.4.4" class="ltx_td ltx_align_center ltx_border_r">0.26</td>
<td id="S5.T5.1.1.6.4.5" class="ltx_td ltx_align_center">0.51</td>
<td id="S5.T5.1.1.6.4.6" class="ltx_td ltx_align_center">0.49</td>
<td id="S5.T5.1.1.6.4.7" class="ltx_td ltx_align_center ltx_border_r">0.30</td>
<td id="S5.T5.1.1.6.4.8" class="ltx_td ltx_align_center">0.36</td>
<td id="S5.T5.1.1.6.4.9" class="ltx_td ltx_align_center">0.33</td>
<td id="S5.T5.1.1.6.4.10" class="ltx_td ltx_align_center ltx_border_r">0.13</td>
<td id="S5.T5.1.1.6.4.11" class="ltx_td ltx_align_center">0.36</td>
<td id="S5.T5.1.1.6.4.12" class="ltx_td ltx_align_center">0.37</td>
<td id="S5.T5.1.1.6.4.13" class="ltx_td ltx_align_center ltx_border_rr">0.08</td>
<td id="S5.T5.1.1.6.4.14" class="ltx_td ltx_align_center">0.43</td>
<td id="S5.T5.1.1.6.4.15" class="ltx_td ltx_align_center">0.42</td>
<td id="S5.T5.1.1.6.4.16" class="ltx_td ltx_align_center">0.19</td>
</tr>
<tr id="S5.T5.1.1.7.5" class="ltx_tr">
<td id="S5.T5.1.1.7.5.1" class="ltx_td ltx_align_left ltx_border_r">CodeGemma-Instruct (7B)</td>
<td id="S5.T5.1.1.7.5.2" class="ltx_td ltx_align_center">0.59</td>
<td id="S5.T5.1.1.7.5.3" class="ltx_td ltx_align_center">0.59</td>
<td id="S5.T5.1.1.7.5.4" class="ltx_td ltx_align_center ltx_border_r">0.21</td>
<td id="S5.T5.1.1.7.5.5" class="ltx_td ltx_align_center">0.53</td>
<td id="S5.T5.1.1.7.5.6" class="ltx_td ltx_align_center">0.53</td>
<td id="S5.T5.1.1.7.5.7" class="ltx_td ltx_align_center ltx_border_r">0.13</td>
<td id="S5.T5.1.1.7.5.8" class="ltx_td ltx_align_center">0.52</td>
<td id="S5.T5.1.1.7.5.9" class="ltx_td ltx_align_center">0.54</td>
<td id="S5.T5.1.1.7.5.10" class="ltx_td ltx_align_center ltx_border_r">0.16</td>
<td id="S5.T5.1.1.7.5.11" class="ltx_td ltx_align_center">0.22</td>
<td id="S5.T5.1.1.7.5.12" class="ltx_td ltx_align_center">0.23</td>
<td id="S5.T5.1.1.7.5.13" class="ltx_td ltx_align_center ltx_border_rr">0.02</td>
<td id="S5.T5.1.1.7.5.14" class="ltx_td ltx_align_center">0.46</td>
<td id="S5.T5.1.1.7.5.15" class="ltx_td ltx_align_center">0.47</td>
<td id="S5.T5.1.1.7.5.16" class="ltx_td ltx_align_center">0.13</td>
</tr>
<tr id="S5.T5.1.1.8.6" class="ltx_tr">
<td id="S5.T5.1.1.8.6.1" class="ltx_td ltx_align_left ltx_border_r">Nexusflow-Raven-v2 (13B)</td>
<td id="S5.T5.1.1.8.6.2" class="ltx_td ltx_align_center">0.65</td>
<td id="S5.T5.1.1.8.6.3" class="ltx_td ltx_align_center">0.65</td>
<td id="S5.T5.1.1.8.6.4" class="ltx_td ltx_align_center ltx_border_r">0.39</td>
<td id="S5.T5.1.1.8.6.5" class="ltx_td ltx_align_center">0.73</td>
<td id="S5.T5.1.1.8.6.6" class="ltx_td ltx_align_center">0.72</td>
<td id="S5.T5.1.1.8.6.7" class="ltx_td ltx_align_center ltx_border_r">0.43</td>
<td id="S5.T5.1.1.8.6.8" class="ltx_td ltx_align_center">0.68</td>
<td id="S5.T5.1.1.8.6.9" class="ltx_td ltx_align_center">0.66</td>
<td id="S5.T5.1.1.8.6.10" class="ltx_td ltx_align_center ltx_border_r">0.27</td>
<td id="S5.T5.1.1.8.6.11" class="ltx_td ltx_align_center">0.39</td>
<td id="S5.T5.1.1.8.6.12" class="ltx_td ltx_align_center">0.41</td>
<td id="S5.T5.1.1.8.6.13" class="ltx_td ltx_align_center ltx_border_rr">0.06</td>
<td id="S5.T5.1.1.8.6.14" class="ltx_td ltx_align_center">0.61</td>
<td id="S5.T5.1.1.8.6.15" class="ltx_td ltx_align_center">0.61</td>
<td id="S5.T5.1.1.8.6.16" class="ltx_td ltx_align_center">0.28</td>
</tr>
<tr id="S5.T5.1.1.9.7" class="ltx_tr">
<td id="S5.T5.1.1.9.7.1" class="ltx_td ltx_align_left ltx_border_r">C4AI-Command-R-v01 (35B)</td>
<td id="S5.T5.1.1.9.7.2" class="ltx_td ltx_align_center">0.65</td>
<td id="S5.T5.1.1.9.7.3" class="ltx_td ltx_align_center">0.64</td>
<td id="S5.T5.1.1.9.7.4" class="ltx_td ltx_align_center ltx_border_r">0.39</td>
<td id="S5.T5.1.1.9.7.5" class="ltx_td ltx_align_center">0.73</td>
<td id="S5.T5.1.1.9.7.6" class="ltx_td ltx_align_center">0.71</td>
<td id="S5.T5.1.1.9.7.7" class="ltx_td ltx_align_center ltx_border_r">0.45</td>
<td id="S5.T5.1.1.9.7.8" class="ltx_td ltx_align_center">0.69</td>
<td id="S5.T5.1.1.9.7.9" class="ltx_td ltx_align_center">0.68</td>
<td id="S5.T5.1.1.9.7.10" class="ltx_td ltx_align_center ltx_border_r">0.23</td>
<td id="S5.T5.1.1.9.7.11" class="ltx_td ltx_align_center"><span id="S5.T5.1.1.9.7.11.1" class="ltx_text ltx_font_bold">0.59</span></td>
<td id="S5.T5.1.1.9.7.12" class="ltx_td ltx_align_center"><span id="S5.T5.1.1.9.7.12.1" class="ltx_text ltx_font_bold">0.60</span></td>
<td id="S5.T5.1.1.9.7.13" class="ltx_td ltx_align_center ltx_border_rr"><span id="S5.T5.1.1.9.7.13.1" class="ltx_text ltx_font_bold">0.22</span></td>
<td id="S5.T5.1.1.9.7.14" class="ltx_td ltx_align_center"><span id="S5.T5.1.1.9.7.14.1" class="ltx_text ltx_framed ltx_framed_underline">0.66</span></td>
<td id="S5.T5.1.1.9.7.15" class="ltx_td ltx_align_center"><span id="S5.T5.1.1.9.7.15.1" class="ltx_text ltx_framed ltx_framed_underline">0.66</span></td>
<td id="S5.T5.1.1.9.7.16" class="ltx_td ltx_align_center"><span id="S5.T5.1.1.9.7.16.1" class="ltx_text ltx_framed ltx_framed_underline">0.32</span></td>
</tr>
<tr id="S5.T5.1.1.10.8" class="ltx_tr">
<td id="S5.T5.1.1.10.8.1" class="ltx_td ltx_align_left ltx_border_r">Meta-Llama-3-70B-Instruct (70B)</td>
<td id="S5.T5.1.1.10.8.2" class="ltx_td ltx_align_center">0.61</td>
<td id="S5.T5.1.1.10.8.3" class="ltx_td ltx_align_center">0.61</td>
<td id="S5.T5.1.1.10.8.4" class="ltx_td ltx_align_center ltx_border_r">0.31</td>
<td id="S5.T5.1.1.10.8.5" class="ltx_td ltx_align_center">0.59</td>
<td id="S5.T5.1.1.10.8.6" class="ltx_td ltx_align_center">0.58</td>
<td id="S5.T5.1.1.10.8.7" class="ltx_td ltx_align_center ltx_border_r">0.21</td>
<td id="S5.T5.1.1.10.8.8" class="ltx_td ltx_align_center">0.65</td>
<td id="S5.T5.1.1.10.8.9" class="ltx_td ltx_align_center">0.64</td>
<td id="S5.T5.1.1.10.8.10" class="ltx_td ltx_align_center ltx_border_r">0.23</td>
<td id="S5.T5.1.1.10.8.11" class="ltx_td ltx_align_center">0.22</td>
<td id="S5.T5.1.1.10.8.12" class="ltx_td ltx_align_center">0.22</td>
<td id="S5.T5.1.1.10.8.13" class="ltx_td ltx_align_center ltx_border_rr">0.01</td>
<td id="S5.T5.1.1.10.8.14" class="ltx_td ltx_align_center">0.52</td>
<td id="S5.T5.1.1.10.8.15" class="ltx_td ltx_align_center">0.51</td>
<td id="S5.T5.1.1.10.8.16" class="ltx_td ltx_align_center">0.19</td>
</tr>
<tr id="S5.T5.1.1.11.9" class="ltx_tr">
<td id="S5.T5.1.1.11.9.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r"><span id="S5.T5.1.1.11.9.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Granite-20B-FunctionCalling</span></td>
<td id="S5.T5.1.1.11.9.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.1.11.9.2.1" class="ltx_text ltx_font_bold">0.86</span></td>
<td id="S5.T5.1.1.11.9.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.1.11.9.3.1" class="ltx_text ltx_font_bold">0.85</span></td>
<td id="S5.T5.1.1.11.9.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S5.T5.1.1.11.9.4.1" class="ltx_text ltx_font_bold">0.63</span></td>
<td id="S5.T5.1.1.11.9.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.1.11.9.5.1" class="ltx_text ltx_font_bold">0.84</span></td>
<td id="S5.T5.1.1.11.9.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.1.11.9.6.1" class="ltx_text ltx_font_bold">0.82</span></td>
<td id="S5.T5.1.1.11.9.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S5.T5.1.1.11.9.7.1" class="ltx_text ltx_font_bold">0.58</span></td>
<td id="S5.T5.1.1.11.9.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.1.11.9.8.1" class="ltx_text ltx_font_bold">0.76</span></td>
<td id="S5.T5.1.1.11.9.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.1.11.9.9.1" class="ltx_text ltx_font_bold">0.73</span></td>
<td id="S5.T5.1.1.11.9.10" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S5.T5.1.1.11.9.10.1" class="ltx_text ltx_font_bold">0.35</span></td>
<td id="S5.T5.1.1.11.9.11" class="ltx_td ltx_align_center ltx_border_bb">0.51</td>
<td id="S5.T5.1.1.11.9.12" class="ltx_td ltx_align_center ltx_border_bb">0.52</td>
<td id="S5.T5.1.1.11.9.13" class="ltx_td ltx_align_center ltx_border_bb ltx_border_rr">0.15</td>
<td id="S5.T5.1.1.11.9.14" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.1.11.9.14.1" class="ltx_text ltx_font_bold">0.74</span></td>
<td id="S5.T5.1.1.11.9.15" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.1.11.9.15.1" class="ltx_text ltx_font_bold">0.73</span></td>
<td id="S5.T5.1.1.11.9.16" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T5.1.1.11.9.16.1" class="ltx_text ltx_font_bold">0.43</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 5: </span>Function Calling Academic Benchmarks: Function Name Detection. Best performance is highlighted in <span id="S5.T5.5.1" class="ltx_text ltx_font_bold">bold</span>, second best is <span id="S5.T5.6.2" class="ltx_text ltx_framed ltx_framed_underline">underlined</span>. All evaluations are done in a <span id="S5.T5.7.3" class="ltx_text ltx_font_italic">zero-shot</span> manner. </figcaption>
</figure>
<div id="S5.SS3.SSS2.p3" class="ltx_para ltx_noindent">
<p id="S5.SS3.SSS2.p3.1" class="ltx_p"><span id="S5.SS3.SSS2.p3.1.1" class="ltx_text ltx_font_bold">Full Function Calling:</span> Table <a href="#S5.T6" title="Table 6 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> reports on the models’ performance on the API-Bank, ToolBench, and ToolAlpaca datasets that are out-of-domain and evaluated in a zero-shot manner. No single model outperforms all other models across datasets. Note that datasets like ToolAlpaca and API-Bank come with training data split which we never used for training <span id="S5.SS3.SSS2.p3.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span>, but could not guarantee that the other models were not trained with it too. Averaging out the F1 scores across datasets shows that <span id="S5.SS3.SSS2.p3.1.3" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> achieves an F1 score of 0.87 when predicting the function name; second best by 0.01 to Cohere’s Command-R (a 35B model) which provides an F1 score of 0.88. When predicting the arguments, <span id="S5.SS3.SSS2.p3.1.4" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> average F1 score lags behind the best model (Cohere’s Command-R) by 0.03; 0.62 vs. 0.59.</p>
</div>
<div id="S5.SS3.SSS2.p4" class="ltx_para ltx_noindent">
<p id="S5.SS3.SSS2.p4.1" class="ltx_p"><span id="S5.SS3.SSS2.p4.1.1" class="ltx_text ltx_font_bold">Function Name Hallucination:</span> Hallucinations have been a major drawback of large language models. In the context of calling and executing APIs, hallucinations can have adverse consequences. In Figure <a href="#S5.F3" title="Figure 3 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we compare the models’ Function Name Detection Scores (average F1) over all the datasets (except BFCL, which uses AST-based metrics) and their hallucination rates. Ideally, we want models to have high performance and low hallucination rates placing them in the top left corner of the plot. <span id="S5.SS3.SSS2.p4.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> has the highest performance with less than 0.1 hallucination rate.</p>
</div>
<figure id="S5.T6" class="ltx_table">
<div id="S5.T6.1" class="ltx_inline-block ltx_transformed_outer" style="width:397.5pt;height:143.3pt;vertical-align:-0.6pt;"><span class="ltx_transformed_inner" style="transform:translate(-109.7pt,39.4pt) scale(0.644424035839366,0.644424035839366) ;">
<table id="S5.T6.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T6.1.1.1.1" class="ltx_tr">
<th id="S5.T6.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_border_tt" rowspan="2"></th>
<th id="S5.T6.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="6">Func-Name+Args Det. (F1 Func-Name | F1 Args)</th>
<th id="S5.T6.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2">F1 Average</th>
</tr>
<tr id="S5.T6.1.1.2.2" class="ltx_tr">
<th id="S5.T6.1.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T6.1.1.2.2.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T6.1.1.2.2.1.1.1" class="ltx_tr">
<td id="S5.T6.1.1.2.2.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">API-Bank</td>
</tr>
<tr id="S5.T6.1.1.2.2.1.1.2" class="ltx_tr">
<td id="S5.T6.1.1.2.2.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">L-1</td>
</tr>
</table>
</th>
<th id="S5.T6.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T6.1.1.2.2.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T6.1.1.2.2.2.1.1" class="ltx_tr">
<td id="S5.T6.1.1.2.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">API-Bank</td>
</tr>
<tr id="S5.T6.1.1.2.2.2.1.2" class="ltx_tr">
<td id="S5.T6.1.1.2.2.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">L-2</td>
</tr>
</table>
</th>
<th id="S5.T6.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T6.1.1.2.2.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T6.1.1.2.2.3.1.1" class="ltx_tr">
<td id="S5.T6.1.1.2.2.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">ToolBench</td>
</tr>
<tr id="S5.T6.1.1.2.2.3.1.2" class="ltx_tr">
<td id="S5.T6.1.1.2.2.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">HS</td>
</tr>
</table>
</th>
<th id="S5.T6.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T6.1.1.2.2.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T6.1.1.2.2.4.1.1" class="ltx_tr">
<td id="S5.T6.1.1.2.2.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">ToolBench</td>
</tr>
<tr id="S5.T6.1.1.2.2.4.1.2" class="ltx_tr">
<td id="S5.T6.1.1.2.2.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">B</td>
</tr>
</table>
</th>
<th id="S5.T6.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T6.1.1.2.2.5.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T6.1.1.2.2.5.1.1" class="ltx_tr">
<td id="S5.T6.1.1.2.2.5.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Tool-Alpaca</td>
</tr>
</table>
</th>
<th id="S5.T6.1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">
<table id="S5.T6.1.1.2.2.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T6.1.1.2.2.6.1.1" class="ltx_tr">
<td id="S5.T6.1.1.2.2.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Nexus</td>
</tr>
<tr id="S5.T6.1.1.2.2.6.1.2" class="ltx_tr">
<td id="S5.T6.1.1.2.2.6.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Raven</td>
</tr>
</table>
</th>
<th id="S5.T6.1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">
<table id="S5.T6.1.1.2.2.7.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T6.1.1.2.2.7.1.1" class="ltx_tr">
<td id="S5.T6.1.1.2.2.7.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">Func</td>
</tr>
<tr id="S5.T6.1.1.2.2.7.1.2" class="ltx_tr">
<td id="S5.T6.1.1.2.2.7.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">Name</td>
</tr>
</table>
</th>
<th id="S5.T6.1.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Args</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T6.1.1.3.1" class="ltx_tr">
<td id="S5.T6.1.1.3.1.1" class="ltx_td ltx_align_left ltx_border_t">Functionary-small-v2.4 (7B)</td>
<td id="S5.T6.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">0.78 | 0.70</td>
<td id="S5.T6.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">0.54 | 0.45</td>
<td id="S5.T6.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">0.73 | 0.68</td>
<td id="S5.T6.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">0.65 | 0.33</td>
<td id="S5.T6.1.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t">0.88 | <span id="S5.T6.1.1.3.1.6.1" class="ltx_text ltx_font_bold">0.47</span>
</td>
<td id="S5.T6.1.1.3.1.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.82 | 0.64</td>
<td id="S5.T6.1.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t">0.73</td>
<td id="S5.T6.1.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t">0.55</td>
</tr>
<tr id="S5.T6.1.1.4.2" class="ltx_tr">
<td id="S5.T6.1.1.4.2.1" class="ltx_td ltx_align_left">Gorilla-openfunctions-v2 (7B)</td>
<td id="S5.T6.1.1.4.2.2" class="ltx_td ltx_align_center">0.43 | 0.41</td>
<td id="S5.T6.1.1.4.2.3" class="ltx_td ltx_align_center">0.12 | 0.12</td>
<td id="S5.T6.1.1.4.2.4" class="ltx_td ltx_align_center">0.86 | 0.69</td>
<td id="S5.T6.1.1.4.2.5" class="ltx_td ltx_align_center">0.41 | 0.27</td>
<td id="S5.T6.1.1.4.2.6" class="ltx_td ltx_align_center">0.69 | 0.39</td>
<td id="S5.T6.1.1.4.2.7" class="ltx_td ltx_align_center ltx_border_r">0.81 | 0.65</td>
<td id="S5.T6.1.1.4.2.8" class="ltx_td ltx_align_center">0.55</td>
<td id="S5.T6.1.1.4.2.9" class="ltx_td ltx_align_center">0.42</td>
</tr>
<tr id="S5.T6.1.1.5.3" class="ltx_tr">
<td id="S5.T6.1.1.5.3.1" class="ltx_td ltx_align_left">Hermes-2-Pro-Mistral (7B)</td>
<td id="S5.T6.1.1.5.3.2" class="ltx_td ltx_align_center">
<span id="S5.T6.1.1.5.3.2.1" class="ltx_text ltx_font_bold">0.93</span> | <span id="S5.T6.1.1.5.3.2.2" class="ltx_text ltx_font_bold">0.77</span>
</td>
<td id="S5.T6.1.1.5.3.3" class="ltx_td ltx_align_center">0.54 | 0.25</td>
<td id="S5.T6.1.1.5.3.4" class="ltx_td ltx_align_center">0.51 | 0.40</td>
<td id="S5.T6.1.1.5.3.5" class="ltx_td ltx_align_center">0.56 | 0.26</td>
<td id="S5.T6.1.1.5.3.6" class="ltx_td ltx_align_center">0.80 | 0.26</td>
<td id="S5.T6.1.1.5.3.7" class="ltx_td ltx_align_center ltx_border_r">0.90 | 0.63</td>
<td id="S5.T6.1.1.5.3.8" class="ltx_td ltx_align_center">0.71</td>
<td id="S5.T6.1.1.5.3.9" class="ltx_td ltx_align_center">0.43</td>
</tr>
<tr id="S5.T6.1.1.6.4" class="ltx_tr">
<td id="S5.T6.1.1.6.4.1" class="ltx_td ltx_align_left">Mistral-Instruct-v0.3 (7B)</td>
<td id="S5.T6.1.1.6.4.2" class="ltx_td ltx_align_center">0.79 | 0.69</td>
<td id="S5.T6.1.1.6.4.3" class="ltx_td ltx_align_center">0.69 | 0.46</td>
<td id="S5.T6.1.1.6.4.4" class="ltx_td ltx_align_center">0.60 | 0.47</td>
<td id="S5.T6.1.1.6.4.5" class="ltx_td ltx_align_center">0.04 | 0.16</td>
<td id="S5.T6.1.1.6.4.6" class="ltx_td ltx_align_center">0.33 | 0.33</td>
<td id="S5.T6.1.1.6.4.7" class="ltx_td ltx_align_center ltx_border_r">0.71 | 0.54</td>
<td id="S5.T6.1.1.6.4.8" class="ltx_td ltx_align_center">0.53</td>
<td id="S5.T6.1.1.6.4.9" class="ltx_td ltx_align_center">0.44</td>
</tr>
<tr id="S5.T6.1.1.7.5" class="ltx_tr">
<td id="S5.T6.1.1.7.5.1" class="ltx_td ltx_align_left">CodeGemma-Instruct (7B)</td>
<td id="S5.T6.1.1.7.5.2" class="ltx_td ltx_align_center">0.77 | 0.57</td>
<td id="S5.T6.1.1.7.5.3" class="ltx_td ltx_align_center">0.59 | 0.38</td>
<td id="S5.T6.1.1.7.5.4" class="ltx_td ltx_align_center">0.65 | 0.50</td>
<td id="S5.T6.1.1.7.5.5" class="ltx_td ltx_align_center">0.54 | 0.22</td>
<td id="S5.T6.1.1.7.5.6" class="ltx_td ltx_align_center">0.59 | 0.31</td>
<td id="S5.T6.1.1.7.5.7" class="ltx_td ltx_align_center ltx_border_r">0.84 | 0.68</td>
<td id="S5.T6.1.1.7.5.8" class="ltx_td ltx_align_center">0.66</td>
<td id="S5.T6.1.1.7.5.9" class="ltx_td ltx_align_center">0.44</td>
</tr>
<tr id="S5.T6.1.1.8.6" class="ltx_tr">
<td id="S5.T6.1.1.8.6.1" class="ltx_td ltx_align_left">Nexusflow-Raven-v2 (13B)</td>
<td id="S5.T6.1.1.8.6.2" class="ltx_td ltx_align_center">0.51 | 0.42</td>
<td id="S5.T6.1.1.8.6.3" class="ltx_td ltx_align_center">0.28 | 0.22</td>
<td id="S5.T6.1.1.8.6.4" class="ltx_td ltx_align_center">
<span id="S5.T6.1.1.8.6.4.1" class="ltx_text ltx_font_bold">0.92</span> | 0.65</td>
<td id="S5.T6.1.1.8.6.5" class="ltx_td ltx_align_center">0.89 | 0.35</td>
<td id="S5.T6.1.1.8.6.6" class="ltx_td ltx_align_center">0.85 | 0.37</td>
<td id="S5.T6.1.1.8.6.7" class="ltx_td ltx_align_center ltx_border_r">0.92 | <span id="S5.T6.1.1.8.6.7.1" class="ltx_text ltx_font_bold">0.75</span>
</td>
<td id="S5.T6.1.1.8.6.8" class="ltx_td ltx_align_center">0.73</td>
<td id="S5.T6.1.1.8.6.9" class="ltx_td ltx_align_center">0.46</td>
</tr>
<tr id="S5.T6.1.1.9.7" class="ltx_tr">
<td id="S5.T6.1.1.9.7.1" class="ltx_td ltx_align_left">C4AI-Command-R-v01 (35B)</td>
<td id="S5.T6.1.1.9.7.2" class="ltx_td ltx_align_center">
<span id="S5.T6.1.1.9.7.2.1" class="ltx_text ltx_font_bold">0.93</span> | 0.76</td>
<td id="S5.T6.1.1.9.7.3" class="ltx_td ltx_align_center">0.77 | 0.54</td>
<td id="S5.T6.1.1.9.7.4" class="ltx_td ltx_align_center">0.85 | 0.77</td>
<td id="S5.T6.1.1.9.7.5" class="ltx_td ltx_align_center">0.88 | 0.49</td>
<td id="S5.T6.1.1.9.7.6" class="ltx_td ltx_align_center">
<span id="S5.T6.1.1.9.7.6.1" class="ltx_text ltx_font_bold">0.90</span> | 0.42</td>
<td id="S5.T6.1.1.9.7.7" class="ltx_td ltx_align_center ltx_border_r">
<span id="S5.T6.1.1.9.7.7.1" class="ltx_text ltx_font_bold">0.93</span> | 0.71</td>
<td id="S5.T6.1.1.9.7.8" class="ltx_td ltx_align_center"><span id="S5.T6.1.1.9.7.8.1" class="ltx_text ltx_font_bold">0.88</span></td>
<td id="S5.T6.1.1.9.7.9" class="ltx_td ltx_align_center"><span id="S5.T6.1.1.9.7.9.1" class="ltx_text ltx_font_bold">0.62</span></td>
</tr>
<tr id="S5.T6.1.1.10.8" class="ltx_tr">
<td id="S5.T6.1.1.10.8.1" class="ltx_td ltx_align_left">Meta-Llama-3-70B-Instruct (70B)</td>
<td id="S5.T6.1.1.10.8.2" class="ltx_td ltx_align_center">0.85 | 0.67</td>
<td id="S5.T6.1.1.10.8.3" class="ltx_td ltx_align_center">0.69 | 0.52</td>
<td id="S5.T6.1.1.10.8.4" class="ltx_td ltx_align_center">0.91 | <span id="S5.T6.1.1.10.8.4.1" class="ltx_text ltx_font_bold">0.86</span>
</td>
<td id="S5.T6.1.1.10.8.5" class="ltx_td ltx_align_center">
<span id="S5.T6.1.1.10.8.5.1" class="ltx_text ltx_font_bold">0.91</span> | <span id="S5.T6.1.1.10.8.5.2" class="ltx_text ltx_font_bold">0.56</span>
</td>
<td id="S5.T6.1.1.10.8.6" class="ltx_td ltx_align_center">0.78 | 0.43</td>
<td id="S5.T6.1.1.10.8.7" class="ltx_td ltx_align_center ltx_border_r">0.70 | 0.52</td>
<td id="S5.T6.1.1.10.8.8" class="ltx_td ltx_align_center">0.81</td>
<td id="S5.T6.1.1.10.8.9" class="ltx_td ltx_align_center"><span id="S5.T6.1.1.10.8.9.1" class="ltx_text ltx_framed ltx_framed_underline">0.59</span></td>
</tr>
<tr id="S5.T6.1.1.11.9" class="ltx_tr">
<td id="S5.T6.1.1.11.9.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S5.T6.1.1.11.9.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Granite-20B-FunctionCalling</span></td>
<td id="S5.T6.1.1.11.9.2" class="ltx_td ltx_align_center ltx_border_bb">0.91 | 0.71</td>
<td id="S5.T6.1.1.11.9.3" class="ltx_td ltx_align_center ltx_border_bb">
<span id="S5.T6.1.1.11.9.3.1" class="ltx_text ltx_font_bold">0.83</span> |<span id="S5.T6.1.1.11.9.3.2" class="ltx_text ltx_font_bold"> 0.60</span>
</td>
<td id="S5.T6.1.1.11.9.4" class="ltx_td ltx_align_center ltx_border_bb">0.87 | 0.71</td>
<td id="S5.T6.1.1.11.9.5" class="ltx_td ltx_align_center ltx_border_bb">0.82 | 0.36</td>
<td id="S5.T6.1.1.11.9.6" class="ltx_td ltx_align_center ltx_border_bb">0.89 | 0.44</td>
<td id="S5.T6.1.1.11.9.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">0.92 | 0.72</td>
<td id="S5.T6.1.1.11.9.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T6.1.1.11.9.8.1" class="ltx_text ltx_framed ltx_framed_underline">0.87</span></td>
<td id="S5.T6.1.1.11.9.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T6.1.1.11.9.9.1" class="ltx_text ltx_framed ltx_framed_underline">0.59</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 6: </span>Function Calling Academic Benchmarks: Full Function Calling. Best performance is highlighted in <span id="S5.T6.5.1" class="ltx_text ltx_font_bold">bold</span>, second best is <span id="S5.T6.6.2" class="ltx_text ltx_framed ltx_framed_underline">underlined</span>. All evaluations are done in a <span id="S5.T6.7.3" class="ltx_text ltx_font_italic">zero-shot</span> manner.</figcaption>
</figure>
<figure id="S5.T7" class="ltx_table">
<div id="S5.T7.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:357.7pt;height:159.3pt;vertical-align:-0.8pt;"><span class="ltx_transformed_inner" style="transform:translate(-44.6pt,19.7pt) scale(0.80059710316146,0.80059710316146) ;">
<table id="S5.T7.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T7.1.1.1.1" class="ltx_tr">
<th id="S5.T7.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt"></th>
<th id="S5.T7.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="3">API-Bank-Response-Level 1</th>
<th id="S5.T7.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="3">API-Bank-Response-Level 2</th>
</tr>
<tr id="S5.T7.1.1.2.2" class="ltx_tr">
<th id="S5.T7.1.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">Models</th>
<th id="S5.T7.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">BertScore</th>
<th id="S5.T7.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rouge-L</th>
<th id="S5.T7.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t">BLEU</th>
<th id="S5.T7.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">BertScore</th>
<th id="S5.T7.1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Rouge-L</th>
<th id="S5.T7.1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">BLEU</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T7.1.1.3.1" class="ltx_tr">
<th id="S5.T7.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Functionary-small-v2.4 (7B)</th>
<td id="S5.T7.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">0.34</td>
<td id="S5.T7.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">0.23</td>
<td id="S5.T7.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.05</td>
<td id="S5.T7.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">0.35</td>
<td id="S5.T7.1.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t">0.23</td>
<td id="S5.T7.1.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t">0.05</td>
</tr>
<tr id="S5.T7.1.1.4.2" class="ltx_tr">
<th id="S5.T7.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Gorilla-openfunctions-v2 (7B)</th>
<td id="S5.T7.1.1.4.2.2" class="ltx_td ltx_align_center">0.56</td>
<td id="S5.T7.1.1.4.2.3" class="ltx_td ltx_align_center">0.33</td>
<td id="S5.T7.1.1.4.2.4" class="ltx_td ltx_align_center ltx_border_r">0.32</td>
<td id="S5.T7.1.1.4.2.5" class="ltx_td ltx_align_center">0.51</td>
<td id="S5.T7.1.1.4.2.6" class="ltx_td ltx_align_center">0.26</td>
<td id="S5.T7.1.1.4.2.7" class="ltx_td ltx_align_center">0.25</td>
</tr>
<tr id="S5.T7.1.1.5.3" class="ltx_tr">
<th id="S5.T7.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Hermes-2-Pro-Mistral (7B)</th>
<td id="S5.T7.1.1.5.3.2" class="ltx_td ltx_align_center">0.45</td>
<td id="S5.T7.1.1.5.3.3" class="ltx_td ltx_align_center">0.18</td>
<td id="S5.T7.1.1.5.3.4" class="ltx_td ltx_align_center ltx_border_r">0.09</td>
<td id="S5.T7.1.1.5.3.5" class="ltx_td ltx_align_center">0.42</td>
<td id="S5.T7.1.1.5.3.6" class="ltx_td ltx_align_center">0.14</td>
<td id="S5.T7.1.1.5.3.7" class="ltx_td ltx_align_center">0.06</td>
</tr>
<tr id="S5.T7.1.1.6.4" class="ltx_tr">
<th id="S5.T7.1.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Mistral-Instruct-v0.3 (7B)</th>
<td id="S5.T7.1.1.6.4.2" class="ltx_td ltx_align_center">0.52</td>
<td id="S5.T7.1.1.6.4.3" class="ltx_td ltx_align_center">0.29</td>
<td id="S5.T7.1.1.6.4.4" class="ltx_td ltx_align_center ltx_border_r">0.22</td>
<td id="S5.T7.1.1.6.4.5" class="ltx_td ltx_align_center">0.46</td>
<td id="S5.T7.1.1.6.4.6" class="ltx_td ltx_align_center">0.20</td>
<td id="S5.T7.1.1.6.4.7" class="ltx_td ltx_align_center">0.14</td>
</tr>
<tr id="S5.T7.1.1.7.5" class="ltx_tr">
<th id="S5.T7.1.1.7.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">CodeGemma-Instruct (7B)</th>
<td id="S5.T7.1.1.7.5.2" class="ltx_td ltx_align_center">0.14</td>
<td id="S5.T7.1.1.7.5.3" class="ltx_td ltx_align_center">0.03</td>
<td id="S5.T7.1.1.7.5.4" class="ltx_td ltx_align_center ltx_border_r">0.00</td>
<td id="S5.T7.1.1.7.5.5" class="ltx_td ltx_align_center">0.09</td>
<td id="S5.T7.1.1.7.5.6" class="ltx_td ltx_align_center">0.02</td>
<td id="S5.T7.1.1.7.5.7" class="ltx_td ltx_align_center">0.01</td>
</tr>
<tr id="S5.T7.1.1.8.6" class="ltx_tr">
<th id="S5.T7.1.1.8.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Nexusflow-Raven-v2 (13B)</th>
<td id="S5.T7.1.1.8.6.2" class="ltx_td ltx_align_center">0.41</td>
<td id="S5.T7.1.1.8.6.3" class="ltx_td ltx_align_center">0.16</td>
<td id="S5.T7.1.1.8.6.4" class="ltx_td ltx_align_center ltx_border_r">0.11</td>
<td id="S5.T7.1.1.8.6.5" class="ltx_td ltx_align_center">0.38</td>
<td id="S5.T7.1.1.8.6.6" class="ltx_td ltx_align_center">0.11</td>
<td id="S5.T7.1.1.8.6.7" class="ltx_td ltx_align_center">0.06</td>
</tr>
<tr id="S5.T7.1.1.9.7" class="ltx_tr">
<th id="S5.T7.1.1.9.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">C4AI-Command-R-v01 (35B)</th>
<td id="S5.T7.1.1.9.7.2" class="ltx_td ltx_align_center">0.39</td>
<td id="S5.T7.1.1.9.7.3" class="ltx_td ltx_align_center">0.15</td>
<td id="S5.T7.1.1.9.7.4" class="ltx_td ltx_align_center ltx_border_r">0.07</td>
<td id="S5.T7.1.1.9.7.5" class="ltx_td ltx_align_center">0.39</td>
<td id="S5.T7.1.1.9.7.6" class="ltx_td ltx_align_center">0.15</td>
<td id="S5.T7.1.1.9.7.7" class="ltx_td ltx_align_center">0.06</td>
</tr>
<tr id="S5.T7.1.1.10.8" class="ltx_tr">
<th id="S5.T7.1.1.10.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Meta-Llama-3-70B-Instruct (70B)</th>
<td id="S5.T7.1.1.10.8.2" class="ltx_td ltx_align_center"><span id="S5.T7.1.1.10.8.2.1" class="ltx_text ltx_font_bold">0.69</span></td>
<td id="S5.T7.1.1.10.8.3" class="ltx_td ltx_align_center"><span id="S5.T7.1.1.10.8.3.1" class="ltx_text ltx_font_bold">0.48</span></td>
<td id="S5.T7.1.1.10.8.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S5.T7.1.1.10.8.4.1" class="ltx_text ltx_font_bold">0.47</span></td>
<td id="S5.T7.1.1.10.8.5" class="ltx_td ltx_align_center"><span id="S5.T7.1.1.10.8.5.1" class="ltx_text ltx_font_bold">0.65</span></td>
<td id="S5.T7.1.1.10.8.6" class="ltx_td ltx_align_center"><span id="S5.T7.1.1.10.8.6.1" class="ltx_text ltx_font_bold">0.40</span></td>
<td id="S5.T7.1.1.10.8.7" class="ltx_td ltx_align_center"><span id="S5.T7.1.1.10.8.7.1" class="ltx_text ltx_font_bold">0.40</span></td>
</tr>
<tr id="S5.T7.1.1.11.9" class="ltx_tr">
<th id="S5.T7.1.1.11.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S5.T7.1.1.11.9.1.1" class="ltx_text ltx_font_bold ltx_font_smallcaps">Granite-20B-FunctionCalling</span></th>
<td id="S5.T7.1.1.11.9.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T7.1.1.11.9.2.1" class="ltx_text ltx_framed ltx_framed_underline">0.68</span></td>
<td id="S5.T7.1.1.11.9.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T7.1.1.11.9.3.1" class="ltx_text ltx_framed ltx_framed_underline">0.47</span></td>
<td id="S5.T7.1.1.11.9.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S5.T7.1.1.11.9.4.1" class="ltx_text ltx_font_bold">0.47</span></td>
<td id="S5.T7.1.1.11.9.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T7.1.1.11.9.5.1" class="ltx_text ltx_framed ltx_framed_underline">0.61</span></td>
<td id="S5.T7.1.1.11.9.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T7.1.1.11.9.6.1" class="ltx_text ltx_framed ltx_framed_underline">0.36</span></td>
<td id="S5.T7.1.1.11.9.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T7.1.1.11.9.7.1" class="ltx_text ltx_framed ltx_framed_underline">0.37</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 7: </span>API-Bank Response generation dataset evaluation. Results are averaged across each dataset per model. Best performance is highlighted in <span id="S5.T7.5.1" class="ltx_text ltx_font_bold">bold</span>, second best is <span id="S5.T7.6.2" class="ltx_text ltx_framed ltx_framed_underline">underlined</span>. All evaluations are done in a <span id="S5.T7.7.3" class="ltx_text ltx_font_italic">zero-shot</span> manner.</figcaption>
</figure>
<figure id="S5.F3" class="ltx_figure"><img src="/html/2407.00121/assets/figures/hallucination-ood-new.png" id="S5.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="439" height="247" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Performance vs. Hallucination rates for Out-of-Domain Function Calling</figcaption>
</figure>
</section>
<section id="S5.SS3.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.3.3 </span>Response Generation</h4>

<div id="S5.SS3.SSS3.p1" class="ltx_para ltx_noindent">
<p id="S5.SS3.SSS3.p1.1" class="ltx_p">In Table <a href="#S5.T7" title="Table 7 ‣ 5.3.2 Function Calling Academic Benchmarks ‣ 5.3 Evaluation Results ‣ 5 Experimental Setup and Evaluation ‣ 4.2 Training ‣ 4.1 Training Data Mixture Creation ‣ 4 Instruct Tuning ‣ Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, we show models’ performance on response generation task. We use API-Bank dataset and follow their response generation task evaluation with BertScore, Rouge-L, and BLUE. Meta-Llama-3-70B-Instruct has the best performance across the three metrics with <span id="S5.SS3.SSS3.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> coming in close second (difference in performance ranged between 1-5%). Both models significantly outperform all other evaluated models. The gap is larger when we compare <span id="S5.SS3.SSS3.p1.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> to the ones specifically trained for function calling such as Functionary-small-v2.5 and Gorilla-openfunctions-v2.</p>
</div>
</section>
<section id="S5.SS3.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.3.4 </span>Further Improvements</h4>

<div id="S5.SS3.SSS4.p1" class="ltx_para ltx_noindent">
<p id="S5.SS3.SSS4.p1.1" class="ltx_p">We have instruct-tuned the <span id="S5.SS3.SSS4.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> in such a way that it develops implicit function searching capability from a long list of functions. For example, in out-of-domain evaluation tasks, for ToolAlpaca the model needs to find the Function from a list of 94 functions, similarly, it has access to 15 and 20 functions for ToolBench-B and ToolBench-HS, respectively. Due to this reason, the prompt with all the function libraries increases the context length and <span id="S5.SS3.SSS4.p1.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-Code-Instruct</span> supports up to 8192 context length, so we were not able to add the full signature of each Function in the library. Currently, each function in the library contains a function name, a description, and for each function, we have provided a list of respected arguments with their descriptions. However, to fit the prompt in the max context-length, we had to remove the type, required-fields, and optional-fields values for each argument from the specifications. For further performance exploration, we will assess options to include the entire function specification without truncation including exploiting the benefit of Rotary Position Embedding <cite class="ltx_cite ltx_citemacro_citep">(Su et al., <a href="#bib.bib39" title="" class="ltx_ref">2023</a>)</cite> and its innate support for longer context lengths within some of the other Granite models.</p>
</div>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para ltx_noindent">
<p id="S6.p1.1" class="ltx_p">In this paper, we introduced <span id="S6.p1.1.1" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span>, a capable function calling open model with Apache 2 license. <span id="S6.p1.1.2" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> is trained using a suite of datasets transformed from semantic parsing, task-oriented dialog, personal assistants and conversational domains. The training setup is a multi-task learning approach where granular tasks in function calling such as function detection, parameter detection, sequencing, and next best function are used for instruction tuning the model. We performed an extensive evaluation of <span id="S6.p1.1.3" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> in comparison to other state-of-the-art function calling models. On multiple out-of-domain datasets, including Berkeley Function Calling Leaderboard, <span id="S6.p1.1.4" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> provides the best performance among the models that have open licenses. Even compared to multiple proprietary models with much larger sizes, <span id="S6.p1.1.5" class="ltx_text ltx_font_smallcaps">Granite-20B-FunctionCalling</span> showed on-par and in some cases better performance on multiple datasets and tasks.</p>
</div>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">AI@Meta (2024)</span>
<span class="ltx_bibblock">
AI@Meta.

</span>
<span class="ltx_bibblock">Llama 3 model card.

</span>
<span class="ltx_bibblock">2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md</a>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Basu et al. (2024)</span>
<span class="ltx_bibblock">
Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A. Lastras.

</span>
<span class="ltx_bibblock">Api-blend: A comprehensive corpora for training and benchmarking api llms, 2024.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2024a)</span>
<span class="ltx_bibblock">
Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang.

</span>
<span class="ltx_bibblock">Advancing tool-augmented large language models: Integrating insights from errors in inference trees, 2024a.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2024b)</span>
<span class="ltx_bibblock">
Wei Chen, Zhiyuan Li, and Mingyuan Ma.

</span>
<span class="ltx_bibblock">Octopus: On-device language model for function calling of software apis, 2024b.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chung et al. (2022)</span>
<span class="ltx_bibblock">
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.

</span>
<span class="ltx_bibblock">Scaling instruction-finetuned language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.11416</em>, 2022.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">CodeGemma Team et al. (2024)</span>
<span class="ltx_bibblock">
CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, and Hui.

</span>
<span class="ltx_bibblock">Codegemma: Open code models based on gemma.

</span>
<span class="ltx_bibblock">2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://goo.gle/codegemma" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://goo.gle/codegemma</a>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">CohereForAI (2024)</span>
<span class="ltx_bibblock">
CohereForAI.

</span>
<span class="ltx_bibblock">C4ai command-r: A 35 billion parameter generative model for reasoning, summarization, and question answering.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Hugging Face Models</em>, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://huggingface.co/CohereForAI/c4ai-command-r-v01" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/CohereForAI/c4ai-command-r-v01</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Crouse et al. (2023)</span>
<span class="ltx_bibblock">
Maxwell Crouse, Ibrahim Abdelaziz, Kinjal Basu, Soham Dan, Sadhana Kumaravel, Achille Fokoue, Pavan Kapanipathi, and Luis Lastras.

</span>
<span class="ltx_bibblock">Formally specifying the high-level behavior of llm-based agents.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.08535</em>, 2023.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2023)</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms, 2023.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2023)</span>
<span class="ltx_bibblock">
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.

</span>
<span class="ltx_bibblock">Pal: Program-aided language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp.  10764–10799. PMLR, 2023.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guo et al. (2024)</span>
<span class="ltx_bibblock">
Zhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen, and Rameswar Panda.

</span>
<span class="ltx_bibblock">Api pack: A massive multi-programming language dataset for api call generation, 2024.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gur et al. (2023)</span>
<span class="ltx_bibblock">
Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust.

</span>
<span class="ltx_bibblock">A real-world webagent with planning, long context understanding, and program synthesis.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning Representations</em>, 2023.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He-Yueya et al. (2023)</span>
<span class="ltx_bibblock">
Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman.

</span>
<span class="ltx_bibblock">Solving math word problems by combining language models with symbolic solvers.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.09102</em>, 2023.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jain et al. (2024)</span>
<span class="ltx_bibblock">
Arushi Jain, Shubham Paliwal, Monika Sharma, Lovekesh Vig, and Gautam Shroff.

</span>
<span class="ltx_bibblock">Smartflow: Robotic process automation using llms, 2024.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2023)</span>
<span class="ltx_bibblock">
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.

</span>
<span class="ltx_bibblock">Mistral 7b.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.06825</em>, 2023.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Komeili et al. (2022)</span>
<span class="ltx_bibblock">
Mojtaba Komeili, Kurt Shuster, and Jason Weston.

</span>
<span class="ltx_bibblock">Internet-augmented dialogue generation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  8460–8478, 2022.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023a)</span>
<span class="ltx_bibblock">
Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter.

</span>
<span class="ltx_bibblock">Chain of code: Reasoning with a language model-augmented code emulator.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.04474</em>, 2023a.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2024)</span>
<span class="ltx_bibblock">
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al.

</span>
<span class="ltx_bibblock">Synthetic data (almost) from scratch: Generalized instruction tuning for language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.13064</em>, 2024.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023b)</span>
<span class="ltx_bibblock">
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.

</span>
<span class="ltx_bibblock">Api-bank: A comprehensive benchmark for tool-augmented llms, 2023b.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin (2004)</span>
<span class="ltx_bibblock">
Chin-Yew Lin.

</span>
<span class="ltx_bibblock">Rouge: A package for automatic evaluation of summaries.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Text summarization branches out</em>, pp.  74–81, 2004.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">MeetKai (2024)</span>
<span class="ltx_bibblock">
MeetKai.

</span>
<span class="ltx_bibblock">Functionary-7b-v1.4: A language model for function interpretation and execution.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Hugging Face Models</em>, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://huggingface.co/meetkai/functionary-7b-v1.4" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/meetkai/functionary-7b-v1.4</a>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mishra et al. (2024)</span>
<span class="ltx_bibblock">
Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al.

</span>
<span class="ltx_bibblock">Granite code models: A family of open foundation models for code intelligence.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2405.04324</em>, 2024.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mishra et al. (2022)</span>
<span class="ltx_bibblock">
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Cross-task generalization via natural language crowdsourcing instructions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  3470–3487, 2022.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Muennighoff et al. (2023)</span>
<span class="ltx_bibblock">
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.

</span>
<span class="ltx_bibblock">Crosslingual generalization through multitask finetuning.

</span>
<span class="ltx_bibblock">In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  15991–16111, Toronto, Canada, July 2023. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.acl-long.891</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2023.acl-long.891" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2023.acl-long.891</a>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nexusflow.ai (2023)</span>
<span class="ltx_bibblock">
Nexusflow.ai.

</span>
<span class="ltx_bibblock">Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://nexusflow.ai/blogs/ravenv2" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://nexusflow.ai/blogs/ravenv2</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nous-Research (2023)</span>
<span class="ltx_bibblock">
Nous-Research.

</span>
<span class="ltx_bibblock">Nous-hermes-13b.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/NousResearch/Nous-Hermes-13b" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/NousResearch/Nous-Hermes-13b</a>, 2023.

</span>
<span class="ltx_bibblock">Model on Hugging Face.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ouyang et al. (2022)</span>
<span class="ltx_bibblock">
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.

</span>
<span class="ltx_bibblock">Training language models to follow instructions with human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:27730–27744, 2022.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Papineni et al. (2002)</span>
<span class="ltx_bibblock">
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.

</span>
<span class="ltx_bibblock">Bleu: a method for automatic evaluation of machine translation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 40th annual meeting of the Association for Computational Linguistics</em>, pp.  311–318, 2002.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Parisi et al. (2022)</span>
<span class="ltx_bibblock">
Aaron Parisi, Yao Zhao, and Noah Fiedel.

</span>
<span class="ltx_bibblock">Talm: Tool augmented language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2205.12255</em>, 2022.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Patel et al. (2021)</span>
<span class="ltx_bibblock">
Arkil Patel, Satwik Bhattamishra, and Navin Goyal.

</span>
<span class="ltx_bibblock">Are NLP models really able to solve simple math word problems?

</span>
<span class="ltx_bibblock">In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em>, pp.  2080–2094, Online, June 2021. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.naacl-main.168</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2021.naacl-main.168" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2021.naacl-main.168</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Patil et al. (2023)</span>
<span class="ltx_bibblock">
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez.

</span>
<span class="ltx_bibblock">Gorilla: Large language model connected with massive apis.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.15334</em>, 2023.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Qin et al. (2023)</span>
<span class="ltx_bibblock">
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.

</span>
<span class="ltx_bibblock">Toolllm: Facilitating large language models to master 16000+ real-world apis.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.16789</em>, 2023.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rafailov et al. (2023)</span>
<span class="ltx_bibblock">
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.

</span>
<span class="ltx_bibblock">Direct preference optimization: Your language model is secretly a reward model, 2023.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Reid et al. (2024)</span>
<span class="ltx_bibblock">
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.

</span>
<span class="ltx_bibblock">Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.05530</em>, 2024.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Roziere et al. (2023)</span>
<span class="ltx_bibblock">
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.

</span>
<span class="ltx_bibblock">Code llama: Open foundation models for code.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.12950</em>, 2023.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schick et al. (2023)</span>
<span class="ltx_bibblock">
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.

</span>
<span class="ltx_bibblock">Toolformer: Language models can teach themselves to use tools.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2302.04761, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:256697342" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:256697342</a>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shinn et al. (2023)</span>
<span class="ltx_bibblock">
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao.

</span>
<span class="ltx_bibblock">Reflexion: Language agents with verbal reinforcement learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Thirty-seventh Conference on Neural Information Processing Systems</em>, 2023.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Song et al. (2023)</span>
<span class="ltx_bibblock">
Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li.

</span>
<span class="ltx_bibblock">Restgpt: Connecting large language models with real-world restful apis, 2023.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Su et al. (2023)</span>
<span class="ltx_bibblock">
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.

</span>
<span class="ltx_bibblock">Roformer: Enhanced transformer with rotary position embedding.

</span>
<span class="ltx_bibblock">2023.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sudalairaj et al. (2024)</span>
<span class="ltx_bibblock">
Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D Cox, and Akash Srivastava.

</span>
<span class="ltx_bibblock">Lab: Large-scale alignment for chatbots.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.01081</em>, 2024.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tang et al. (2023)</span>
<span class="ltx_bibblock">
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun.

</span>
<span class="ltx_bibblock">Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.05301</em>, 2023.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023)</span>
<span class="ltx_bibblock">
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Self-instruct: Aligning language models with self-generated instructions.

</span>
<span class="ltx_bibblock">In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.acl-long.754</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2023.acl-long.754" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2023.acl-long.754</a>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2021)</span>
<span class="ltx_bibblock">
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.

</span>
<span class="ltx_bibblock">Finetuned language models are zero-shot learners.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2021.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2023a)</span>
<span class="ltx_bibblock">
Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu.

</span>
<span class="ltx_bibblock">Rewoo: Decoupling reasoning from observations for efficient augmented language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.18323</em>, 2023a.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2023b)</span>
<span class="ltx_bibblock">
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang.

</span>
<span class="ltx_bibblock">On the tool manipulation capability of open-source large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.16504</em>, 2023b.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yan et al. (2024)</span>
<span class="ltx_bibblock">
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez.

</span>
<span class="ltx_bibblock">Berkeley function calling leaderboard.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html</a>, 2024.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2024a)</span>
<span class="ltx_bibblock">
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.

</span>
<span class="ltx_bibblock">Swe-agent: Agent-computer interfaces enable automated software engineering, 2024a.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2024b)</span>
<span class="ltx_bibblock">
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.

</span>
<span class="ltx_bibblock">Gpt4tools: Teaching large language model to use tools via self-instruction.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024b.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2023)</span>
<span class="ltx_bibblock">
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.

</span>
<span class="ltx_bibblock">Mm-react: Prompting chatgpt for multimodal reasoning and action.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.11381</em>, 2023.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2022)</span>
<span class="ltx_bibblock">
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.

</span>
<span class="ltx_bibblock">React: Synergizing reasoning and acting in language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations</em>, 2022.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2019)</span>
<span class="ltx_bibblock">
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.

</span>
<span class="ltx_bibblock">Bertscore: Evaluating text generation with bert.

</span>
<span class="ltx_bibblock"><em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1904.09675</em>, 2019.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</section>
</section>
</section>
</div>
</div>
</figure>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.00119" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.00121" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.00121">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.00121" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.00122" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 13:36:36 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.00121
Large language models (LLMs)  have recently shown tremendous promise in serving as the backbone to agentic systems, as demonstrated by their performance in multi-faceted, challenging benchmarks like SWE-Bench and Agent-Bench.
However, to realize the true potential of LLMs as autonomous agents, they must learn to identify, call, and interact with external tools and application program interfaces (APIs)  to complete complex tasks. These tasks together are termedfunction calling.
Endowing LLMs with function calling abilities leads to a myriad of advantages, such as access to current and domain-specific information in databases and knowledge sources, and the ability to outsource tasks that can be reliably performed by tools, e.g., a Python interpreter or calculator.
While there has been significant progress in function calling with LLMs, there is still a dearth of open models that perform on par with proprietary LLMs like GPT, Claude, and Gemini.
Therefore, in this work, we introduce theGranite-20B-FunctionCalling111The model will be available soon athttps://huggingface.co/ibm-granite/model under an Apache 2.0 license. The model is trained using a multi-task training approach on seven fundamental tasks encompassed in function calling, those being Nested Function Calling, Function Chaining, Parallel Functions, Function Name Detection, Parameter-Value Pair Detection, Next-Best Function, and Response Generation. We present a comprehensive evaluation on multiple out-of-domain datasets comparingGranite-20B-FunctionCallingto more than 15 other best proprietary and open models.Granite-20B-FunctionCallingprovides the best performance among all open models on the Berkeley Function Calling Leaderboard and fourth overall.
As a result of the diverse tasks and datasets used for training our model, we show thatGranite-20B-FunctionCallinghas better generalizability on multiple tasks in seven different evaluation datasets.
abstract
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
Abstract
2407.00121
1IntroductionLarge language models (LLMs)  have garnered significant attention due to their broad applicability to an important set of challenging domains, e.g., programming, reasoning, and multi-modal interaction. Increasingly, applying these models to solve real-world problems requires them to act as autonomous agents powering intelligent decision-making in specific environments222Auto-GPT:https://github.com/Significant-Gravitas/AutoGPT333BabyAGI:https://github.com/yoheinakajima/babyagi. For LLMs to serve as autonomous agents, they must perform accurately on two fundamental capabilities: (a)  reasoning and planning, and (b)  function calling, which includes identifying, calling, and interacting with tools and APIs in external environments. In this work, we focus on improving LLMs’ function calling abilities.Function calling provides a means for language models to leverage external tools and resources.
These tools can make available to an LLM specific, up-to-date information that would otherwise be inaccessible (e.g., stored in a dynamic knowledge base)  and thus reduce its proclivity for hallucinating responses. This is particularly crucial in enterprise use cases where a significant portion of relevant data is stored in a structured format accessible only via storage engines. In addition to knowledge access, function calling can allow an LLM to outsource tasks that are out of scope for a generalized language model. Most commonly, these tasks involve compute-heavy operations, e.g., program execution, numerical calculation, or retrieval, and are otherwise a frequent source of LLM hallucinations.The importance of function calling has spurred the development of several recent
data generation efforts for fine-tuningand evaluation of models. Typically, however, the fine-tuned models from datasets like ToolLLM, ToolAlpaca, and Gorillaunderperform in one (or more)  of three key dimensions: (a) Generalizability:While the datasets are generated using diverse sets of APIs (e.g., ToolLLama uses RapidAPIs444https://rapidapi.com/hub, ToolAlpaca uses public APIs555https://github.com/public-apis/public-apis, and Gorilla uses TensorFlow Hub, PyTorch Hub, and Hugging Face Hub) , work fromhas shown that models trained on these datasets have difficulty generalizing to out-of-domain datasets. (b) Granular tasks:Function calling, as an umbrella term, can encompass multiple granular sub-tasks such as function-name detection, slot filling666Slot,parameter, andargumentare used interchangeably.or parameter-value pair detection, and detecting the ordered sequence of functions needed to be called. Existing models trained to perform function calling lack the ability to handle these granular tasks independently, and hence, perform poorly on such sub-tasks. (c) Openness:The best performing models are proprietary and the ones that have open licenses (e.g., Gorilla)  are trained using data generated from OpenAI models.Figure 1:Step-by-step building process ofGranite-20B-FunctionCallingTo address the aforementioned limitations, in this work, we focus on introducing function-calling abilities to models with an inherent focus on granular tasks. Figure1shows an overview of howGranite-20B-FunctionCallingwas trained. The datasets used for training are API-Blendthat include tasks such as function name detection, slot filling, parallel functions, multiple functions, sequencing777Sequencingandchainingare used interchangeably., and calling APIs888FunctionandAPIare used interchangeably.using multiple programming languages.
We build upon Granite code models by instruction tuning them for function calling using the datasets for granular tasks with a multi-task learning approach. Granite code models are trained on data that is license permissible following IBM’s AI Ethics principles for trustworthy enterprise usage. Being part of the Granite family, we releaseGranite-20B-FunctionCallingunder Apache 2.0 license. Finally, in this work, we perform a comprehensive evaluation of the open and proprietary models using Berkeley Function Calling Leaderboard (BFCL) , four Function Calling Academic Benchmarks, and Response Generation Benchmark from API-Bankto evaluate the generalizability of function-calling models.Granite-20B-FunctionCallingis on par with the best open model on BFCL and fourth overall. Furthermore, compared to other models based on the out-of-domain datasets,Granite-20B-FunctionCallingshows significant generalizability.
Figure2shows howGranite-20B-FunctionCallingcompares to the top two open models (according to BFCL)  on various tasks where despite only having 20B parameters, it performs as well or better than Meta-Llama-3-70B-Instruct which has 70B parameters.Figure 2:Evaluation ofGranite-20B-FunctionCallingagainst the bestopenfunction calling models (according to BFCL) 
introduction
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
1Introduction
2407.00121
2Related Work2.1Instruction TuningOur work is an instantiation ofinstruction tuning, a fine-tuning method that improves an LLM’s ability to solve natural language tasks. It involves taking a large collection of NLP datasets, reformulating those datasets into a set of instruction-following tasks, and then fine-tuning an LLM on the modified data. While the earliest versions of instruction tuning straightforwardly combined large datasets together, the most recent iterations use more sophisticated mixtures of tasks to achieve the best results. Our work draws largely upon API-Blendand API Pack, two recently introduced instruction-tuning datasets specifically focused on tasks related to APIs, e.g., slot filling and API intent detection.Instruction-tuned models often significantly outperform their base models on a wide range of tasks, particularly in the zero-shot setting. Further improvement has been observed through alignment of the model after instruction-fine-tuning, e.g., TP-LLaMAuses Direct Preference Optimizationin addition to fine-tuning.There is also growing interest in developing smaller models that match or surpass the accuracy of larger proprietary LLMs in function calling tasks. These compact models are crucial in the increasing prevalence of on-device LLMs, enabling efficient and effective performance on local devices.2.2Function Calling by LLMsFunction calling augmentation has broadened the scope of problems addressable by LLMs to include those that cannot be solved with internal knowledge alone. For instance, prior work has demonstrated the use of API-enhanced LLMs to solve problems requiring up-to-date information retrieval, intricate mathematical calculations, internet use, task orchestration, and even programming abilities.Multiple strategies have been proposed for how best to enable LLM function calling. One line of prior research has investigated the design of elaborate prompting approaches, best exemplified by the popular ReACT prompting framework. Such prompting methods can vary in their design, with some works optimizing for cost, raw performance, or a blend of both. More relevant to our approach are methods that train models to directly output function calls. Typically, these works will use some form of self-supervision to enable scaling to the breadth of domains required for general-purpose function use.Recently, many language models with function-calling capabilities have been introduced. They broadly fall into two categories: pre-trained models which are capable of function-calling, and models fine-tuned specifically for function-calling. While the pre-trained models enable function-calling using a combination of supervised and preference fine-tuning, details of the datasets used to train models for these tasks are not generally available. On the other hand, specialized function-calling models mostly rely on synthetic data generated from proprietary state-of-the-art models. Models like Gorilla, ToolLlama, ToolAlpaca, and the NousResearch Hermes series of modelsutilize GPT-4 or ChatGPT to generate synthetic instruction tuning datasets and fine-tune a base model such as the Llama or Mistral model for function-calling tasks. The NexusRaven modelsare one of the few open-source models that focus on building function-calling models for commercial purposes by avoiding using proprietary models for synthetic data generation.In section5, we compare our model to the above models and show thatGranite-20B-FunctionCallingprovides the best or comparable performance amongst all open models across multiple tasks.
body
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
2Related Work
2407.00121
3Multi-Task Training DataIn this section, we describe our detailed approach to fine-tune theGranite-20B-Code-Instructmodel with multi-task data related to functions to buildGranite-20B-FunctionCalling, a robust model designed for function-calling. We use API-BLEND, a diverse corpora of multiple API datasets for training LLMs. It consists of five datasets with a total of about 160K training examples: SeqSGD, SeqSNIPS, SeqTopV2, SeqATIS, and SeqMultiWOZ.A key contribution to the process of buildingGranite-20B-FunctionCallingis multi-task training, where we reuse the same data in different formats with distinct instructions for different function-calling related tasks. We have identified six underlying sub-tasks for function calling and divided them into two broad categories based on the difficulty levels: (A) High-Level Function Calling Taskswhich are complex tasks for an LLM and typically handle multiple functions; and (B) Low-Level Function Calling Taskswhich are simpler tasks for an LLM and relate to either function names or only parameter-value pairs. We have included“Response Generation”as the seventh task in our training data since producing natural language responses is one of the fundamental goals of an LLM. Table1demonstrates the task-wise mapping of each dataset. Below, we describe each task in detail.High-Level Function Calling TasksLow-Level Function Calling TasksDatasetsNested Func.CallingFunc.ChainingParallelFunc.Next-BestFunc.Func. NameDetectionParam-ValPair DetectionResponseGenerationSeqSGD✔✔✔✔✔SeqSNIPS✔✔✔✔✔SeqTopV2✔✔✔✔✔SeqATIS✔✔✔✔✔SeqMultiWOZ✔✔✔✔Glaive-V2✔✔Table 1:Training Datasets with Task mappingIn the rest of the section, we describe how to unify the data of different datasets in the same format for model training and then describe each of these training tasks.3.1Data UnificationIn addition to the datasets in API-BLEND(i.e., SeqSGD, SeqSNIPS, SeqTopV2, SeqATIS, SeqMultiWOZ) , we also use Glaive-V2999https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2to prepare the training data mixture forGranite-20B-FunctionCalling, where each dataset is multi-purposed for different function calling related tasks with different instructions. These datasets come from different sources and have various function formats that require unification to an identical representation for better usability.In this unification process, we convert all the APIs, tools, and functions from the data into a JSON format representation. We choose this format because (a)  JSON is a language-independent, human-readable, and widely used data format for code-related tasks; (b)  it is easily parsable to insert/extract information; and (c)  many web services, APIs, and tools accept JSON objects and generate responses in JSON format.InGranite-20B-FunctionCalling, we unify the modeloutputrepresentation of function calls to the following format:⬇{"name":"<FUNCTION-NAME>","arguments":{"<PARAMETER-1>":"VALUE-1","<PARAMETER-2>":...}}With a similar JSON representation, we express the functions in the library to be passed as input to the model:⬇{"name":"<FUNCTION-NAME>","description:"<FUNCTION-DESCRIPTION>""arguments":{"<PARAMETER-1>":{"description":"...",...}...}}3.2High-Level Function Calling TasksIn general, these tasks are challenging for LLMs to accomplish since they require the LLM to generate multiple function calls with the parameters and their values. In the following sections, we describe different types of high-level function-calling tasks with examples.Nested Function CallingThe main characteristic of this task is in the output function sequence, where the current function’s output becomes an input to the next function. So, the answer to a user query is a sequence of nested function calls selected from the function library. Furthermore, the parameters of these function calls need to be filled by extracting the values from the user query. The example below lists two functions in the output under<function_call>tag101010In all of our examples, we use<function_call>tag to represent a function in JSON form.and the output of the first function will populate thedestinationparameter in the second function.⬇{"input":"What’sthetypicaldrivingtimebetweenLasVegasandtheGrandCanyon?","output:"<function_call>{"name":"get_location","arguments":{"point_on_map":"theGrandCanyon"}}<function_call>{"name":"get_estimated_duration","arguments":{"source":"LasVegas","method_travel":"driving","destination":"<function_response>get_location"}}"}Function ChainingIn this task, a model needs to call multiple functions in a sequence to answer a user query. However, unlike Nested Function Calling, these functions do not have to be nested. Also, for each function, the parameters whose values are present in the user query must be passed as arguments. The following example demonstrates how three functions must be called in sequence to answer the query.⬇{"input":"FindaproviderinSF.GettheweatherinSFonMarch3rd.BookanappointmentwithAbusinessman’shaircutonMarch3rdat18:15.","output:"<function_call>{"name":"find_provider","arguments":{"city":"SF"}}<function_call>{"name":"get_weather","arguments":{"city":"SF","date":"march3rd"}}<function_call>{"name":"book_appointment","arguments":{"appointment_date":"march3rd","appointment_time":"18:15","stylist_name":"Abusinessman’shaircut"}}"}Parallel FunctionsSimilar to the Function Chaining task, here, the answer to a user query requires the same function to be called multiple times (in parallel) . Also, the parameters should be mapped with the values extracted from the user query. In the below example, theget_weatherfunction has been called twice with different parameter-value pairs to answer the user query.⬇{"input":"Whatistheweatherlikeinthenext3daysinNewYorkandinthenext5daysinLosAngeles?","output:"<function_call>{"name":"get_weather","arguments":{"current_location":"NewYork","time_range":"next3days"}}<function_call>{"name":"get_weather","arguments":{"current_location":"LosAngeles","time_range":"next5days"}}"}3.3Low-Level Function Calling TasksCompared to High-Level tasks, these tasks are simpler because they either request function names (without parameters)  or parameter-value pairs for a function. To excel in High-Level function calling tasks, it is crucial for any LLM to master these foundational sub-tasks. In what follows, we demonstrate three types of low-level function-calling tasks.Next-Best FunctionIn this task, given the function library along with the user query and the partial function sequence, the models are supposed to select the next most suitable function from the function library. It only requires the model to choose one function name without any parameters. The following example demonstrates how the partial function sequence (provided under<|partial_function_call_list|>)  can be passed with the query and the model is supposed to predict the next best function.⬇{"input":"GetthedatesforRaidersVsLionsinOakland.Buy4ticketsforRaidersVsLionsinOaklandtomorrow.<|partial_function_call_list|><function_call>{"name":"get_event_dates","arguments":{}}","output:"<function_call>{"name":"buy_event_tickets","arguments":{}}"}Function Name DetectionThis task expects the model to produce only the sequence of function names (without parameters)  from the function library that are required to answer the user query. This task closely resembles Function Chaining (a High-Level task) , with the sole distinction being it does not necessitate the model to populate the function’s arguments. Below is an example of this task.⬇{"input":"Findthehotelnamedautumnhouse.BookahotelnamedAutumnHousefor2peoplefor4daysonSunday.Findarestaurantinsouthwithnametajtandooriandpricerangeexpensive.","output:"<function_call>{"name":"find_hotel","arguments":{}}<function_call>{"name":"book_hotel","arguments":{}}<function_call>{"name":"find_restaurant","arguments":{}}"}Parameter-Value Pair DetectionIn this task, when provided with a user query or a user-agent conversation along with a list of parameters and their descriptions, the model must identify all the parameters for which the values are present in the query or conversation. See an example of this task below. To keep the data format consistent and JSON parsable, we use the function name -“dummy”for all the data under this task.⬇{"input":"GetaluxuryridetoGreentreeApartmentswith1seat.","output:"<function_call>{"name":"dummy","arguments":{"destination":"GreentreeApartments","number_of_seats":"1","ride_type":"Luxury"}}"}3.4Response GenerationNatural language response generation is a crucial feature of any LLM. In this task, the model must comprehend an ongoing conversation between a user and an AI assistant. Then, it generates a natural language response, answering the most recent user utterance. Such responses are needed to chit-chat with the user, ask clarifying questions, or synthesize a function call’s output into a natural language response.⬇{"USER":"Ihavesomechicken,broccoli,andcheese.WhatcanImakewiththese?","AGENT":"<function_call>{"name":"search_recipes","arguments":{"ingredients":["chicken","broccoli","cheese"]}}<function_response>{"recipes":[{"name":"ChickenandBroccoliCasserole","ingredients":["chicken","broccoli","cheese"],"instructions":"Preheatovento350degreesF(175degreesC) .Inalargebowl,mixtogetherchicken,broccoli,andcheese.Bakeinpreheatedovenfor25to30minutes,oruntilcheeseismeltedandbubbly."},{"name":"CheesyChickenandBroccoliPasta","ingredients":["chicken","broccoli","cheese"],"instructions":"Cookpastaaccordingtopackageinstructions.Inalargeskillet,cookchickenuntilnolongerpink.Addbroccoliandcheese,cookuntilcheeseismelted."}]}""ASSISTANT":"Hereareacoupleofrecipesyoucanmakewithchicken,broccoli,andcheese:..."}
body
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
3Multi-Task Training Data
2407.00121
4Instruct Tuning4.1Training Data Mixture CreationAfter generating the data for various tasks, the next step is to create a training data mixture including all the data. We programmatically generate the mixture of data by following a weighted configuration for datasets and tasks. Following is anexampleof the weighted configuration, where the total mixture samples will be divided between Function Chaining and Next-Best Function in a 3:5 ratio. Within the Function Chaining portion, the allocation is split between SeqSGD and Glaive-V2 in a 2:3 ratio. Similarly, the Next-Best Function chunk will be divided in a 2:1 ratio between SeqTopV2 and SeqSNIPS.⬇[{"instruction_name":"FunctionChaining","datasets":{"SeqSGD":2,"Glaive-V2":3},"weight":3},{"instruction_name":"Next-BestFunction","datasets":{"SeqTopV2":2,"SeqSNIPS":1},"weight":5}]TaskInstructionNested Function CallingFunction ChainingParallel Functions⬇SYSTEM:Youareahelpfulassistantwithaccesstothefollowingfunctioncalls.Yourtaskistoproduceasequenceoffunctioncallsnecessarytogenerateresponsetotheuserutterance.Usethefollowingfunctioncallsasrequired.\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\nUSER:{QUERY}\nASSISTANT:Next-Best Function⬇SYSTEM:Youareahelpfulassistantwithaccesstothefollowingfunctioncalls.Yourtaskistoproducethenextfunctioncallnecessarytogenerateresponsetotheuserutterancegiventhepartialfunctionlist.Usethefollowingfunctioncallsasrequiredandreturnonlyfunction"name"withempty"arguments"dictionaryinyourresponse.Onceallthenecessaryfunctionsarecalled,pleasereturn"<|endoftext|>".\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\nUSER:{QUERY}\nASSISTANT:Function Name Detection⬇SYSTEM:Youareahelpfulassistantwithaccesstothefollowingfunctioncalls.Yourtaskistoproduceasequenceoffunctioncallsnecessarytogenerateresponsetotheuserutterance.Usethefollowingfunctioncallsasrequiredandreturnonlyfunction"name"withempty"arguments"dictionaryinyourresponse.Ifnofunctionisrelevant,pleasereturn"<no_function_call>"followedby"<|endoftext|>".\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\nUSER:{QUERY}\nASSISTANT:Parameter-Value Pair Detection⬇SYSTEM:Youareahelpfulassistantwithaccesstothefollowingfunctioncalls.Yourtaskistofindallthenecessaryargumentsandtheirvaluesfromtheuserutterancetogenerateresponse.Usethefollowingfunctioncallsasrequiredandfillonlytheargumentswhosevaluesarepresentintheuserutterance.\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\nUSER:{QUERY}\nASSISTANT:Response Generation⬇SYSTEM:Youareahelpfulassistantwithaccesstothefollowingfunctioncalls.YourtaskistounderstandthegivenconversationwithfunctioncallsandresponsesandgeneratenaturallanguageresponseastheASSISTANTtocontinuetheconversation.Youmayusethefollowingfunctioncallstounderstandhowtorespondtotheuserquery.\n<|function_call_library|>\n{API_SPEC_INSTRUCTION}\n\n{CONV}\nASSISTANT:Table 2:Task specific instructionsAlso, in this step, the training data is embedded with the instructions. These instructions are based on the tasks associated with the data. Table4.1showcases all the instructions (task-wise)  we have used in our training. The “<|function_call_library|>” tag has been used for the function library demonstrated in the prompt with the placeholder named -{API_SPEC_INSTRUCTION}. As the name suggests, the{QUERY}and{CONVERSATION}serve as placeholders for user queries or a user-agent conversation, respectively.4.2TrainingGranite-20B-FunctionCallingis instruct-tuned version ofGranite-20B-Code-Instruct111111https://huggingface.co/ibm-granite/granite-20b-code-instruct. For training data, we created a mixture of 142K examples spanning all the tasks’ datasets discussed above. We then trained our model using QLoRA fine-tuningbased on our multi-task training mixture discussed above. In particular, we trainedGranite-20B-FunctionCallinga QLoRA rank of 8, alpha of 32 and a dropout of 0.1. We also used a learning rate of 5e-5 and ApexFusedAdam as our optimizer with a linear learning rate scheduler. Training was done using a single node of 8 A100_80GB GPUs with 800GB of RAM for a total of 3 epochs.5Experimental Setup and EvaluationIn the section below, we detail our extensive evaluation on various evaluation datasets and public leaderboard. We provide a comprehensive comparison of ourGranite-20B-FunctionCalling, open sourced with Apache 2.0 license, to other open and proprietary function calling models.5.1DatasetsThe evaluation datasets and leaderboards for function calling are gaining a lot of traction in the recent past. In particular, to evaluate the models’ generalizability, we evaluatedGranite-20B-FunctionCallingon a variety of function calling benchmarks, all of which are out-of-domain evaluation for our model. It is worth noting that some of these datasets; e.g. ToolAlpaca and ToolLLM, have training data releases. However, wedid not useany of these benchmarks to trainGranite-20B-FunctionCallingand we only used the datasets in1.121212We could not verify whether some (or all)  of the out-of-domain datasets were used in other models’ training sets.Table3depicts the details of the evaluation datasets we used. We list the details of each of these evaluation datasets below.DatasetTest InstancesTesting tasksMetricsBFCL1,700Function CallingAST, Execution AccuracyRelevancyAccuracyToolLLM491Function CallingFunc. matching (F1) RestGPT157Function CallingFunc. matching (F1) API-Bank473Function CallingFunc. and Param. matching (F1) 478Response GenerationBERTscore, ROUGE, BLEUToolBench214Function CallingFunc. and Param. matching (F1) ToolAlpaca100Function CallingFunc. and Param. matching (F1) NexusRaven318Function CallingFunc. and Param. matching (F1) Table 3:Evaluation Datasets(1) Berkeley Function-Calling Leaderboard (BFCL) 131313BFCL:https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.htmlis a comprehensive function calling leaderboard that includes a dataset of over 1,700 instances. The leaderboard evaluates tasks that include (a)  Simple Function, Multiple Function, Parallel Function, and Parallel Multiple Function for Python Language; and (b)  for non-Python, they evaluate function relevance detection, REST API, JavaScript, and Java. 
(2) ToolBenchis a subset of the data in ToolBench (as released by the authors)  focused on HomeSearch and Booking domains. 
(3) ToolLLM141414ToolLLM also calls their benchmark ToolBench. To disambiguate, in this paper we use the term ToolLLM to refer to their benchmark dataset.is synthetically generated using ChatGPT. The approach uses an initial collection of 16,000 APIs from RapidAPI151515https://rapidapi.com/for synthetic data generation. The evaluation is done on the three test sets categorized based on complexity; G1 – single-tool, G2 – intra-category multi-tool, G3 – intra-collection multi-tool. 
(4) RestGPTis a function calling dataset that has 157 test examples with 85 APIs from Spotify and TMDB. This dataset focuses only on testing model’s ability to detect function names. 
(5) API-Bankhas 314 tool-use dialogues with 753 API calls to assess LLMs’ capabilities in planning, retrieving, and calling APIs. 
(6) ToolAlpacais a synthetic data generation approach that has both training and evaluation benchmarks. It contains 271 tool-use instances spanning 50 distinct categories. Similar to, we used the simulated part of ToolAlpaca which has a total of 100 test examples. 
(7) NexusRaven API Evaluation161616https://huggingface.co/datasets/Nexusflow/NexusRaven_API_evaluationis another function calling dataset with 318 test examples covering a total of 65 different APIs.5.2Evaluation MetricsBelow, we define the metrics we adopted for specific tasks in function calling.BFCL Metrics171717https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html#metrics:BFCL evaluates multiple tasks using the following four metrics. 
(1) AST summarycompares the abstract syntax tree of the function output to the ground truth and the function definition. It captures the correctness of the functions called, their parameters (required or not) , and the parameter types. 
(2) Execution Summarycompares the execution output from generated and ground-truth function calls. This metric is used to evaluate REST APIs and non-REST data samples. 
(3) Relevanceevaluates the model’s ability to detect no function calls when the given list of functions is irrelevant to the user query. This inversely captures the hallucination rate of models. 
(4) Overall Accuracyis the weighted average of all individual data splits in BFCL.The same metrics described above cannot be used for our out-of-domain datasets because of missing information, varied formats, and response generation task. For example, ToolLLM datasets has missing arguments, ToolAlpaca has missing argument types, and API-Bank has response generation task. Therefore, we use the following metrics to evaluate the models on other datasets:F1 measure: Based on, we opted for standard metrics like precision, recall, and F1 scores which focus on exactly matching API and parameters’ names. The reason behind this is that APIs are very specific and unless everything (e.g., name, parameters, input/output format, etc.)  matches the API specifications, executing such APIs will not be possible. We report F1 for matching function names as well as parameter names and values.Longest Common Subsequence (LCS)  and Exact match: We also used LCS fromto capture the overlap between the gold and predicted sequences of APIs. This allows us to compute models’ ability to predict APIs in the correct sequence as required by the user. Similarly, exact match scorechecks if all APIs are predicted by the model and are in the same order.BERTScore, ROUGE-L and BLEU: We follow the evaluation in API-Bank, a dialog dataset that also evaluates model responses based on language generation metrics such as Rouge-L, BertScore, and BLEU.Hallucination Rate: We compute the hallucination rate as the number of samples where the model predicted an API not provided in the function library.5.3Evaluation ResultsTables4,5,6,7, and Figure3depicts an extensive evaluation ofGranite-20B-FunctionCallingmodel in comparison to other state of the art function calling models. In order to detail this evaluation and analyses, below we categorize the results into (a)  Berkeley Function Calling Leaderboard Evaluation, and (b)  Function calling academic benchmarks.ModelOrganizationLicenseAST SummaryExec. SummaryRelevanceOverall Acc.Claude-3.5-Sonnet-20240620 (Prompt) AnthropicProprietary91.3189.5085.4290.00GPT-4-0125-Preview (Prompt) OpenAIProprietary91.2288.1070.4288.00Gemini-1.5-Pro-Preview-0514 (FC) GoogleProprietary87.9283.3289.5886.35Granite-20B-FunctionCallingIBMApache 2.084.1186.5087.0884.71Gorilla-OpenFunctions-v2 (FC) GorillaApache 2.089.3881.5561.2584.71Meta-Llama-3-70B-Instruct (Prompt) MetaMetaLlama 387.7485.3269.1783.88FireFunction-v2FireworksApache 2.086.4480.2656.6781.88Mistral-Medium-2312 (Prompt) Mistral AIProprietary83.7673.4788.3381.35Functionary-Medium-v2.4 (FC) MeetKaiMIT85.6175.7174.1780.47Command-R-Plus (Prompt)  (Opt.) Coherecc-by-nc-4.083.6086.7454.1780.35Functionary-Small-v2.4 (FC) MeetKaiMIT83.5576.3167.9279.94Mistral-large-2402 (FC Auto) Mistral AIProprietary64.7360.0184.1768.76Nexusflow-Raven-v2 (FC) NexusflowApache 2.065.1973.8957.567.35DBRX-Instruct (Prompt) DatabricksDatabricks66.6274.9255.8365.88Snowflake-arctic-Instruct (Prompt) SnowflakeApache 2.061.0980.0459.5865.18Table 4:Berkeley Function Calling Benchmark: Top 15 models by Overall Accuracy (as of 06/25/2024) . All evaluations are done in azero-shotmanner.5.3.1BFCL Leaderboard Evaluation ResultsTable4shows thatGranite-20B-FunctionCallingis ranked fourth on the overall accuracy metric among the top 15 models on BFCL and is highest among models with open licenses181818We have picked the best performing version of each model. For example, Gemini-1.5-Pro-Preview-0514 (FC)  and Gemini-1.5-Pro-Preview-0409 (FC)  are both part of the leaderboard but for our evaluation, we consider the best of Gemini-1.5-Pro.. While it is tied with the Gorillamodel, it is important to note that the latter was finetuned on data that are (a)  generated from ChatGPT, and (b)  similar data to the test set and hasn’t generalized well to other datasets as shown in Table5and Figure3. In the context of model sizes,Granite-20B-FunctionCallingis one of the smallest models in the list. Specifically, the ones better thanGranite-20B-FunctionCallingin the ranking are all significantly larger in size.For the BFCL evaluation dataset, we highlight concerns in certain categories, particularly the Java, JavaScript, and REST API evaluations. We are concerned with how the Java and JavaScript categories evaluate a function-calling model’s capabilities to follow language-specific syntax, for instance how objects are instantiated and called in Java and JavaScript utilizing language-specific context and norms. For the REST API category, we observed significant brittleness in the evaluation due to issues with API availability and API call limits.5.3.2Function Calling Academic BenchmarksTables5and6focus on evaluating the models’ performance on Function Matching using F1-measure, LCS, and Exact Match. In this experiment, we reuse the model handlers from the BFCL code base, including the optimized prompts for each model. However, since the Cohere Command-R-v01 and Mistral-Instruct-v0.3 handlers available in BFCL use the REST API interface for inference, we reimplement handlers for these models, utilizing local models using prompts suggested by the respective model developers for function calling.Function Name Detection:On ToolLLM datasets (G1, G2, and G3)  and RestGPT,Granite-20B-FunctionCallingperforms the best on detecting function names given a natural language utterance with8%better F1 score than the next best function calling model, as shown in Table5. Since these datasets have multiple functions in sequence, we also compute sequencing metrics; exact score and LCS. On this front,Granite-20B-FunctionCallingmodel also outperforms other function calling models by7%on LCS and11%on Exact Match scores.ToolLLM-G1ToolLLM-G2ToolLLM-G3RestGPTAverageFunc.MatchLCSExactScoreFunc.MatchLCSExactScoreFunc.MatchLCSExactScoreFunc.MatchLCSExactScoreFunc.MatchLCSExactScoreFunctionary-small-v2.4 (7B) 0.000.000.000.000.000.000.000.000.000.290.300.060.070.070.02Gorilla-openfunctions-v2 (7B) 0.590.590.280.480.480.220.510.520.240.210.210.010.440.450.19Hermes-2-Pro-Mistral (7B) 0.000.000.000.000.000.000.000.000.000.030.030.010.010.010.00Mistral-Instruct-v0.3 (7B) 0.490.490.260.510.490.300.360.330.130.360.370.080.430.420.19CodeGemma-Instruct (7B) 0.590.590.210.530.530.130.520.540.160.220.230.020.460.470.13Nexusflow-Raven-v2 (13B) 0.650.650.390.730.720.430.680.660.270.390.410.060.610.610.28C4AI-Command-R-v01 (35B) 0.650.640.390.730.710.450.690.680.230.590.600.220.660.660.32Meta-Llama-3-70B-Instruct (70B) 0.610.610.310.590.580.210.650.640.230.220.220.010.520.510.19Granite-20B-FunctionCalling0.860.850.630.840.820.580.760.730.350.510.520.150.740.730.43Table 5:Function Calling Academic Benchmarks: Function Name Detection. Best performance is highlighted inbold, second best isunderlined. All evaluations are done in azero-shotmanner.Full Function Calling:Table6reports on the models’ performance on the API-Bank, ToolBench, and ToolAlpaca datasets that are out-of-domain and evaluated in a zero-shot manner. No single model outperforms all other models across datasets. Note that datasets like ToolAlpaca and API-Bank come with training data split which we never used for trainingGranite-20B-FunctionCalling, but could not guarantee that the other models were not trained with it too. Averaging out the F1 scores across datasets shows thatGranite-20B-FunctionCallingachieves an F1 score of 0.87 when predicting the function name; second best by 0.01 to Cohere’s Command-R (a 35B model)  which provides an F1 score of 0.88. When predicting the arguments,Granite-20B-FunctionCallingaverage F1 score lags behind the best model (Cohere’s Command-R)  by 0.03; 0.62 vs. 0.59.Function Name Hallucination:Hallucinations have been a major drawback of large language models. In the context of calling and executing APIs, hallucinations can have adverse consequences. In Figure3, we compare the models’ Function Name Detection Scores (average F1)  over all the datasets (except BFCL, which uses AST-based metrics)  and their hallucination rates. Ideally, we want models to have high performance and low hallucination rates placing them in the top left corner of the plot.Granite-20B-FunctionCallinghas the highest performance with less than 0.1 hallucination rate.Func-Name+Args Det. (F1 Func-Name | F1 Args) F1 AverageAPI-BankL-1API-BankL-2ToolBenchHSToolBenchBTool-AlpacaNexusRavenFuncNameArgsFunctionary-small-v2.4 (7B) 0.78 | 0.700.54 | 0.450.73 | 0.680.65 | 0.330.88 |0.470.82 | 0.640.730.55Gorilla-openfunctions-v2 (7B) 0.43 | 0.410.12 | 0.120.86 | 0.690.41 | 0.270.69 | 0.390.81 | 0.650.550.42Hermes-2-Pro-Mistral (7B) 0.93|0.770.54 | 0.250.51 | 0.400.56 | 0.260.80 | 0.260.90 | 0.630.710.43Mistral-Instruct-v0.3 (7B) 0.79 | 0.690.69 | 0.460.60 | 0.470.04 | 0.160.33 | 0.330.71 | 0.540.530.44CodeGemma-Instruct (7B) 0.77 | 0.570.59 | 0.380.65 | 0.500.54 | 0.220.59 | 0.310.84 | 0.680.660.44Nexusflow-Raven-v2 (13B) 0.51 | 0.420.28 | 0.220.92| 0.650.89 | 0.350.85 | 0.370.92 |0.750.730.46C4AI-Command-R-v01 (35B) 0.93| 0.760.77 | 0.540.85 | 0.770.88 | 0.490.90| 0.420.93| 0.710.880.62Meta-Llama-3-70B-Instruct (70B) 0.85 | 0.670.69 | 0.520.91 |0.860.91|0.560.78 | 0.430.70 | 0.520.810.59Granite-20B-FunctionCalling0.91 | 0.710.83|0.600.87 | 0.710.82 | 0.360.89 | 0.440.92 | 0.720.870.59Table 6:Function Calling Academic Benchmarks: Full Function Calling. Best performance is highlighted inbold, second best isunderlined. All evaluations are done in azero-shotmanner.API-Bank-Response-Level 1API-Bank-Response-Level 2ModelsBertScoreRouge-LBLEUBertScoreRouge-LBLEUFunctionary-small-v2.4 (7B) 0.340.230.050.350.230.05Gorilla-openfunctions-v2 (7B) 0.560.330.320.510.260.25Hermes-2-Pro-Mistral (7B) 0.450.180.090.420.140.06Mistral-Instruct-v0.3 (7B) 0.520.290.220.460.200.14CodeGemma-Instruct (7B) 0.140.030.000.090.020.01Nexusflow-Raven-v2 (13B) 0.410.160.110.380.110.06C4AI-Command-R-v01 (35B) 0.390.150.070.390.150.06Meta-Llama-3-70B-Instruct (70B) 0.690.480.470.650.400.40Granite-20B-FunctionCalling0.680.470.470.610.360.37Table 7:API-Bank Response generation dataset evaluation. Results are averaged across each dataset per model. Best performance is highlighted inbold, second best isunderlined. All evaluations are done in azero-shotmanner.Figure 3:Performance vs. Hallucination rates for Out-of-Domain Function Calling5.3.3Response GenerationIn Table7, we show models’ performance on response generation task. We use API-Bank dataset and follow their response generation task evaluation with BertScore, Rouge-L, and BLUE. Meta-Llama-3-70B-Instruct has the best performance across the three metrics withGranite-20B-FunctionCallingcoming in close second (difference in performance ranged between 1-5%) . Both models significantly outperform all other evaluated models. The gap is larger when we compareGranite-20B-FunctionCallingto the ones specifically trained for function calling such as Functionary-small-v2.5 and Gorilla-openfunctions-v2.5.3.4Further ImprovementsWe have instruct-tuned theGranite-20B-FunctionCallingin such a way that it develops implicit function searching capability from a long list of functions. For example, in out-of-domain evaluation tasks, for ToolAlpaca the model needs to find the Function from a list of 94 functions, similarly, it has access to 15 and 20 functions for ToolBench-B and ToolBench-HS, respectively. Due to this reason, the prompt with all the function libraries increases the context length andGranite-20B-Code-Instructsupports up to 8192 context length, so we were not able to add the full signature of each Function in the library. Currently, each function in the library contains a function name, a description, and for each function, we have provided a list of respected arguments with their descriptions. However, to fit the prompt in the max context-length, we had to remove the type, required-fields, and optional-fields values for each argument from the specifications. For further performance exploration, we will assess options to include the entire function specification without truncation including exploiting the benefit of Rotary Position Embeddingand its innate support for longer context lengths within some of the other Granite models.6ConclusionIn this paper, we introducedGranite-20B-FunctionCalling, a capable function calling open model with Apache 2 license.Granite-20B-FunctionCallingis trained using a suite of datasets transformed from semantic parsing, task-oriented dialog, personal assistants and conversational domains. The training setup is a multi-task learning approach where granular tasks in function calling such as function detection, parameter detection, sequencing, and next best function are used for instruction tuning the model. We performed an extensive evaluation ofGranite-20B-FunctionCallingin comparison to other state-of-the-art function calling models. On multiple out-of-domain datasets, including Berkeley Function Calling Leaderboard,Granite-20B-FunctionCallingprovides the best performance among the models that have open licenses. Even compared to multiple proprietary models with much larger sizes,Granite-20B-FunctionCallingshowed on-par and in some cases better performance on multiple datasets and tasks.ReferencesAI@Meta (2024) AI@Meta.Llama 3 model card.2024.URLhttps://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.Basu et al. (2024) Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A. Lastras.Api-blend: A comprehensive corpora for training and benchmarking api llms, 2024.Chen et al. (2024a) Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang.Advancing tool-augmented large language models: Integrating insights from errors in inference trees, 2024a.Chen et al. (2024b) Wei Chen, Zhiyuan Li, and Mingyuan Ma.Octopus: On-device language model for function calling of software apis, 2024b.Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.Scaling instruction-finetuned language models.arXiv preprint arXiv:2210.11416, 2022.CodeGemma Team et al. (2024) CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, and Hui.Codegemma: Open code models based on gemma.2024.URLhttps://goo.gle/codegemma.CohereForAI (2024) CohereForAI.C4ai command-r: A 35 billion parameter generative model for reasoning, summarization, and question answering.Hugging Face Models, 2024.URLhttps://huggingface.co/CohereForAI/c4ai-command-r-v01.Crouse et al. (2023) Maxwell Crouse, Ibrahim Abdelaziz, Kinjal Basu, Soham Dan, Sadhana Kumaravel, Achille Fokoue, Pavan Kapanipathi, and Luis Lastras.Formally specifying the high-level behavior of llm-based agents.arXiv preprint arXiv:2310.08535, 2023.Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.Qlora: Efficient finetuning of quantized llms, 2023.Gao et al. (2023) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.Pal: Program-aided language models.InInternational Conference on Machine Learning, pp.  10764–10799. PMLR, 2023.Guo et al. (2024) Zhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen, and Rameswar Panda.Api pack: A massive multi-programming language dataset for api call generation, 2024.Gur et al. (2023) Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust.A real-world webagent with planning, long context understanding, and program synthesis.InThe Twelfth International Conference on Learning Representations, 2023.He-Yueya et al. (2023) Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman.Solving math word problems by combining language models with symbolic solvers.arXiv preprint arXiv:2304.09102, 2023.Jain et al. (2024) Arushi Jain, Shubham Paliwal, Monika Sharma, Lovekesh Vig, and Gautam Shroff.Smartflow: Robotic process automation using llms, 2024.Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.Mistral 7b.arXiv preprint arXiv:2310.06825, 2023.Komeili et al. (2022) Mojtaba Komeili, Kurt Shuster, and Jason Weston.Internet-augmented dialogue generation.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  8460–8478, 2022.Li et al. (2023a) Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter.Chain of code: Reasoning with a language model-augmented code emulator.arXiv preprint arXiv:2312.04474, 2023a.Li et al. (2024) Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al.Synthetic data (almost)  from scratch: Generalized instruction tuning for language models.arXiv preprint arXiv:2402.13064, 2024.Li et al. (2023b) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.Api-bank: A comprehensive benchmark for tool-augmented llms, 2023b.Lin (2004) Chin-Yew Lin.Rouge: A package for automatic evaluation of summaries.InText summarization branches out, pp.  74–81, 2004.MeetKai (2024) MeetKai.Functionary-7b-v1.4: A language model for function interpretation and execution.Hugging Face Models, 2024.URLhttps://huggingface.co/meetkai/functionary-7b-v1.4.Mishra et al. (2024) Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al.Granite code models: A family of open foundation models for code intelligence.arXiv preprint arXiv:2405.04324, 2024.Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.Cross-task generalization via natural language crowdsourcing instructions.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  3470–3487, 2022.Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.Crosslingual generalization through multitask finetuning.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.) ,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  15991–16111, Toronto, Canada, July 2023. Association for Computational Linguistics.doi:10.18653/v1/2023.acl-long.891.URLhttps://aclanthology.org/2023.acl-long.891.Nexusflow.ai (2023) Nexusflow.ai.Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling, 2023.URLhttps://nexusflow.ai/blogs/ravenv2.Nous-Research (2023) Nous-Research.Nous-hermes-13b.https://huggingface.co/NousResearch/Nous-Hermes-13b, 2023.Model on Hugging Face.Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022.Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.Bleu: a method for automatic evaluation of machine translation.InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.  311–318, 2002.Parisi et al. (2022) Aaron Parisi, Yao Zhao, and Noah Fiedel.Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022.Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal.Are NLP models really able to solve simple math word problems?In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.) ,Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  2080–2094, Online, June 2021. Association for Computational Linguistics.doi:10.18653/v1/2021.naacl-main.168.URLhttps://aclanthology.org/2021.naacl-main.168.Patil et al. (2023) Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez.Gorilla: Large language model connected with massive apis.arXiv preprint arXiv:2305.15334, 2023.Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023.Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.Direct preference optimization: Your language model is secretly a reward model, 2023.Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024.Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023.Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.Toolformer: Language models can teach themselves to use tools.ArXiv, abs/2302.04761, 2023.URLhttps://api.semanticscholar.org/CorpusID:256697342.Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao.Reflexion: Language agents with verbal reinforcement learning.InThirty-seventh Conference on Neural Information Processing Systems, 2023.Song et al. (2023) Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li.Restgpt: Connecting large language models with real-world restful apis, 2023.Su et al. (2023) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.Roformer: Enhanced transformer with rotary position embedding.2023.Sudalairaj et al. (2024) Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D Cox, and Akash Srivastava.Lab: Large-scale alignment for chatbots.arXiv preprint arXiv:2403.01081, 2024.Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun.Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.arXiv preprint arXiv:2306.05301, 2023.Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.Self-instruct: Aligning language models with self-generated instructions.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.) ,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.doi:10.18653/v1/2023.acl-long.754.URLhttps://aclanthology.org/2023.acl-long.754.Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.Finetuned language models are zero-shot learners.InInternational Conference on Learning Representations, 2021.Xu et al. (2023a) Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu.Rewoo: Decoupling reasoning from observations for efficient augmented language models.arXiv preprint arXiv:2305.18323, 2023a.Xu et al. (2023b) Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang.On the tool manipulation capability of open-source large language models.arXiv preprint arXiv:2305.16504, 2023b.Yan et al. (2024) Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez.Berkeley function calling leaderboard.https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024.Yang et al. (2024a) John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.Swe-agent: Agent-computer interfaces enable automated software engineering, 2024a.Yang et al. (2024b) Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.Gpt4tools: Teaching large language model to use tools via self-instruction.Advances in Neural Information Processing Systems, 36, 2024b.Yang et al. (2023) Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.Mm-react: Prompting chatgpt for multimodal reasoning and action.arXiv preprint arXiv:2303.11381, 2023.Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.React: Synergizing reasoning and acting in language models.InThe Eleventh International Conference on Learning Representations, 2022.Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2019.
body
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
4Instruct Tuning
2407.00121
5Experimental Setup and EvaluationIn the section below, we detail our extensive evaluation on various evaluation datasets and public leaderboard. We provide a comprehensive comparison of ourGranite-20B-FunctionCalling, open sourced with Apache 2.0 license, to other open and proprietary function calling models.5.1DatasetsThe evaluation datasets and leaderboards for function calling are gaining a lot of traction in the recent past. In particular, to evaluate the models’ generalizability, we evaluatedGranite-20B-FunctionCallingon a variety of function calling benchmarks, all of which are out-of-domain evaluation for our model. It is worth noting that some of these datasets; e.g. ToolAlpaca and ToolLLM, have training data releases. However, wedid not useany of these benchmarks to trainGranite-20B-FunctionCallingand we only used the datasets in1.121212We could not verify whether some (or all)  of the out-of-domain datasets were used in other models’ training sets.Table3depicts the details of the evaluation datasets we used. We list the details of each of these evaluation datasets below.DatasetTest InstancesTesting tasksMetricsBFCL1,700Function CallingAST, Execution AccuracyRelevancyAccuracyToolLLM491Function CallingFunc. matching (F1) RestGPT157Function CallingFunc. matching (F1) API-Bank473Function CallingFunc. and Param. matching (F1) 478Response GenerationBERTscore, ROUGE, BLEUToolBench214Function CallingFunc. and Param. matching (F1) ToolAlpaca100Function CallingFunc. and Param. matching (F1) NexusRaven318Function CallingFunc. and Param. matching (F1) Table 3:Evaluation Datasets(1) Berkeley Function-Calling Leaderboard (BFCL) 131313BFCL:https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.htmlis a comprehensive function calling leaderboard that includes a dataset of over 1,700 instances. The leaderboard evaluates tasks that include (a)  Simple Function, Multiple Function, Parallel Function, and Parallel Multiple Function for Python Language; and (b)  for non-Python, they evaluate function relevance detection, REST API, JavaScript, and Java. 
(2) ToolBenchis a subset of the data in ToolBench (as released by the authors)  focused on HomeSearch and Booking domains. 
(3) ToolLLM141414ToolLLM also calls their benchmark ToolBench. To disambiguate, in this paper we use the term ToolLLM to refer to their benchmark dataset.is synthetically generated using ChatGPT. The approach uses an initial collection of 16,000 APIs from RapidAPI151515https://rapidapi.com/for synthetic data generation. The evaluation is done on the three test sets categorized based on complexity; G1 – single-tool, G2 – intra-category multi-tool, G3 – intra-collection multi-tool. 
(4) RestGPTis a function calling dataset that has 157 test examples with 85 APIs from Spotify and TMDB. This dataset focuses only on testing model’s ability to detect function names. 
(5) API-Bankhas 314 tool-use dialogues with 753 API calls to assess LLMs’ capabilities in planning, retrieving, and calling APIs. 
(6) ToolAlpacais a synthetic data generation approach that has both training and evaluation benchmarks. It contains 271 tool-use instances spanning 50 distinct categories. Similar to, we used the simulated part of ToolAlpaca which has a total of 100 test examples. 
(7) NexusRaven API Evaluation161616https://huggingface.co/datasets/Nexusflow/NexusRaven_API_evaluationis another function calling dataset with 318 test examples covering a total of 65 different APIs.5.2Evaluation MetricsBelow, we define the metrics we adopted for specific tasks in function calling.BFCL Metrics171717https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html#metrics:BFCL evaluates multiple tasks using the following four metrics. 
(1) AST summarycompares the abstract syntax tree of the function output to the ground truth and the function definition. It captures the correctness of the functions called, their parameters (required or not) , and the parameter types. 
(2) Execution Summarycompares the execution output from generated and ground-truth function calls. This metric is used to evaluate REST APIs and non-REST data samples. 
(3) Relevanceevaluates the model’s ability to detect no function calls when the given list of functions is irrelevant to the user query. This inversely captures the hallucination rate of models. 
(4) Overall Accuracyis the weighted average of all individual data splits in BFCL.The same metrics described above cannot be used for our out-of-domain datasets because of missing information, varied formats, and response generation task. For example, ToolLLM datasets has missing arguments, ToolAlpaca has missing argument types, and API-Bank has response generation task. Therefore, we use the following metrics to evaluate the models on other datasets:F1 measure: Based on, we opted for standard metrics like precision, recall, and F1 scores which focus on exactly matching API and parameters’ names. The reason behind this is that APIs are very specific and unless everything (e.g., name, parameters, input/output format, etc.)  matches the API specifications, executing such APIs will not be possible. We report F1 for matching function names as well as parameter names and values.Longest Common Subsequence (LCS)  and Exact match: We also used LCS fromto capture the overlap between the gold and predicted sequences of APIs. This allows us to compute models’ ability to predict APIs in the correct sequence as required by the user. Similarly, exact match scorechecks if all APIs are predicted by the model and are in the same order.BERTScore, ROUGE-L and BLEU: We follow the evaluation in API-Bank, a dialog dataset that also evaluates model responses based on language generation metrics such as Rouge-L, BertScore, and BLEU.Hallucination Rate: We compute the hallucination rate as the number of samples where the model predicted an API not provided in the function library.5.3Evaluation ResultsTables4,5,6,7, and Figure3depicts an extensive evaluation ofGranite-20B-FunctionCallingmodel in comparison to other state of the art function calling models. In order to detail this evaluation and analyses, below we categorize the results into (a)  Berkeley Function Calling Leaderboard Evaluation, and (b)  Function calling academic benchmarks.ModelOrganizationLicenseAST SummaryExec. SummaryRelevanceOverall Acc.Claude-3.5-Sonnet-20240620 (Prompt) AnthropicProprietary91.3189.5085.4290.00GPT-4-0125-Preview (Prompt) OpenAIProprietary91.2288.1070.4288.00Gemini-1.5-Pro-Preview-0514 (FC) GoogleProprietary87.9283.3289.5886.35Granite-20B-FunctionCallingIBMApache 2.084.1186.5087.0884.71Gorilla-OpenFunctions-v2 (FC) GorillaApache 2.089.3881.5561.2584.71Meta-Llama-3-70B-Instruct (Prompt) MetaMetaLlama 387.7485.3269.1783.88FireFunction-v2FireworksApache 2.086.4480.2656.6781.88Mistral-Medium-2312 (Prompt) Mistral AIProprietary83.7673.4788.3381.35Functionary-Medium-v2.4 (FC) MeetKaiMIT85.6175.7174.1780.47Command-R-Plus (Prompt)  (Opt.) Coherecc-by-nc-4.083.6086.7454.1780.35Functionary-Small-v2.4 (FC) MeetKaiMIT83.5576.3167.9279.94Mistral-large-2402 (FC Auto) Mistral AIProprietary64.7360.0184.1768.76Nexusflow-Raven-v2 (FC) NexusflowApache 2.065.1973.8957.567.35DBRX-Instruct (Prompt) DatabricksDatabricks66.6274.9255.8365.88Snowflake-arctic-Instruct (Prompt) SnowflakeApache 2.061.0980.0459.5865.18Table 4:Berkeley Function Calling Benchmark: Top 15 models by Overall Accuracy (as of 06/25/2024) . All evaluations are done in azero-shotmanner.5.3.1BFCL Leaderboard Evaluation ResultsTable4shows thatGranite-20B-FunctionCallingis ranked fourth on the overall accuracy metric among the top 15 models on BFCL and is highest among models with open licenses181818We have picked the best performing version of each model. For example, Gemini-1.5-Pro-Preview-0514 (FC)  and Gemini-1.5-Pro-Preview-0409 (FC)  are both part of the leaderboard but for our evaluation, we consider the best of Gemini-1.5-Pro.. While it is tied with the Gorillamodel, it is important to note that the latter was finetuned on data that are (a)  generated from ChatGPT, and (b)  similar data to the test set and hasn’t generalized well to other datasets as shown in Table5and Figure3. In the context of model sizes,Granite-20B-FunctionCallingis one of the smallest models in the list. Specifically, the ones better thanGranite-20B-FunctionCallingin the ranking are all significantly larger in size.For the BFCL evaluation dataset, we highlight concerns in certain categories, particularly the Java, JavaScript, and REST API evaluations. We are concerned with how the Java and JavaScript categories evaluate a function-calling model’s capabilities to follow language-specific syntax, for instance how objects are instantiated and called in Java and JavaScript utilizing language-specific context and norms. For the REST API category, we observed significant brittleness in the evaluation due to issues with API availability and API call limits.5.3.2Function Calling Academic BenchmarksTables5and6focus on evaluating the models’ performance on Function Matching using F1-measure, LCS, and Exact Match. In this experiment, we reuse the model handlers from the BFCL code base, including the optimized prompts for each model. However, since the Cohere Command-R-v01 and Mistral-Instruct-v0.3 handlers available in BFCL use the REST API interface for inference, we reimplement handlers for these models, utilizing local models using prompts suggested by the respective model developers for function calling.Function Name Detection:On ToolLLM datasets (G1, G2, and G3)  and RestGPT,Granite-20B-FunctionCallingperforms the best on detecting function names given a natural language utterance with8%better F1 score than the next best function calling model, as shown in Table5. Since these datasets have multiple functions in sequence, we also compute sequencing metrics; exact score and LCS. On this front,Granite-20B-FunctionCallingmodel also outperforms other function calling models by7%on LCS and11%on Exact Match scores.ToolLLM-G1ToolLLM-G2ToolLLM-G3RestGPTAverageFunc.MatchLCSExactScoreFunc.MatchLCSExactScoreFunc.MatchLCSExactScoreFunc.MatchLCSExactScoreFunc.MatchLCSExactScoreFunctionary-small-v2.4 (7B) 0.000.000.000.000.000.000.000.000.000.290.300.060.070.070.02Gorilla-openfunctions-v2 (7B) 0.590.590.280.480.480.220.510.520.240.210.210.010.440.450.19Hermes-2-Pro-Mistral (7B) 0.000.000.000.000.000.000.000.000.000.030.030.010.010.010.00Mistral-Instruct-v0.3 (7B) 0.490.490.260.510.490.300.360.330.130.360.370.080.430.420.19CodeGemma-Instruct (7B) 0.590.590.210.530.530.130.520.540.160.220.230.020.460.470.13Nexusflow-Raven-v2 (13B) 0.650.650.390.730.720.430.680.660.270.390.410.060.610.610.28C4AI-Command-R-v01 (35B) 0.650.640.390.730.710.450.690.680.230.590.600.220.660.660.32Meta-Llama-3-70B-Instruct (70B) 0.610.610.310.590.580.210.650.640.230.220.220.010.520.510.19Granite-20B-FunctionCalling0.860.850.630.840.820.580.760.730.350.510.520.150.740.730.43Table 5:Function Calling Academic Benchmarks: Function Name Detection. Best performance is highlighted inbold, second best isunderlined. All evaluations are done in azero-shotmanner.Full Function Calling:Table6reports on the models’ performance on the API-Bank, ToolBench, and ToolAlpaca datasets that are out-of-domain and evaluated in a zero-shot manner. No single model outperforms all other models across datasets. Note that datasets like ToolAlpaca and API-Bank come with training data split which we never used for trainingGranite-20B-FunctionCalling, but could not guarantee that the other models were not trained with it too. Averaging out the F1 scores across datasets shows thatGranite-20B-FunctionCallingachieves an F1 score of 0.87 when predicting the function name; second best by 0.01 to Cohere’s Command-R (a 35B model)  which provides an F1 score of 0.88. When predicting the arguments,Granite-20B-FunctionCallingaverage F1 score lags behind the best model (Cohere’s Command-R)  by 0.03; 0.62 vs. 0.59.Function Name Hallucination:Hallucinations have been a major drawback of large language models. In the context of calling and executing APIs, hallucinations can have adverse consequences. In Figure3, we compare the models’ Function Name Detection Scores (average F1)  over all the datasets (except BFCL, which uses AST-based metrics)  and their hallucination rates. Ideally, we want models to have high performance and low hallucination rates placing them in the top left corner of the plot.Granite-20B-FunctionCallinghas the highest performance with less than 0.1 hallucination rate.Func-Name+Args Det. (F1 Func-Name | F1 Args) F1 AverageAPI-BankL-1API-BankL-2ToolBenchHSToolBenchBTool-AlpacaNexusRavenFuncNameArgsFunctionary-small-v2.4 (7B) 0.78 | 0.700.54 | 0.450.73 | 0.680.65 | 0.330.88 |0.470.82 | 0.640.730.55Gorilla-openfunctions-v2 (7B) 0.43 | 0.410.12 | 0.120.86 | 0.690.41 | 0.270.69 | 0.390.81 | 0.650.550.42Hermes-2-Pro-Mistral (7B) 0.93|0.770.54 | 0.250.51 | 0.400.56 | 0.260.80 | 0.260.90 | 0.630.710.43Mistral-Instruct-v0.3 (7B) 0.79 | 0.690.69 | 0.460.60 | 0.470.04 | 0.160.33 | 0.330.71 | 0.540.530.44CodeGemma-Instruct (7B) 0.77 | 0.570.59 | 0.380.65 | 0.500.54 | 0.220.59 | 0.310.84 | 0.680.660.44Nexusflow-Raven-v2 (13B) 0.51 | 0.420.28 | 0.220.92| 0.650.89 | 0.350.85 | 0.370.92 |0.750.730.46C4AI-Command-R-v01 (35B) 0.93| 0.760.77 | 0.540.85 | 0.770.88 | 0.490.90| 0.420.93| 0.710.880.62Meta-Llama-3-70B-Instruct (70B) 0.85 | 0.670.69 | 0.520.91 |0.860.91|0.560.78 | 0.430.70 | 0.520.810.59Granite-20B-FunctionCalling0.91 | 0.710.83|0.600.87 | 0.710.82 | 0.360.89 | 0.440.92 | 0.720.870.59Table 6:Function Calling Academic Benchmarks: Full Function Calling. Best performance is highlighted inbold, second best isunderlined. All evaluations are done in azero-shotmanner.API-Bank-Response-Level 1API-Bank-Response-Level 2ModelsBertScoreRouge-LBLEUBertScoreRouge-LBLEUFunctionary-small-v2.4 (7B) 0.340.230.050.350.230.05Gorilla-openfunctions-v2 (7B) 0.560.330.320.510.260.25Hermes-2-Pro-Mistral (7B) 0.450.180.090.420.140.06Mistral-Instruct-v0.3 (7B) 0.520.290.220.460.200.14CodeGemma-Instruct (7B) 0.140.030.000.090.020.01Nexusflow-Raven-v2 (13B) 0.410.160.110.380.110.06C4AI-Command-R-v01 (35B) 0.390.150.070.390.150.06Meta-Llama-3-70B-Instruct (70B) 0.690.480.470.650.400.40Granite-20B-FunctionCalling0.680.470.470.610.360.37Table 7:API-Bank Response generation dataset evaluation. Results are averaged across each dataset per model. Best performance is highlighted inbold, second best isunderlined. All evaluations are done in azero-shotmanner.Figure 3:Performance vs. Hallucination rates for Out-of-Domain Function Calling5.3.3Response GenerationIn Table7, we show models’ performance on response generation task. We use API-Bank dataset and follow their response generation task evaluation with BertScore, Rouge-L, and BLUE. Meta-Llama-3-70B-Instruct has the best performance across the three metrics withGranite-20B-FunctionCallingcoming in close second (difference in performance ranged between 1-5%) . Both models significantly outperform all other evaluated models. The gap is larger when we compareGranite-20B-FunctionCallingto the ones specifically trained for function calling such as Functionary-small-v2.5 and Gorilla-openfunctions-v2.5.3.4Further ImprovementsWe have instruct-tuned theGranite-20B-FunctionCallingin such a way that it develops implicit function searching capability from a long list of functions. For example, in out-of-domain evaluation tasks, for ToolAlpaca the model needs to find the Function from a list of 94 functions, similarly, it has access to 15 and 20 functions for ToolBench-B and ToolBench-HS, respectively. Due to this reason, the prompt with all the function libraries increases the context length andGranite-20B-Code-Instructsupports up to 8192 context length, so we were not able to add the full signature of each Function in the library. Currently, each function in the library contains a function name, a description, and for each function, we have provided a list of respected arguments with their descriptions. However, to fit the prompt in the max context-length, we had to remove the type, required-fields, and optional-fields values for each argument from the specifications. For further performance exploration, we will assess options to include the entire function specification without truncation including exploiting the benefit of Rotary Position Embeddingand its innate support for longer context lengths within some of the other Granite models.6ConclusionIn this paper, we introducedGranite-20B-FunctionCalling, a capable function calling open model with Apache 2 license.Granite-20B-FunctionCallingis trained using a suite of datasets transformed from semantic parsing, task-oriented dialog, personal assistants and conversational domains. The training setup is a multi-task learning approach where granular tasks in function calling such as function detection, parameter detection, sequencing, and next best function are used for instruction tuning the model. We performed an extensive evaluation ofGranite-20B-FunctionCallingin comparison to other state-of-the-art function calling models. On multiple out-of-domain datasets, including Berkeley Function Calling Leaderboard,Granite-20B-FunctionCallingprovides the best performance among the models that have open licenses. Even compared to multiple proprietary models with much larger sizes,Granite-20B-FunctionCallingshowed on-par and in some cases better performance on multiple datasets and tasks.ReferencesAI@Meta (2024) AI@Meta.Llama 3 model card.2024.URLhttps://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.Basu et al. (2024) Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A. Lastras.Api-blend: A comprehensive corpora for training and benchmarking api llms, 2024.Chen et al. (2024a) Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang.Advancing tool-augmented large language models: Integrating insights from errors in inference trees, 2024a.Chen et al. (2024b) Wei Chen, Zhiyuan Li, and Mingyuan Ma.Octopus: On-device language model for function calling of software apis, 2024b.Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.Scaling instruction-finetuned language models.arXiv preprint arXiv:2210.11416, 2022.CodeGemma Team et al. (2024) CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, and Hui.Codegemma: Open code models based on gemma.2024.URLhttps://goo.gle/codegemma.CohereForAI (2024) CohereForAI.C4ai command-r: A 35 billion parameter generative model for reasoning, summarization, and question answering.Hugging Face Models, 2024.URLhttps://huggingface.co/CohereForAI/c4ai-command-r-v01.Crouse et al. (2023) Maxwell Crouse, Ibrahim Abdelaziz, Kinjal Basu, Soham Dan, Sadhana Kumaravel, Achille Fokoue, Pavan Kapanipathi, and Luis Lastras.Formally specifying the high-level behavior of llm-based agents.arXiv preprint arXiv:2310.08535, 2023.Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.Qlora: Efficient finetuning of quantized llms, 2023.Gao et al. (2023) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.Pal: Program-aided language models.InInternational Conference on Machine Learning, pp.  10764–10799. PMLR, 2023.Guo et al. (2024) Zhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen, and Rameswar Panda.Api pack: A massive multi-programming language dataset for api call generation, 2024.Gur et al. (2023) Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust.A real-world webagent with planning, long context understanding, and program synthesis.InThe Twelfth International Conference on Learning Representations, 2023.He-Yueya et al. (2023) Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman.Solving math word problems by combining language models with symbolic solvers.arXiv preprint arXiv:2304.09102, 2023.Jain et al. (2024) Arushi Jain, Shubham Paliwal, Monika Sharma, Lovekesh Vig, and Gautam Shroff.Smartflow: Robotic process automation using llms, 2024.Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.Mistral 7b.arXiv preprint arXiv:2310.06825, 2023.Komeili et al. (2022) Mojtaba Komeili, Kurt Shuster, and Jason Weston.Internet-augmented dialogue generation.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  8460–8478, 2022.Li et al. (2023a) Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter.Chain of code: Reasoning with a language model-augmented code emulator.arXiv preprint arXiv:2312.04474, 2023a.Li et al. (2024) Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al.Synthetic data (almost)  from scratch: Generalized instruction tuning for language models.arXiv preprint arXiv:2402.13064, 2024.Li et al. (2023b) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.Api-bank: A comprehensive benchmark for tool-augmented llms, 2023b.Lin (2004) Chin-Yew Lin.Rouge: A package for automatic evaluation of summaries.InText summarization branches out, pp.  74–81, 2004.MeetKai (2024) MeetKai.Functionary-7b-v1.4: A language model for function interpretation and execution.Hugging Face Models, 2024.URLhttps://huggingface.co/meetkai/functionary-7b-v1.4.Mishra et al. (2024) Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al.Granite code models: A family of open foundation models for code intelligence.arXiv preprint arXiv:2405.04324, 2024.Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.Cross-task generalization via natural language crowdsourcing instructions.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  3470–3487, 2022.Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.Crosslingual generalization through multitask finetuning.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.) ,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  15991–16111, Toronto, Canada, July 2023. Association for Computational Linguistics.doi:10.18653/v1/2023.acl-long.891.URLhttps://aclanthology.org/2023.acl-long.891.Nexusflow.ai (2023) Nexusflow.ai.Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling, 2023.URLhttps://nexusflow.ai/blogs/ravenv2.Nous-Research (2023) Nous-Research.Nous-hermes-13b.https://huggingface.co/NousResearch/Nous-Hermes-13b, 2023.Model on Hugging Face.Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022.Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.Bleu: a method for automatic evaluation of machine translation.InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.  311–318, 2002.Parisi et al. (2022) Aaron Parisi, Yao Zhao, and Noah Fiedel.Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022.Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal.Are NLP models really able to solve simple math word problems?In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.) ,Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  2080–2094, Online, June 2021. Association for Computational Linguistics.doi:10.18653/v1/2021.naacl-main.168.URLhttps://aclanthology.org/2021.naacl-main.168.Patil et al. (2023) Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez.Gorilla: Large language model connected with massive apis.arXiv preprint arXiv:2305.15334, 2023.Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023.Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.Direct preference optimization: Your language model is secretly a reward model, 2023.Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024.Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023.Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.Toolformer: Language models can teach themselves to use tools.ArXiv, abs/2302.04761, 2023.URLhttps://api.semanticscholar.org/CorpusID:256697342.Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao.Reflexion: Language agents with verbal reinforcement learning.InThirty-seventh Conference on Neural Information Processing Systems, 2023.Song et al. (2023) Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li.Restgpt: Connecting large language models with real-world restful apis, 2023.Su et al. (2023) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.Roformer: Enhanced transformer with rotary position embedding.2023.Sudalairaj et al. (2024) Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D Cox, and Akash Srivastava.Lab: Large-scale alignment for chatbots.arXiv preprint arXiv:2403.01081, 2024.Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun.Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.arXiv preprint arXiv:2306.05301, 2023.Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.Self-instruct: Aligning language models with self-generated instructions.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.) ,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.doi:10.18653/v1/2023.acl-long.754.URLhttps://aclanthology.org/2023.acl-long.754.Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.Finetuned language models are zero-shot learners.InInternational Conference on Learning Representations, 2021.Xu et al. (2023a) Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu.Rewoo: Decoupling reasoning from observations for efficient augmented language models.arXiv preprint arXiv:2305.18323, 2023a.Xu et al. (2023b) Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang.On the tool manipulation capability of open-source large language models.arXiv preprint arXiv:2305.16504, 2023b.Yan et al. (2024) Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez.Berkeley function calling leaderboard.https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024.Yang et al. (2024a) John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.Swe-agent: Agent-computer interfaces enable automated software engineering, 2024a.Yang et al. (2024b) Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.Gpt4tools: Teaching large language model to use tools via self-instruction.Advances in Neural Information Processing Systems, 36, 2024b.Yang et al. (2023) Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.Mm-react: Prompting chatgpt for multimodal reasoning and action.arXiv preprint arXiv:2303.11381, 2023.Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.React: Synergizing reasoning and acting in language models.InThe Eleventh International Conference on Learning Representations, 2022.Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2019.
body
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
5Experimental Setup and Evaluation
2407.00121
6ConclusionIn this paper, we introducedGranite-20B-FunctionCalling, a capable function calling open model with Apache 2 license.Granite-20B-FunctionCallingis trained using a suite of datasets transformed from semantic parsing, task-oriented dialog, personal assistants and conversational domains. The training setup is a multi-task learning approach where granular tasks in function calling such as function detection, parameter detection, sequencing, and next best function are used for instruction tuning the model. We performed an extensive evaluation ofGranite-20B-FunctionCallingin comparison to other state-of-the-art function calling models. On multiple out-of-domain datasets, including Berkeley Function Calling Leaderboard,Granite-20B-FunctionCallingprovides the best performance among the models that have open licenses. Even compared to multiple proprietary models with much larger sizes,Granite-20B-FunctionCallingshowed on-par and in some cases better performance on multiple datasets and tasks.ReferencesAI@Meta (2024) AI@Meta.Llama 3 model card.2024.URLhttps://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.Basu et al. (2024) Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A. Lastras.Api-blend: A comprehensive corpora for training and benchmarking api llms, 2024.Chen et al. (2024a) Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang.Advancing tool-augmented large language models: Integrating insights from errors in inference trees, 2024a.Chen et al. (2024b) Wei Chen, Zhiyuan Li, and Mingyuan Ma.Octopus: On-device language model for function calling of software apis, 2024b.Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.Scaling instruction-finetuned language models.arXiv preprint arXiv:2210.11416, 2022.CodeGemma Team et al. (2024) CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, and Hui.Codegemma: Open code models based on gemma.2024.URLhttps://goo.gle/codegemma.CohereForAI (2024) CohereForAI.C4ai command-r: A 35 billion parameter generative model for reasoning, summarization, and question answering.Hugging Face Models, 2024.URLhttps://huggingface.co/CohereForAI/c4ai-command-r-v01.Crouse et al. (2023) Maxwell Crouse, Ibrahim Abdelaziz, Kinjal Basu, Soham Dan, Sadhana Kumaravel, Achille Fokoue, Pavan Kapanipathi, and Luis Lastras.Formally specifying the high-level behavior of llm-based agents.arXiv preprint arXiv:2310.08535, 2023.Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.Qlora: Efficient finetuning of quantized llms, 2023.Gao et al. (2023) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.Pal: Program-aided language models.InInternational Conference on Machine Learning, pp.  10764–10799. PMLR, 2023.Guo et al. (2024) Zhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen, and Rameswar Panda.Api pack: A massive multi-programming language dataset for api call generation, 2024.Gur et al. (2023) Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust.A real-world webagent with planning, long context understanding, and program synthesis.InThe Twelfth International Conference on Learning Representations, 2023.He-Yueya et al. (2023) Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman.Solving math word problems by combining language models with symbolic solvers.arXiv preprint arXiv:2304.09102, 2023.Jain et al. (2024) Arushi Jain, Shubham Paliwal, Monika Sharma, Lovekesh Vig, and Gautam Shroff.Smartflow: Robotic process automation using llms, 2024.Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.Mistral 7b.arXiv preprint arXiv:2310.06825, 2023.Komeili et al. (2022) Mojtaba Komeili, Kurt Shuster, and Jason Weston.Internet-augmented dialogue generation.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  8460–8478, 2022.Li et al. (2023a) Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter.Chain of code: Reasoning with a language model-augmented code emulator.arXiv preprint arXiv:2312.04474, 2023a.Li et al. (2024) Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al.Synthetic data (almost)  from scratch: Generalized instruction tuning for language models.arXiv preprint arXiv:2402.13064, 2024.Li et al. (2023b) Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.Api-bank: A comprehensive benchmark for tool-augmented llms, 2023b.Lin (2004) Chin-Yew Lin.Rouge: A package for automatic evaluation of summaries.InText summarization branches out, pp.  74–81, 2004.MeetKai (2024) MeetKai.Functionary-7b-v1.4: A language model for function interpretation and execution.Hugging Face Models, 2024.URLhttps://huggingface.co/meetkai/functionary-7b-v1.4.Mishra et al. (2024) Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al.Granite code models: A family of open foundation models for code intelligence.arXiv preprint arXiv:2405.04324, 2024.Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.Cross-task generalization via natural language crowdsourcing instructions.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  3470–3487, 2022.Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.Crosslingual generalization through multitask finetuning.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.) ,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  15991–16111, Toronto, Canada, July 2023. Association for Computational Linguistics.doi:10.18653/v1/2023.acl-long.891.URLhttps://aclanthology.org/2023.acl-long.891.Nexusflow.ai (2023) Nexusflow.ai.Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling, 2023.URLhttps://nexusflow.ai/blogs/ravenv2.Nous-Research (2023) Nous-Research.Nous-hermes-13b.https://huggingface.co/NousResearch/Nous-Hermes-13b, 2023.Model on Hugging Face.Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022.Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.Bleu: a method for automatic evaluation of machine translation.InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.  311–318, 2002.Parisi et al. (2022) Aaron Parisi, Yao Zhao, and Noah Fiedel.Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022.Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal.Are NLP models really able to solve simple math word problems?In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.) ,Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  2080–2094, Online, June 2021. Association for Computational Linguistics.doi:10.18653/v1/2021.naacl-main.168.URLhttps://aclanthology.org/2021.naacl-main.168.Patil et al. (2023) Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez.Gorilla: Large language model connected with massive apis.arXiv preprint arXiv:2305.15334, 2023.Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023.Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.Direct preference optimization: Your language model is secretly a reward model, 2023.Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024.Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023.Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.Toolformer: Language models can teach themselves to use tools.ArXiv, abs/2302.04761, 2023.URLhttps://api.semanticscholar.org/CorpusID:256697342.Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao.Reflexion: Language agents with verbal reinforcement learning.InThirty-seventh Conference on Neural Information Processing Systems, 2023.Song et al. (2023) Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li.Restgpt: Connecting large language models with real-world restful apis, 2023.Su et al. (2023) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.Roformer: Enhanced transformer with rotary position embedding.2023.Sudalairaj et al. (2024) Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D Cox, and Akash Srivastava.Lab: Large-scale alignment for chatbots.arXiv preprint arXiv:2403.01081, 2024.Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun.Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.arXiv preprint arXiv:2306.05301, 2023.Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.Self-instruct: Aligning language models with self-generated instructions.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.) ,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.  13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.doi:10.18653/v1/2023.acl-long.754.URLhttps://aclanthology.org/2023.acl-long.754.Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.Finetuned language models are zero-shot learners.InInternational Conference on Learning Representations, 2021.Xu et al. (2023a) Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu.Rewoo: Decoupling reasoning from observations for efficient augmented language models.arXiv preprint arXiv:2305.18323, 2023a.Xu et al. (2023b) Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang.On the tool manipulation capability of open-source large language models.arXiv preprint arXiv:2305.16504, 2023b.Yan et al. (2024) Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez.Berkeley function calling leaderboard.https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html, 2024.Yang et al. (2024a) John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.Swe-agent: Agent-computer interfaces enable automated software engineering, 2024a.Yang et al. (2024b) Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.Gpt4tools: Teaching large language model to use tools via self-instruction.Advances in Neural Information Processing Systems, 36, 2024b.Yang et al. (2023) Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.Mm-react: Prompting chatgpt for multimodal reasoning and action.arXiv preprint arXiv:2303.11381, 2023.Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.React: Synergizing reasoning and acting in language models.InThe Eleventh International Conference on Learning Representations, 2022.Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2019.
conclusion
Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
6Conclusion