GeckOpt: LLM System Efficiency via Intent-Based Tool Selection
Michael Fore Simranjit Singh Dimitrios Stamoulis
















  Microsoft Corporation
  Reston
  VA
  USA

mifore@microsoft.com



  Microsoft Corporation
  Silicon Valley Campus
  CA
  USA

simsingh@microsoft.com



  Microsoft Corporation
  Redmond
  WA
  USA

stamoulis.dimitrios@microsoft.com














In this preliminary study, we investigate a GPT-driven intent-based reasoning approach to streamline tool selection for large language models (LLMs) aimed at system efficiency. By identifying the intent behind user prompts at runtime, we narrow down the API toolset required for task execution, reducing token consumption by up to 24.6%. Early results on a real-world, massively parallel Copilot platform with over 100 GPT-4-Turbo nodes show cost reductions and potential towards improving LLM-based system efficiency.







<ccs2012>
   <concept>
       <concept_id>10010147.10010178</concept_id>
       <concept_desc>Computing methodologies Artificial intelligence</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010224</concept_id>
       <concept_desc>Computing methodologies Computer vision</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010179</concept_id>
       <concept_desc>Computing methodologies Natural language processing</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Artificial intelligence
[300]Computing methodologies Computer vision
[300]Computing methodologies Natural language processing






  




  
    < g r a p h i c s >

  Intent-based tool selection with . For each user query, the agent identifies relevant API libraries - without initially executing specific tools! Preliminary results indicate that streamlining the toolset reduces token costs associated with function calls by approximately 25%, while encouraging multi-tool execution over fewer GPT requests.
  An overview of our proposed GeckOpt framework.
  









GeckOpt: LLM System Efficiency via Intent-Based Tool Selection
    Dimitrios Stamoulis
    April 29, 2024
==============================================================










§ PRELIMINARY INVESTIGATION


Large language models (LLMs) augment platforms by facilitating the reasoning, planning, and execution of system tools via natural language instructions, thanks to advancements in prompting techniques <cit.>. These techniques, acting as LLM-based planners, determine the sequences of tools (i.e., underlying system APIs) to complete tasks. Despite the impressive LLM performance, this usually comes at considerable hardware cost.

In this initial examination, we embrace these advancements, while adopting a system-oriented view to identify bottlenecks in tool-calling processes. By analyzing an in-house large-scale Copilot system, namely  <cit.>, we observe that current compositional tool-calling methods often result in multi-step × single-tool executions (Fig. <ref> top). This contrasts with the more desirable multi-step × multi-tool approach, which aggregates multiple API calls within a single GPT step (Fig. <ref> bottom). Since each step translates to a GPT request and token consumption, the total number of steps significantly impacts system cost.

Our objective is to strike a balance between the inclination of compositional reasoning - breaking down tasks into multiple steps for task comprehension for the LLM-planner - and minimizing the system steps required for task completion. Our early analysis reveals that, as is to be intuitively expected, presenting the LLM with a narrower selection of tools, combined with a clearer understanding of the task, encourages the aggregation of more tools per step. To this end, we draw inspiration from the concept of LLM intent, where state-of-the-art methodologies demonstrate LLMs' capability to discern “user intent” and utilize it to categorize tasks into granular user-defined templates <cit.>.

: Our methodology begins with an offline phase where tasks are mapped to intents and associated tools with minimal human involvement (Table  <ref>). Next, at runtime and for each prompt, we first engage the LLM agent to determine the task's intent and to identify the relevant subsets of API libraries - ahead of pinpointing specific tools at this stage! This intent-driven “gating,” while it incurs the minor cost of an extra API call, effectively narrows down the tool selection pool for all subsequent compositional prompting to proceed as usual. The result is a reduction in token requirements and a move towards executing multiple tools within a single API call, optimizing both token usage and system resource allocation. Given that our approach is fully GPT-driven, it adeptly handles failures where the initial API selection is incorrect, with the agent being instructed via prompting to revert to the full toolset.



§ PRELIMINARY RESULTS


Experimental setup: as a representative Copilot platform, we use our  framework <cit.> that spans several remote sensing tasks, open-source datasets, and API tools for loading, filtering, processing, and visualizing data. We report agent performance metrics on the  benchmark for CoT <cit.> and ReAct <cit.> prompting. We refer the reader to <cit.> for further details.

Results: Table <ref> shows that  reduces token usage across various baselines up to 24.6% with small performance variation. While we note slight deviations, possibly attributable to employing non-zero temperature settings in LLM function calling to foster creative answers in visual question answering, we observe that cost reductions come at negligible performance degradation within 1% in terms of success rates. Extrapolated across numerous user sessions over our cloud-native platform, such token reduction by almost a quarter can culminate in substantial cloud cost savings. 

Limitations - Future Work: We note that this is a preliminary investigation where geospatial tasks are well-suited to intent-based granularity. An important next step is to confirm the generalizability of the methodology across various function-calling benchmarks. Last, as our study considers only cloud endpoints, we are currently expanding to local LLM execution. Given the correlation between token usage and operational costs, we anticipate additional improvements for LLM system utilization, underscoring the potential of intent-based tool selection for enhancing hardware efficiency.

ACM-Reference-Format


