Coding Reliable LLM-based Integrated Task and Knowledge Agents with GenieWorksheets
conversational agents;framework;task-oriented dialog agents;task and knowledge agents
Yes
No ethics review needed.
What’s the key intuition behind Genie's controllability? Is it essentially a combination of a hand-crafted workflow and LLM? How can we guarantee that the LLM strictly follows the policy?

I believe a more detailed running example that demonstrates the entire process concretely would be very helpful for understanding the specific design and underlying mechanisms.
This paper presents a new framework for LLM-based ToD agents that addresses several key challenges in the field. It provides mechanisms for developers to better enforce agent controllability, support complex compositions of queries and API calls, and more. Experimental results demonstrate the system’s effectiveness.
The paper introduces Genie, a framework designed to make LLM-based task-oriented agents more reliable and user-friendly. Traditional conversational agents struggle with handling complex logic, maintaining context, and accurately responding to dynamic user inputs.
 Genie tackles these issues with the Genie Worksheet—a high-level, declarative specification tool that lets developers define agent behavior, control conversation flow, and integrate knowledge sources directly. The result is an agent that can handle mixed user queries, remember important details across long interactions, and reliably follow developer-defined instructions. In tests, Genie outperformed state-of-the-art models and GPT-4-based systems on several metrics, showing a big boost in accuracy and task completion.
1. Overall, I find this paper somewhat difficult to follow, though this might be due to my limited background in dialogue system literature. I would suggest adding a more concrete, running example to illustrate the workflow more clearly. Currently, the example in Figure 1 feels too high-level and abstract.

2. One key challenge addressed in this paper is that LLMs may not reliably follow instructions in a purely LLM-based setting, whereas Genie provides better mechanisms for enforcing controllability. What’s the key intuition behind this? Is it essentially a combination of a hand-crafted workflow and LLM? This might need a better explanation in the paper.

3. Additionally, the paper seems to present a comprehensive system rather than a focused core message or insight, which might make it more suitable as a demo paper than a research paper. It would benefit from a more explicit explanation of the main scientific question it addresses and a clearer, more focused insight.

Addressing these questions would greatly enhance my understanding of this work, and I may reassess my judgment based on the feedback.
Yes
No ethics review needed.
1. what happens if GENIE is run on top of T5 or the previous SOTA was implemented on top of GPT-4-turbo?
1. The main contribution of this paper is Genie Worksheet, a new, declarative language to support dialog policy specification, dialog knowledge integration, and dialog state tracking. This seems like a conceptually neat, and empirically useful idea in support of the developers of conversational agents. 

2. When Genie Worksheet is integrated to a base LLM via in-context-learning, the authors are able to achieve promising empirical results:

On the STARV2 dataset, GENIE (based on GPT-4-turbo) significantly outperforms the prior SOTA method (based on T5) by up to 20.5% in system action F1 scores. This is technically a new SOTA on this benchmark, though I would've preferred to see a more careful ablation study as explained in the weaknesses section below.

In real-world user studies, GENIE (based on GPT-4-Turbe) also achieves notably higher execution accuracy, dialogue act accuracy, and goal completion rates compared to a GPT-4-Turbo with function-calling baseline.
This paper introduces GENIE, a programmable framework designed to make it easier for developers to rapidly build conversational agents by specifying the desired behavior of the agent via declarative language. 

The main technical contribution of this paper is Genie Worksheet, a declarative specification language that serves multiple objectives, including (1) allowing developers to define complex agent behaviors as declarative policy, (2) integrating both structured and unstructured knowledge into the conversation history to support accurate, knowledge-informed conversations, (3) supporting contextual and complex managements of the dialogue state tracking, even through the complex mixed-initiative scenarios. 

The authors present promising empirical results where GENIE (based on GPT-4-turbo) outperforms previous state-of-the-art models (based on T5) as well as GPT-4-turbo with function calling, validated by both existing benchmarks and real-user studies. 

In a nutshell, this paper demonstrates that with a symbolically guided design of a worksheet that specifies the conversation history and policy carefully, it is possible to considerably increase the performance and the reliability of LLMs serving as knowledge-intensive conversational agents. This finding is perhaps analogous to the recent trend of test-time inference methods that can increase the performance of the base LLMs. The proposed approach relies on the capabilities of underlying LLMs to accurately work with the Genie Worksheet via In-Context-Learning instructions and examples, thus is subject to the errors that may be caused by the underlying LLMs. 

Overall this paper reads a bit like a white paper of a product release due to the emphasis on the design objectives for the developers and relatively less focus on the technical implementations, thorough ablation studies, or comprehensive comparisons against other competitive baselines.
1. One caveat of the STARV2 result is that the previous SOTA is from 2022, and it was based on a significantly weaker base LLM, T5. Thus it's not clear whether the major performance is coming from the Worksheet or from the base LLMs.

2. In the abstract of the paper, it is stated that GENIE outperforms GPT-4 with function calling. Initially I thought GENIE based on GPT-4-turbo was outperforming GPT-4, but later it turns out what the authors meant by GPT-4 was still GPT-4-Turbo. It would be good to correct this to avoid potential confusion. (otherwise the claim reads bigger than what it is.)

3. The technical details of this paper is rather thin. For example, the fact that Genie is based on GPT-4-Turbo is mentioned only though the footnote 1, which points to Appendix B for further details. When I reached to Appendix B, there were just two short sentences. I'm not sure if others can successfully reproduce the work presented in this paper based just on what's provided in this paper.
Yes
No ethics review needed.
For the GPT4-function calling baseline. I’m wondering how are you constructing the prompts? For Genie, you provide the GPT-4 with dialog history, the available functions and task-specific workflows in your prompts, does the baseline method have the access to the same information?
1. The proposed Genie worksheet seems to provide a easier way for non-technical developers to build LLM-based agents with customized work flows. 

2. The user studies show that there is a significantly higher preference rate for Genie framework.
This paper proposed a agent framework called Genie worksheets, which allows developers to easily build customized LLM agent workflows for specific downstream applications. In particular, the developers only needs to specify the available knowledge bases, the database schema and a few in-context examples or expected workflows in the worksheet, and then the LLM agent will be able to assist users by interacting with the knowledge bases and ask follow-up questions as needed. The authors evaluated the proposed framework on a static dialog state tracking dataset and conducted user-studies to measure the user’s preference between Genie and GPT-4 function calling.
I find the paper hard to follow. I’m not sure if this work should be placed under the LLM agent framework category or the agent demo system category. If the main contribution is a new agent framework, then it should be compared against works such as LangChain [1] or AutoGen [2], and the authors should provide a detailed discussion about the novelty of this work. Per my understanding, Genie just allows the developer to specify APIs and workflows using the spreadsheet/excel and other frameworks like Langchain just achieve those purpose with python and prompts. If this is the case, then I don’t see too much technical novelty of Genie compared to prior works. 

[1] https://github.com/langchain-ai/langchain

[2] Wu, Qingyun, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger and Chi Wang. “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation.” (2023).
Yes
No ethics review needed.
I don't have questions.
The paper presents a novel framework called Genie, designed to create task-oriented dialogue agents capable of handling complex user interactions and knowledge queries. It effectively summarizes the limitations of existing dialogue agents in managing conditional logic, integrating knowledge sources, and adhering to consistency in instructions. The introduction of Genie Worksheet, a programmable framework, offers developers a declarative approach to define dialogue tasks, knowledge base schemas, API interfaces, and examples, thereby enhancing the expressiveness of large language model (LLM) parsers. The evaluation of Genie across restaurant reservations, course registrations, and ticket submissions demonstrates its potential to optimize dialogue performance. Furthermore, the inclusion of testing results in the appendix, which showcases both successful and less effective outcomes, aids other researchers in understanding the strengths and possible weaknesses of the framework.
This article introduces a new framework called Genie, designed to create task-oriented conversational agents capable of handling complex user interactions and knowledge queries. Genie is implemented through a programmable framework called the Genie Worksheet, which allows developers to define dialogue tasks, knowledge base schemas, API interfaces, and examples in a declarative manner for the large language model (LLM) parser to understand. Genie’s core advantage lies in its ability to provide reliable and controlled agent strategies, achieved through its highly expressive specifications.
Despite simplifying the development of dialogue agents, Genie Worksheet may still pose challenges for non-expert users, indicating a need for more automated approaches to building and utilizing the Genie framework. Additionally, the focus on specific applications, such as restaurant reservations and course registrations, raises questions about Genie’s generalizability across broader domains, which requires further validation. In real-world user studies, Genie may struggle when users refuse to provide information or frequently change their responses, highlighting a limitation in practical performance and suggesting areas for future optimization. Lastly, while the paper emphasizes performance improvements, it would benefit from including results related to the resource consumption of Genie. It appears that Genie may consume significantly more resources compared to traditional dialogue agents, which could present a drawback in practical applications.
Yes
No ethics review needed.
1. Worksheet Construction: How are worksheets created? Are column names fixed as in Figure 2, or can they be customized based on task requirements?

2. Worksheet Conversion for LLM Input: How is the worksheet formatted for input into the LLM? Specifically, what process is used to convert the two-dimensional table structure into a suitable prompt?

3. Reproducibility and Open-Source Plans: Given the complexity of the methodology, replicating the approach based solely on prompts may be challenging. Are there plans to open-source the code to facilitate reproducibility?
1. Innovative Worksheet Structure: Genie provides a structured approach using Genie Worksheets, allowing precise control over task-oriented agent behavior and knowledge integration.

2. Improved Reliability: By combining symbolic rules with LLM-based responses, Genie reduces errors like hallucinations and improves task consistency, enhancing reliability in complex dialogues.
The paper introduces Genie, a framework for developing reliable, task-oriented conversational agents powered by large language models (LLMs). Using Genie Worksheets, a high-level specification language, developers can precisely define conversation flows, task logic, and access to both structured and unstructured knowledge sources. This structured approach enables Genie agents to manage complex, context-sensitive dialogues with high accuracy. Unlike conventional LLMs, Genie uses a symbolic agent policy, reducing common issues like hallucinations and inconsistency. Genie significantly outperforms models like GPT-4 in real-world scenarios, such as restaurant reservations and course enrollment, achieving superior execution accuracy and task completion rates.
1. Limited innovation: This work utilizes a worksheet structure to support overall dialogue understanding, which is then used for DST and policy prediction, followed by response generation. The methodological innovation is limited.

2. Insufficient Experimental Comparisons: The paper lacks comparisons with other LLM-based methods designed for task-oriented dialogue (TOD) and omits performance benchmarking against other large models like Llama, and other TOD datasets like MultiWOZ.

3. Reproducibility and Transferability Concerns: With no code provided and experiments limited to specific domains, the method's reproducibility and adaptability to new tasks are uncertain.
A programmable framework for creating reliable LLM-based task-oriented agents that integrate various knowledge sources.
@inproceedings{
anonymous2024coding,
title={Coding Reliable {LLM}-based Integrated Task and Knowledge Agents with GenieWorksheets},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=G6S9B7fr74},
note={under review}
}
Large Language Models (LLMs) present an opportunity to create automated assistants that can help users navigate complex tasks. However, existing approaches have limitations in handling conditional logic, integrating knowledge sources, and consistently following instructions. Researchers and industry professionals often employ ad hoc pipelines to construct conversational agents. These pipelines aim to maintain context, address failure cases, and minimize hallucinations, yet frequently fail to achieve these objectives. To this end, we present Genie – a programmable framework for creating task-oriented conversational agents that are designed to handle complex user interactions and knowledge queries. Unlike LLMs, Genie provides reliable grounded responses, with controllable agent policies through its expressive specification, Genie Worksheet. In contrast to dialog trees, it is resilient to diverse user queries, helpful with knowledge sources, and offers ease of programming policies through its declarative paradigm. The agents built using Genie outperforms the state-of-the-art method on complex logic domains in STARV2 dataset by up to 20.5%. Additionally, through a real-user study involving 62 participants, we show that Genie beats the GPT-4 with function calling baseline by 21.1%, 20.1%, and 61% on execution accuracy, dialogue act accuracy, and goal completion rate, respectively, on three diverse real-world domains.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
conversational agents framework task-oriented dialog agents task and knowledge agents
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/eb84ab71fadf87a56f819b747ae6be8585ead55c.pdf
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/daf8a4e92d6af5593322e0b50d04dcd0ba4c3dd6.zip
Coding Reliable LLM-based Integrated Task and Knowledge Agents with GenieWorksheets
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission