Task Facet Learning: A Structured Approach to Prompt Optimization
prompt optimization;large language model
Yes
No ethics review needed.
The impact of mini-batch size. 

The impact of aggregation.
The strategy for creating mini-batches to generate edits and then aggregating them at the batch level to yield the final edit in a feedback mechanism is interesting. 

Experimental results achieves SOTA.
This paper proposes a prompt optimization method $U_{NI}P_{ROMPT}$ to generate complex prompts for covering multiple facets of a task and improving overall accuracy. Its effectiveness is demonstrated through relevant experiments.
1.The paper contains some detailed issues, e.g., $U_{ni}P_{rompt}$ mentioned in  line 100-101; an extra space before "So" mentioned in line 197-198. 

2. The experiments should be re-organized and polished carefully before submission, especially in Section 4.1,  4.2 and 4.5. 

3. Some qualitative experiments are supported be added to validate the differences in report accuracies under various mini-batch sizes.
Yes
No ethics review needed.
See weaknesses.
- UNIPROMPT clusters examples to capture task-specific facets effectively, with a two-tier feedback mechanism ensuring that prompt modifications are generalizable and not overly specific to individual examples.
- The authors provides strong empirical results, with UNIPROMPT outperforming state-of-the-art methods on multiple datasets, notably achieving significant improvements on hate speech classification.
- By focusing on automatic prompt generation without relying heavily on human-engineered prompts, this approach may reduce the manual overhead of prompt tuning.
- Extensive ablations validate the contribution of each component (clustering, feedback structure) to performance.
The paper introduces UNIPROMPT, a method that uses task facet learning to enhance prompt optimization by breaking down prompts into distinct semantic sections, targeting diverse facets such as counter-examples and explanations.
- All experiments are conducted on powerful proprietary LLMs, I wonder whether the proposed method remains effective for opensource LLMs such as LLaMA or Qwen.
- It seems that the proposed method underperforms DSPy on MedQA, while surpassing it on other benchmarks, what might be the reason?
- How to determine the number of clusters when constructing the minibatches? How does this affect the method's effectiveness?
Yes
No ethics review needed.
N/A
(1) The paper introduces a new method for prompt optimization and consider multiple facets of a task and leveraging the structure in the prompt.
(2) The proposed clustering approach and two-tier feedback mechanism provide a systematic way to generate prompt edits that capture generalizable concepts.
(3) The experimental results on several datasets shows that UNIPROMPT outperforms human-tuned prompts and SOTA methods.
This paper introduced a method for prompt optimization called UNIPROMPT. The prompt optimizaiton is deemed as the task of learning multiple facets of a task. UNIPROMPT exploit the structure in the prompt optimization and use a clustering approach to group examples with similar task facets. Experimental results on multiple datasets demonstrate that prompts generated using UNIPROMPT achieve higher accuracy than human-tuned prompts and state-of-the-art methods.
(1) I am skeptical about the current importance of prompt optimization. Prompt optimization is more of a need in an era when large models are not powerful (for example last year). Nowadays, the instruction model follows human instructions very well, and many of them are 0-shot, see llama-3.1 model card. I would like to hear the author's opinion on this.

(2) For the results in table 1, it is recommended to add an average column to make the presentation clearer.

(3) I found that on the ARC task, compared with the llama prompt, the optimized prompt led to a decrease in performance. Is there any explanation?
Yes
No ethics review needed.
1. If I understand correctly, the wrongfully predicted training samples are clustered only once at onset, how does the algorithm handle the wrong samples after the optimization starts? Does it use them to generate feedback?

2. Did you try using other model families? How does the performance look like when the expert model is not as powerful as GPTs? 

3. The clustering is conducted through LLM, but this could be an issue when the number of examples increases. Did you try other clustering approach? which might make it feasible to dynamically cluster the examples along the iterations.
1. The paper is well-structured and the story is well-told. It starts with essential observations that motivate the improvements so that the approach bears heuristic justification. 

2. Experiments are comprehensive, including several baselines, and benchmarks of different types. Experiment results also provide strong support to the thesis of the paper.
This paper presents an improvement to current prompt optimization approach. It does so by clustering the input space and enabling the optimization algorithm to learn multiple facets of a task from a set of batched clusters. The paper discusses the empirical observation that inspires the improvement and conducts experiments to demonstrate its efficacy. Results suggest that the framework achieve better results than existent approach.
My concern on the improvement is its generalization ability. The experiments cover hate speech detection, math, QA but there are many other task types not covered in this dataset. For example, OPRO and [Li et al 2024](https://arxiv.org/abs/2409.15199) ran experiments on BBH which contains a much more diverse task portfolio than presented in the paper. Since the proposed improvement centers around clustering the training examples, whether the samples can be cluster into meaningful subspace has a direct impact on optimization performance. But we don't see sufficient evidence to demonstrate the approach works on a divers task portfolio. 

Besides, the prompt optimization is very sensitive to the initial prompt and iterations (also discussed in Sec 2.1), but the reported metrics don't take the performance variation into consideration. I have concern on the robustness of the results.
@inproceedings{
anonymous2024task,
title={Task Facet Learning: A Structured Approach to Prompt Optimization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ViRDmDAfjg},
note={under review}
}
Given a task in the form of a basic description and its training examples, prompt optimization is the problem of synthesizing the given information into a text prompt for a large language model. Humans solve this problem by also considering the different facets that define a task (e.g., counter-examples, explanations, analogies) and including them in the prompt. However, it is unclear whether existing algorithmic approaches, based on iteratively editing a given prompt or automatically selecting a few in-context examples, can cover the multiple facets required to solve a complex task.  In this work, we view prompt optimization as that of learning multiple facets of a task from a set of training examples. We exploit structure in the prompt optimization problem and break down a prompt into loosely coupled semantic sections. The proposed algorithm, UniPrompt, (1) clusters the input space and uses clustered batches so that each batch likely corresponds to a different facet of the task, and (2) utilizes a feedback mechanism to propose adding, editing or deleting a section, which in turn is aggregated over a batch to capture generalizable facets. Empirical evaluation on multiple datasets and a real-world task shows that prompts generated using UniPrompt obtain higher accuracy than human-tuned prompts and those from state-of-the-art methods. In particular, our algorithm can generate long, complex prompts that existing methods are unable to generate.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
prompt optimization large language model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b37605753266f69c5b8431b9395dda4e457fed66.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Task Facet Learning: A Structured Approach to Prompt Optimization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission