Automatic Task-aware Instruction Optimizer for Black-box LLMs
Large Language Models; Instruction Optimization
Yes
No ethics review needed.
1. Is the task context known by the white-box LLM? How can domain expert be involved in the design of instruction?

2. Does the Thompson sampling introduces bias of gradient estimation? What is the influence of hyperparameters $\beta$ and $B$?
1. The idea of automatic instruction optimizer is important to domain experts who do not have enough experience in prompting engineering. An added benefit would be increased transparency and trustworthiness between human and LLM agents.

2. The zero-order optimization framework with gradient approximation and decomposition is reasonable when involving many parameters.
This work proposed to fine-tune a white-box LLM to generate optimal instruction for the black-box LLM for downstream tasks. Under the assumption of linear optimization landscape of $\phi$, a zero-order gradient approximation approach was leveraged to update the parameters, in conjunction with a decision-making process that employed contextual bandits to select the most informative directions of gradient perturbation. Experiments of $15$ instruction induction and reasoning tasks showed the superior performance of the proposed algorithm.
1. The structure can be significantly improved for better clarification and illustration. There are many re-defined notations and formulations, such as $\phi$, $Theta_t$ and $\mathbf{z} \sim \mathcal{N}(0, I)$. Showing the trivial calculation process makes it difficult to grasp the core structure of the algorithm. It is suggested to first give the overall framework of the algorithm and then briefly present the calculation details.

2. The motivation of introducing Thompson sampling, in consideration of the zero-order gradient approximation, is confused. In addition, additional hyperparameters $\beta$ and $B$ were introduced without enough sensitivity analysis in experiments.

3. There is missing legend in Figure 3.
Yes
No ethics review needed.
The paper focuses on optimizing instructions based on task-specific feedback from a black-box LLM; how well does the optimized instruction generalize to related tasks or domains, especially in cases where task-specific data is sparse?
+ The paper introduces a unique framework, Automatic Instruction Optimizer (AIO), that optimizes instructions for black-box LLMs without requiring access to internal parameters, which is a critical advantage in real-world applications where only API-based models are available.
+ AIO’s use of zeroth-order gradient approximation with Thompson Sampling is innovative, as it enables effective instruction fine-tuning without direct gradient access, thereby overcoming a major limitation in black-box model optimization.
+ The framework significantly reduces API token consumption by reusing black-box LLM feedback adaptively, making it a cost-effective solution compared to existing in-context learning or manual tuning methods.
+ The paper also explores lightweight PEFT variants (e.g., LoRA and Linear Probing) that maintain performance with fewer parameters, showcasing AIO’s flexibility for different fine-tuning needs.
The paper proposes an instruction optimization framework called Automatic Instruction Optimizer (AIO) for black-box Large Language Models (LLMs). LLMs often rely on precise, task-relevant instructions to achieve high-quality outputs, but manual optimization is challenging, especially for black-box models where internal parameters are inaccessible. AIO addresses this by employing a white-box LLM, which can be fine-tuned to generate optimized instructions based on task-specific information and feedback from the black-box LLM. The optimization process involves a novel zeroth-order gradient approximation method, aided by Thompson Sampling, which bypasses the need for direct access to gradients, enabling effective fine-tuning. Extensive experiments demonstrate that AIO not only improves task performance but also reduces API token costs. Compared to existing methods, AIO achieves better performance on various tasks and maintains efficiency through the adaptive reuse of feedback. This framework represents a significant advancement in automating instruction generation, enhancing the usability and adaptability of black-box LLMs across diverse tasks​
+ The experiments mainly use llama+API pairing, which may not fully represent the framework’s effectiveness across a broader range of LLMs. I suggest the authors do more combinations of white-box LLMs, e.g., Mistral, for making the study of the method more comprehensive.
Yes
No ethics review needed.
Please see the weakness section.
The paper proposes the AIO for the instruction optimization. The AIO provides some theoretical analysis and demonstrations.
This paper proposes a task-aware instruction optimization for black-box LLMs. This method leverages both white LLM and black LLM. AIO adopts the parameter fine-tuning process with gradient approximation and contextual bandit techniques.
1.	The evaluation benchmarks in this paper are not very common. The proposed improvements should be demonstrated in the general benchmarks, including the instructions for the following benchmarks.
2.	The proposed method is unlike other task-aware instruction optimization methods. As there are many methods and literatures in this field, it would be beneficial to compare the proposed methods with other baselines.
3.	Although the white box LLM and black box LLM show improvements, it is obvious that this combination will perform better than the white box LLM and black box LLM individually. So the more analyses on these component designs are needed.
Yes
No ethics review needed.
Kindly refer to the weaknesses.
- This paper demonstrates that fine-tuning a white-box LLM into the instruction optimizer is able to have better instruction generation performance due to a stronger representation power.
- This paper addresses the optimization challenges using zeroth-order gradient approximation and manages to reduce the API cost by reusing historical records.
- It conducts comprehensive experiments to verify the effectiveness of the proposed method, including different combinations of white-box and black-box LLMs, instruction optimization trajectory results, comparison to CoT.
This paper mainly focuses on optimizing task-aware instructions in the absence of task-solver LLMs’ internal parameters and gradient information. Finding that the current instruction optimization methods based on in-context learning are insufficient to match the extreme complexity of modern black-box LLMs, this paper proposes a novel framework, called Automatic Instruction Optimizer (AIO), which fine-tunes a white-box LLM and enables it to adaptively adjust its instructing strategy for a particular task-solver black-box LLM. To address the optimization challenges, AIO adopts zeroth-order gradient approximation and Contextual Bandit techniques, eliminating the need for internal parameters and gradient information.
- A minor point: study the transferability of fine-tuned instruction optimizers across different black-box task-solver LLMs.
- The right-hand side of equation 1 misses a right parenthesis.
- Figure 3 lacks proper legends for different instruction optimization methods.
@inproceedings{
anonymous2024automatic,
title={Automatic Task-aware Instruction Optimizer for Black-box {LLM}s},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=m0ddLnNvXS},
note={under review}
}
Large Language Models (LLMs) have demonstrated superior capabilities in terms of solving various real-world tasks. However, their performance and generated content quality heavily depend on task-relevant instructions, which makes instruction optimization a challenging but critical direction to explore. In particular, as practitioners generally cannot access black-box (or API) LLMs' internal parameters and gradient information, it consequently makes instruction optimization for black-box LLMs especially non-trivial. Existing methods for optimizing black-box LLM instructions mainly focus on in-context learning using manually designed or heuristic disciplines, which can be insufficient due to the extreme complexity of modern black-box LLMs that can contain hundreds of billions of parameters. 
To address these challenges, we propose a novel automatic instruction optimization framework named Automatic Instruction Optimizer (AIO). AIO is designed to perceive target task information and adaptively adjust its task-aware instructing strategy for a task-solver black-box LLM. 
By leveraging a white-box LLM with parameter fine-tuning for enhanced representation power, AIO can automatically update its instructing strategy based on the feedback from task-solver black-box LLM. 
To achieve this goal, AIO adopts a novel LLM parameter fine-tuning process powered by zeroth-order gradient approximation and Contextual Bandit techniques, which can effectively and efficiently help address the challenge of inaccessible black-box LLM internal parameters and gradients, as well as help alleviate expensive API cost concerns by flexibly reusing collected black-box LLM feedback.
Extensive empirical evaluations are presented to demonstrate properties of our proposed AIO, and its effectiveness in comparison with strong baselines.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models; Instruction Optimization
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/09b5a58537db21fd8474a3161a38b8dcfa31d2d6.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/ecccfc2900f1a57b84282d90e0d5b5565bca7775.zip
Automatic Task-aware Instruction Optimizer for Black-box LLMs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission