Evo-Step: Evolutionary Generation and Stepwise Validation for Optimizing LLMs in OR
Large language model; Operations Research; Automated Modeling
Yes
No ethics review needed.
1. How many queries are utilized during instance generation? What is the total cost? What percentage of samples are discarded during the data generation process?
2. The authors mention testing both COPT and GUROBI. It is unclear whether they use the same independent data generation pipeline and prompts for each. 
3. Are all experimental results based on COPT, GUROBI, or are they selecting the best outcomes from either? Additionally, if OMLR is only trained with COPT, is it appropriate to use the original checkpoint directly?
4. What would be the impact of directly applying the proposed "Stepwise Validation Mechanism" to SOTA LLMs, such as GPT-4, in modeling tasks? Would this significantly enhance accuracy and performance?
1.The framework automates the data generation process, enhancing efficiency and effectiveness.
2. The paper is well-structured and easy to follow, facilitating comprehension.
This paper proposes an evolutionary framework for data generation for training LLMs for OR optimization modelling. The data generation consists of two parts, with the first evolutionary one, for instance, generation and the second one for solution assessment and refinement. The results are compared on three OR datasets of different difficulty. Promising results are generated when compared to existing LLMs with prompt engineering techniques and domain OR LLMs.
1. Although the results contribute to specific OR optimization problem modeling, the technique's contribution appears incremental, as the evolution process in data generation has been previously explored.
2. It is recommended to include more comparative studies with SOTA LLMs and provide additional explanations to further validate the results.
Yes
No ethics review needed.
1. How do you fine-tune the LLMs? What technique do you use? Details  are lacking.
2. What is the contribution of this work in solving OR problems? It seems to focus on more how to generate problems and validation, which might be more trivival compared to solve problem itself.
1. This work considers different aspects such as problem generation, validation and LLMs fine-tune.
2. The proposed method achieves good performance on different operation problems.
This work aims to enhance large language models (LLMs) for complex optimization in Operations Research by evolving problem complexity and integrating real-time validation. Fine-tuned on generated data, the Evo-Step model significantly outperforms benchmarks, improving accuracy on challenging tasks, demonstrating the power of evolutionary problem generation for advanced decision-making automation.
1. The presentation is poor especially the diagrams. Their captions are so short to understand what they intend to convey. The caption should be significantly expanded and improved.
2. There are some inaccurate expressions. For example, evolutionary strategy specifically refer to a branch algorithm in the evolutionary optimization instead of an arbitrary strategy related to evolution. 
3. The problem formulation needs further clarification. Normally, we propose a method/framework to solve a or several problems. However, you are trying to solve a large number of problems simultaneously. How to ensure its effectiveness would be a very serious issue. In addition, the problem to solve is also not clear to readers.
4. I suggest to use the proposed method to solve some OR problems and compare it with some traditional methods instead of LLM-based method, which can help demonstrate the effectiveness of your method.
Yes
No ethics review needed.
Due to the presentation, it it hard to fairly evaluate this work at the current stage. As such, I will reconsider this work after the following queries have been addressed:

(1) In this work, I cannot see any evolutionary components such as crossover and mutation. How do you evolve and what do you evolve?

(2) The contribution of this work is not clear or even a bit confused. This work states "increases the complexity of generated problems", why we need to generate problems on our own? A more convincible direction should be that we use LLMs to generate solution to the problems.

(3) How do you fine-tune the open-source LLMs? What technique do you use? 

(4) More experiments are required such the the comparison with tradition algorithms (non-LLM approach). LLMs are unnecessarily suited to every task, thus it is important to justify the strength of LLMs in this task.
Evo-Step exhibits outstanding performance, particularly in managing complex OR tasks, with a significant 17.01% increase in micro-average accuracy on challenging problems.
This study proposes Evo-Step-Instruct, a new framework that incrementally escalates problem complexity using an evolutionary approach to strengthen LLMs' performance in optimization modeling. The framework employs progressive validation to enable immediate error identification and correction.
The presentation requires substantial improvement, as the current version is difficult to comprehend and does not clearly convey the contributions. Here, I will first list some shortcomings:

(1) The caption of figures is not informed enough and it is difficult to understand the content of diagrams.

(2) Evolutionary Strategy (ES) typically refers to a type of evolutionary algorithm focused on optimizing complex problems by mimicking natural selection processes. However, I cannot see any element of Evolutionary Strategy in Figure 1, which is claimed as the example of evolutionary strategy.

(3) The citation format is not good and needs improvement.
Yes
No ethics review needed.
Question 1: Could the authors elaborate on Evo-Step’s computational requirements and its scalability for larger datasets or in different domains?

Question 2: How adaptable is Evo-Step’s framework for domains outside OR, and what modifications would be necessary?
The evolutionary strategies combined with real-time validation ensure high-quality data generation, reducing the need for post-processing, and the approach significantly outperforms baseline models on complex OR problems.
Evo-Step-Instruct is a framework that uses evolutionary problem generation and stepwise validation to enhance LLMs in Operations Research (OR) tasks. Evo-Step incrementally builds datasets of increasing complexity, applying validation checks to prevent error propagation, and yields strong results on OR benchmarks like NL4OPT and IndustryOR.
The computational demands of the approach could limit scalability, and Evo-Step’s OR-specific design may constrain its applicability to other domains.
@inproceedings{
anonymous2024evostep,
title={Evo-Step: Evolutionary Generation and Stepwise Validation for Optimizing {LLM}s in {OR}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=aapUBU9U0D},
note={under review}
}
Large Language Models (LLMs) have revolutionized various domains, but they face challenges when applied to highly specialized fields such as Operations Research (OR). In this work, we present Evo-Step-Instruct, a novel framework that progressively increases the complexity of generated problems using an evolutionary strategy, aimed at enhancing the capabilities of LLMs in optimization modeling. Our framework integrates stepwise validation, which ensures real-time error detection and correction during data generation, thereby improving data quality and preventing error propagation. We fine-tune open-source LLMs, such as LLaMA-3-8B and Mistral-7B, using the generated high-quality dataset, resulting in a model, Evo-Step, that significantly outperforms baseline approaches on key benchmarks including NL4OPT, MAMO, and IndustryOR. Through extensive experiments, Evo-Step demonstrates superior performance, especially in handling complex OR tasks, achieving a notable improvement of 17.01% in micro average accuracy on difficult problems. Our approach represents a substantial advancement in automating complex decision-making processes using LLM,  showcasing the potential of combining evolutionary problem generation with structured validation for fine-tuning LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large language model; Operations Research; Automated Modeling
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/52c19ec3d1c4c5a23fb4f5bd323a4d95f5a5fd73.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Evo-Step: Evolutionary Generation and Stepwise Validation for Optimizing LLMs in OR
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission