WIQOR: A dataset for what-if analysis of Operations Research problems
datasets;operations research;counterfactual reasoning;mathematical reasoning
Yes
No ethics review needed.
See weakness.
And the right part in figure1. is vague.
- The task is novel, challenging, and highly relevant for real-world applications.
- The dataset is thoughtfully designed, with components like problem specifications that enhance clarity and usability.
- The task is easily scalable; complexity can be adjusted by increasing the number of variables, as demonstrated in this work.
- The task can be effectively generated and tackled by large language models like LLaMA, achieving promising accuracy levels.
This paper introduces a new task called Mathematical Program Modification (MPM), where a mathematical program is revised in response to a natural language inquiry, referred to as "what-if" questions. The authors also examine how problem complexity, specifically the number of variables, impacts model performance. Experiments with the Llama-3.1 series show that more challenging problems remain difficult for large language models, indicating significant room for improvement in handling complex modifications.
My main concern lies in the evaluation aspect:
- The paper evaluates only three models from the LLaMA series. As a benchmark, this is somewhat limited. Could the authors include evaluations from more advanced models, such as GPT-4 or Claude-3.5? This would give the community a clearer understanding of model performance on this task.
- The what-if questions are generated by LLaMA models and then evaluated by similar models, which may introduce potential biases. Including results from a broader range of models would help clarify whether such biases are present.
- The paper could benefit from additional analyses, such as self-consistency performance and pass@k accuracy. These insights could add further depth to the evaluation.
Yes
No ethics review needed.
- I believe many experimental results are missing. For instance, have you evaluated whether the generated solution formulas are solvable? Additionally, numerous generated problems need to be tested on the solver.
- How is data quality ensured, given that many problems generated by the model often do not exist in the real world? Are there any automatic checks or manual evaluation steps to filter out unrealistic problems? I believe that the validity of the generated problems needs to be supported by detailed experimental results to be convincing.
- How is data diversity ensured, considering that simple semantic similarity is typically not a good metric for mathematical reasoning tasks?  I believe it is necessary to provide some metrics or analysis to demonstrate the diversity of the dataset and to ensure that mode collapse has not occurred.
- The paper proposes a self-instruction-based approach to synthesize complex reasoning data, using a progressive method to generate more complex problems and solutions.

- The article is well-written, and the method for expanding the dataset is simple and clear.

- It explores some interesting combinations of conditions and variables, which is very helpful for investigating the boundaries of linear programming problems.
This paper develops a novel dataset, using the dataset from NL4OPT as a seed to expand the original dataset. The complexity of generated problems, as well as the number of variables and constraints, is gradually increased to obtain more complex data. The provided canonical form is correctly modified in response to the what-if question.
- The novelty of the data generation method is limited[1,2]. I believe that the self-instruction-based method may face serious mode collapse[3] issues, and it’s necessary to discuss the presence of this problem in mathematical problem synthesis. As for improvements, I think a human-in-the-loop approach could be provided for optimization.
- Using exact match to calculate accuracy is unreasonable; why not provide results on the solver instead? Exact match can be overly restrictive and often misleading in evaluating mathematical and reasoning tasks, as it treats answers as binary (correct or incorrect) without considering near-correct solutions that may only have minor calculation differences. Real-world problem-solving often requires that a solution meet certain functional criteria, like feasibility within constraints or optimality, rather than matching a single predetermined answer exactly. Using solver-based evaluations provides a nuanced assessment, confirming if solutions meet problem requirements under real conditions. Solvers can verify the feasibility and robustness of a solution within the problem’s constraints, accounting for slight variances and rounding errors that exact match might unfairly penalize. This approach aligns the evaluation with real-world performance, offering clearer insights into the model's practical applicability and helping to identify areas for meaningful improvement. By assessing solutions based on their functionality and effectiveness, solver-based evaluation ultimately yields a more refined measure of a model's problem-solving capability, moving beyond rigid surface-level accuracy.
- Can it be extended to more complex mixed-integer programming problems? Mixed-integer programming introduces additional complexities compared to standard linear programming due to the presence of integer constraints, which often lead to a more challenging solution space and can significantly increase the computational difficulty of finding optimal solutions. For example, extending this method to MIP might require adjustments in handling discrete variables, which can limit certain relaxation techniques commonly applied in continuous optimization problems. Additionally, it would be important to investigate how the current framework could address or adapt to the combinatorial nature of integer constraints. This could involve rethinking the approach to feasible region exploration, as well as adapting constraint handling mechanisms to account for mixed integer domains. Overall, this extension could open up new application avenues, and exploring the specific requirements for such an adaptation might reveal valuable insights into the method's versatility and practical scalability across diverse optimization scenarios.
- I haven't seen any code or datasets; I would like to carefully check whether the data questions truly align with real-world solving objectives. 
- I believe it is essential to provide the solvability rate of the generated problems and to include test results across different solvers, comparing their performance.

**Reference**  
[1] Self-Instruct: Aligning Language Models with Self-Generated Instructions.  
[2] Wizardlm: Empowering large language models to follow complex instructions.  
[3] Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data
Yes
No ethics review needed.
- Are all the MP questions included Linear programming? If not, did you treat linear and non-linear differently or did you investigate their performance difference?
- What's the actual prompt you used to evaluate the performance of Llama-3.1? Did you include specification in your prompt? Please show a template example.
- You specification S does not contain specific description about the variables. Does it mean after the modification, the specifications are still guaranteed to be valid?
- What's the motivation for rewriting the optimization problem to the specifications?
- The collected dataset could be a contribution to the community, however if it is actually emulating the real-life scenario is still questionable.
- It is interesting to see how different constraint types and what-if question types pose different levels of challenges for LLMs.
- This paper proposed a new task, mathematical program modification (MPM), whose goal is to revise a math program (MP) based on a natural language instruction.
- The paper presented a dataset WIQOR, of 1946 MPM instances, derived from NL4OPT dataset. WIQOR contains 4 set, train, dev, test-base and a harder test-VarAug. To build the dataset, the authors first modify the original MP with 4 potential ways (LC, CC, CDR, VIM, explained in paper), and then prompt LLMs, as a reverse engineering approach, to generate the **What-if questions** based on the difference between the original MP and the modified MP.
- The author test 3 variants of Llama model on the proposed dataset and the experiment results show it is a challenging dataset.
- This paper is clearly a rushed work. The presentation is draft-like ( For example, the margin of Figure 6, the unfinished appendix A1,2,3), the writing is a bit verbose ( for example Section 2 related work and Section 3.1 & 3.2).

- The evaluation section is limited. 
    - As shown in Table 3, the dataset is only evaluated on very few models, Llama-3.1-8B/70B and a code-Llama. 
    - When comparing ICL variant, zero-shot performance is not tested. 
    - When comparing CoT performance, ablation study should be conducted to compare the performance with and without CoT.
    - As very few models are evaluated, the conclusions drawn in Section 6 is not well supported.

- The motivation for the proposed dataset is not convincing enough, and the scope of the project is relatively small.
    - The whole motivation of the dataset is built on the assumption that in real-life people will communicate/modify a MP in such a way, but your specification S is just summary and do not contain detailed information. Why do you think that's the actual way people communicate?
    - The proposed modification approach should be compared with a NL-to-MP baseline, where the original optimization text + modifications is given to the LLM to generate a new MP. If this actually works better than modifying the existing MP, I don't see the motivation for this approach.
    - How good is the coverages of What-if questions? Do they cover all potential aspects of a MP problem? Are all the MP problems linear programming problems? You can include a limitation section to discuss the coverage of the What-if questions.
We introduce a task to modify mathematical programs based on natural language requests and provide a dataset to support this, aiming to enable non-technical users to interact with optimization models.
@inproceedings{
anonymous2024wiqor,
title={{WIQOR}: A dataset for what-if analysis of Operations Research problems},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=no8ysO3xde},
note={under review}
}
We formalize the mathematical program modification (MPM) task, in which the goal is to revise a mathematical program according to an inquiry expressed in natural language. These inquiries, which we refer to as what-if questions, express a desire to understand how the optimal solution to an optimization problem changes with the addition, deletion or revision of constraints. In detail, each MPM instance
is a triple consisting of: 1) a natural language specification that summarizes an optimization problem, 2) the canonical formulation of the problem, and 3) a natural language what-if question. The goal is to predict the updated canonical formulation with respect to the question. To support the study of this task, we construct WIQOR, a dataset of 1,946 MPM instances, derived from NL4OPT (Ramamonjison et al., 2023), but with the number of decision variables extended to more than 30 for some problems. In experiments, we observe that Llama 3.1 70B instruct under the in-context learning paradigm achieves 69% accuracy on the easiest test instances, but only 36% accuracy on the most complicated problems. We release WIQOR in the hopes of spurring additional study of MPM and ultimately enabling non-technical users to conduct what-if analyses without the help of technical experts.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
datasets operations research counterfactual reasoning mathematical reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a296eed66036610f6598ec460e43e40affd71685.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
WIQOR: A dataset for what-if analysis of Operations Research problems
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission