COOL: Efficient and Reliable Chain-Oriented Objective Logic with Neural Networks Feedback Control for Program Synthesis
program synthesis;Chain-of-Logic;neural network feedback control
I admit that the paper is complex, but all content is well-reasoned and clearly presented. I understand that your confusion may stem from the lack of references to the appendix within the main text sections of the paper. I will ensure to provide these references for clarity and hope that you can spend more time reviewing this paper:

Firstly, in the "Summary," you mentioned, "The submission discusses a technique for synthesis of logical rules from data." Upon review, the paper focuses on DSL program synthesis, where rules are employed to transform a partial program into a complete one. This focus is clearly articulated in the Abstract, Introduction, and Method sections, rather than centering on data synthesis rules as initially stated.

In the "Weakness", you said that the paper was poorly presented and that you failed to capture the actual contribution. We understand that the technical complexity of the paper may have contributed to this impression. Here are some clarifications to address your confusion:

1. what the form of the actual inputs to the system are, and how a user would interact with it
 
Answer: Firstly, Figure 1 on the first page of this paper illustrates the distinction between our DSL program synthesis system and traditional DSL program synthesis systems. It clearly marks the input and output in the figure. Specifically, the input to a DSL program synthesis system is a partial program (a program with non-terminal elements), while the output is a complete program (a program without non-terminal elements). This distinction is also detailed in Section 2.1 on page 4. Additionally, in the experiments section, the tasks utilized are explicitly defined, with the actual code provided in Appendices J through N.

2. how the components of the system are constructed (i.e., what is "DSNN" - is it a feed-forward network? An LLM? What shape does its input take? What is its actual output? Similarly, THD, SSPH, SGH are only explained at the highest possible level)

Answer: Including abstract methods in the main body of the paper and specific implementations in the appendix is a common approach, especially when dealing with complex content. This paper employs a spatiotemporal graph attention network. Detailed information about the specific network structure, inputs and outputs, and the dimensions of each layer is provided in Appendix C (NEURAL NETWORKS IN DSNN) through accompanying figures and tables.  The program format of the direct processing of the neural network is shown in Appendix O (PARTIAL PROGRAM AS NEURAL NETWORK INPUT)

3 what ideas can transfer to other contexts?

Answer: Given the confusion regarding the main research content of this paper, I am uncertain about what you mean by "idea" and "other context." The core research content of this paper centers on DSL program synthesis, which involves transforming partial programs with non-terminal symbols into complete programs consisting solely of terminal symbols through the application of rules. The key improvement lies in enhancing the efficiency of rule application.
We introduce the CoL (Chain-of-Logic) approach to decompose the rule application stage and provide heuristic guidance. Additionally, we enhance the adaptability of CoL through Neural Network Feedback Control (NNFC). Our method's capability in handling complex DSL program synthesis tasks—including increasing problem difficulty and executing multi-DSL collaboration tasks—is confirmed through a series of experiments. 


Although the paper is complex, this is necessary for a thorough discussion. I encourage you to spend more time reviewing it and ask any questions you may have. I am confident in its innovation and contribution, and I believe I can help you recognize the significant advancements in DSL program synthesis technology presented in this paper. 



-------------------------
For more about DSL program synthesis: 
1 ICLR papers closely related to this paper: 
Neural Guided Constraint Logic Programming for Program Synthesis,
ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis
Anonymous link to the project: coologic.org
Clarifying Confusion and hoping you can review it again.
Yes
No ethics review needed.
n/a
n/a
The submission discusses a technique for synthesis of logical rules from data.
The paper fails at presenting the contribution, and I am not able to provide a review of the actual contribution due to the presentation.

After reading the paper, I do not know
* what the form of the actual inputs to the system are, and how a user would interact with it
* how the components of the system are constructed (i.e., what is "DSNN" - is it a feed-forward network? An LLM? What shape does its input take? What is its actual output? Similarly, THD, SSPH, SGH are only explained at the highest possible level)
* what ideas can transfer to other contexts

The only way forward I see is to rewrite the entire submission. I would recommend to ask colleagues who are not closely familiar with this project to review draft submissions, and provide feedback on whether they understand what you are trying to present, before submitting the paper again.
Yes
No ethics review needed.
See weaknesses section for questions
The paper is very thorough and complex. There is a lot of detail and technical content. I have never seen a control system like that used for the neural network feedback control and it seems like a very novel way of synthesizing programs. While I don't fully understand it, the authors analysis of inner coupling as a key factor to the improved synthesis quality of COOL seems interesting.
This paper describes a form of program synthesis called COOL and applies it to the CLUTTR dataset (logical family relation problems embedded in natural language) and a toy quadratic equation dataset. COOL is a combination of a "Chain of Logic" (CoL) DSL and a neural network feedback control system (NFCC) for guiding search. CoL is a framework for sketching templates of useful DSL rule applications to guide the problem solving process. For example, for the CLUTTR dataset, the CoL DSL contains rules "Separate relations and genders", "Reason inverse relations", "Reason Indirect Relations", "Recombine relations and genders, eliminate irrelevant info", while the CoL DSL for quadratic equations contains rules such as 0 + a = a, (a + b) ^ 2 = a^2 + 2ab + b^2. NFCC is a complicated system that uses neural networks to guide and control the synthesis process. Program search is conducted with A* using heuristics I believe from the neural network. The authors show that COOL achieves 100% accuracy, uses fewer tree operations, and spends much less time solving compared to the baseline of a normal DSL without NNFC.
I find this paper very hard to understand.

The method is very complicated, with a lot of moving parts. The method is never described end-to-end in a coherent way — instead, bits of detail are mentioned throughout the paper, but I never get the sense that I fully understand what is going on. Some questions that I cannot answer from reading the paper: how are the neural networks trained? how are the heuristics computed? what is the input and output for a given synthesis problem? what are tree operations? One of the datasets, the quadratic equation dataset, is only described at a high level — I don't even know what the problems look like exactly. 

I would recommend writing the paper in a more linear order. Even if there is not space in the main text, the appendix can contain a full description of the (1) problem input and output, (2) overview of algorithm that transforms input to output prediction, (3) description of how your techniques plug in to the algorithm at different parts. You can also include more detailed definitions for things like tree operations, the DSL rule definitions, etc.

Overall, I am baffled at this paper. I am not foreign to program synthesis, yet most of the content of this paper is beyond me. 

To summarize, the technique is extremely complicated and not adequately explained. This is the main weakness of the paper. 

Beyond this, the evaluation is not very convincing. This is in large part due to the poor explanation — as stated earlier, I don't have a full understanding of what the input and output for each problem is, or even how search is conducted. So it's hard to know what to make of the accuracy metrics or time spent. "tree operation" and "transformation pair" are not defined either.
Yes
No ethics review needed.
1. The process by which DSNNs generate error signals from partial programs is not clearly explained. What constitutes a “correct” path in the training process? Is there a universal ground truth, or does it vary by domain? Also how much data does each DSNN require to perform effectively?
2. Did the dynamic experiments include any resource constraints (e.g., time or computational power limits)?
3. I did not understand what “multidomain tasks” are and what is their role in the experiments. Can you clarify please?
1. The ideas introduced in the paper are interesting. The Chain of Logic structure mirrors human problem-solving steps, allowing for more intuitive management of the program synthesis process within the DSL. This hierarchical structuring likely enhances interpretability.
2. The paper provides some experimental validation, with ablation studies showing the contribution of each component.
The paper introduces a neuro-symbolic program synthesis method that combines rule-based logic with neural feedback, termed Chain-Oriented Objective Logic (COOL). This technique allows fine-grained control over search processes within DSL-induced program spaces. Modularity is achieved through a component called Chain of Logic (CoL), inspired by the activity diagram model from software engineering and system control. CoL structures complex rule applications by assigning heuristic values to rules, enabling more efficient search strategies. Selecting specific rules based on heuristic values at particular stages (e.g., returning, jumping, or terminating synthesis branches) enhances search efficiency by aligning rule applications with promising paths, akin to human problem-solving approaches.

The paper also introduces Neural Network Feedback Control (NNFC), which uses multiple neural networks in series with specific heads for task detection, search pruning, and guidance. This prioritizes rules and tasks, allowing the synthesis process to bypass infeasible paths early, thereby improving the quality of synthesized outputs.

Experiments were conducted on static and dynamic benchmarks, categorized by difficulty level. The results validate both the general applicability of the approach and impressive improvements in accuracy and search efficiency under different task demands. A detailed ablation study clarifies the contribution of each component.
1. The CoL framework is highly domain-specific, and it is unclear how adaptable it would be for tasks outside its designed domain. The reliance on activity diagrams suggests that applying CoL in other fields may require substantial modification, limiting generalizability.
2. The description of NNFC is somewhat opaque, especially regarding how incorrect neural network predictions are “suppressed through filtering.”
3. There is insufficient detail about the design choices behind the Domain-Specific Neural Networks (DSNNs), such as how they are trained and what training data or ground truth is used, especially when handling partial programs.
Yes
No ethics review needed.
See Weaknesses section for a series of questions regarding this system.
The results seem very impressive, showing both an improvement in performance and a reduction in computational cost.
I do not feel I have a sufficient understanding of this paper to summarize it in detail. See the Weaknesses section for details on this. My best attempt to summarize this paper follows.

COOL is a system for neurosymbolic program synthesis, in which programs are generated incrementally in a DSL annotated with specific kinds of assertions and guides that allow for specific guidance of the neural network that drives the search. This leads to significant pruning, improving accuracy while reducing the size of the search space.
This paper provides insufficient details and precision for a reader to understand the nature of the algorithm. The following questions remain unanswered by the introduction, figures, and methods section: What is the artifact a user of this system needs to provide? The baseline is to provide a DSL, what additional information needs to be provided to use COOL?

The details on the amount of effort a user needs to complete to make a DSL in COOL are crucial because without this information, it is unclear how much of the benefits shown in the results section are downstream of the user providing more scaffolding and thus making the task easier.

The figures do not help clarify this to a reader as it is not clear whether they represent the user-provided activity diagram, an execution trace for an activity diagram, or a schematic explanation of how the algorithm functions. It is also unclear what heuristic vectors are, and when logic jumps / abort commands should be used, or even if these are being written by a programmer or the output of some neural network.

Additionally, it is never specified how the DSNN network is trained. How is the data gathered for the DSNN? Is it trained via supervised learning, some kind of wake/sleep loop, or reinforcement learning? What does it mean for the DSNN to be pretrained?
A novel method for complex program synthesis
@inproceedings{
anonymous2024cool,
title={{COOL}: Efficient and Reliable Chain-Oriented Objective Logic with Neural Networks Feedback Control for Program Synthesis},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Pjkes5MdKI},
note={under review}
}
Program synthesis methods, whether formal or neural-based, lack fine-grained control and flexible modularity, which limits their adaptation to complex software development. These limitations stem from rigid Domain-Specific Language (DSL) frameworks and neural network incorrect predictions. To this end, we propose the \textbf{Chain of Logic (CoL)}, which organizes synthesis stages into a chain and provides precise heuristic control to guide the synthesis process. Furthermore, by integrating neural networks with libraries and introducing a \textbf{Neural Network Feedback Control (NNFC)} mechanism, our approach modularizes synthesis and mitigates the impact of neural network mispredictions. Experiments on relational and symbolic synthesis tasks show that CoL significantly enhances the efficiency and reliability of DSL program synthesis across multiple metrics. Specifically, CoL improves accuracy by 70\% while reducing tree operations by 91\% and time by 95\%. Additionally, NNFC further boosts accuracy by 6\%, with a 64\% reduction in tree operations under challenging conditions such as insufficient training
data, increased difficulty, and multidomain synthesis. These improvements confirm COOL as a highly efficient and reliable program synthesis framework.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
program synthesis Chain-of-Logic neural network feedback control
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3a5ad19de866905f2a483508cc0dadf0ed651de1.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/6f183fd5b2b70966616fa0b521941ec0ae63e82e.zip
COOL: Efficient and Reliable Chain-Oriented Objective Logic with Neural Networks Feedback Control for Program Synthesis
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission