BlendRL: A Framework for Merging Symbolic and Neural Policy Learning
Neuro-Symbolic AI;Differentiable Reasoning;Reinforcement Learning;Interpretable AI;First-order logic
Yes
No ethics review needed.
see weakness
1. The paper is clearly written and easy to follow.
2. The utilization of the language model shows a certain level of innovation.
This paper integrates condition-based logic decisions with neural network-based reinforcement learning policies through an LLM-based hybrid module to address the shortcomings of both approaches. It achieved better results in three Atari games compared to standard PPO and logic-based method.
1. The overall concept is not particularly novel, with numerous similar works, such as the well-known fast and slow systems, already existing.
2. The paper and appendix lack crucial details on how the LLM generates rules and calculates hybrid probabilities, and to what extent this is based on the content provided in the prompts. This is essential for determining whether the method can generalize to more diverse tasks.
3. The paper consistently emphasizes complex tasks, yet the experimental environment is not particularly sophisticated. Atari is a basic, outdated, and relatively simple benchmark that has largely been mastered. The three selected tasks are not among the recognized challenging ones in Atari (indeed, they are relatively simple). Truly difficult tasks, such as Montezuma’s Revenge, would be more appropriate as experimental environments.
4. The experiments should compare against more advanced RL algorithms. For example, Agent57 has already achieved a score of 30k on Kangaroo (while blend RL scores less than 20k) and 1000k on Sequest (while blend RL scores less than 5k). Therefore, the current experimental results do not demonstrate the superiority of the method.
5. Overall, the experiments lean towards simpler explorations and lack ablation studies that would reveal the characteristics of the method. For instance, I couldn't find which LLM was used, nor were there detailed experiments on the impact of object-centric representations on the method.
Yes
No ethics review needed.
See weaknesses above
The paper is very well written and easy to follow. The proposed framework is quite novel to the best of my knowdledge, since the level 2 and 1 systems are usually much more separated than in BlendRL, and the results against vanilla PPO or a symbolic approach (NUDGE) are promising.
The paper presents BlendRL, a neuro-symbolic RL framework that allows agents to blend resoning from pixel and object level representations. Unlike previous  neuro-symbolic RL approach BlendRL learns concurrently it's low and haigh level policies. Specifically, a blending module, informed by rules generated by a language model, dynamically determines the optimal mix of neural and symbolic policies based on the task context. 

Authors test BlendRL in Seaquest and Kangaroo (from Atari), where agents must alternate between quick responses and logical planning, results show that BlendRL consistently outperforms purely neural or symbolic agents.
My biggest concern with this work is the lack of empirical comparison with any other neuro-symbolic baselines, e.g. [1-4] 

Since BlendRL also weights heavily on object-based representations and relational learning it would have been good (although I don't consider this critical)  to include some contrast with deep learning approaches for such kind of learning, e.g. [1, 5-6].

While the point above will raise my confidence on the paper, I still thinkt hat the novelty of the approach and the results included outweight the weakpoints.

[1] Borja G. León, Murray Shanahan, and Francesco Belardinelli. In a nutshell, the human asked for this: Latent
goals for following temporal specifications. In International Conference on Learning Representations, 2022.

[2] Kuo, Yen-Ling, Boris Katz, and Andrei Barbu. "Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.

[3] Vaezipoor, Pashootan, et al. "Ltl2action: Generalizing ltl instructions for multi-task rl." International Conference on Machine Learning. PMLR, 2021.

[4] Qiu, Wenjie, Wensen Mao, and He Zhu. "Instructing goal-conditioned reinforcement learning agents with temporal logic objectives." Advances in Neural Information Processing Systems 36 (2024).

[5] Shanahan, Murray, et al. "An explicitly relational neural network architecture." International Conference on Machine Learning. PMLR, 2020.

[6] Feng, Fan, and Sara Magliacane. "Learning dynamic attribute-factored world models for efficient multi-object reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).
Yes
No ethics review needed.
1) If I wanted to apply it to a new game, what should be done to generate the action rules (Section 3.3)? 

2) Does using the NUDGE policies to generate examples for the LLM provide a strong inductive bias to BlendRL?

3) How were the hyperparameters obtained?

4) How did you choose the three Atari games? Why did you use different ones compared to those used in NUDGE?
The idea of learning a mixture of neural policy and symbolic policy is novel. Given the presented experimental results, it seems to lead to nicely interpretable decision rules (at least in the symbolic part). Also, surprisingly to me, it seems that it automatically learns to make "reflex" decisions preferably using the neural policy. 

The overall proposition nicely integrates several previous techniques to offer an end-to-end method, with little human inputs.
The paper proposed to jointly learn a mixture of neural policy and symbolic policy (represented in a differentiable way) via reinforcement learning. In the experiments, the neural policy is simply a CNN-based policy, the differentiable symbolic policy is based on a method called differentiable forward reasoner, and the weights of the mixture can themselves be seen as the output of a trainable policy. The proposed method is tested on three Atari games.
The paper should be more self-contained. For instance, this work builds on several existing propositions (notably by Shindo et al. and Delfosse et al.). The authors should recall and provide more details of those previous works to help the reader more easily understand BlendRL and appreciate its novelty. For instance, I think the current presentation of differentable forward reasoner is too light, e.g., how is the set \mathcal C related to the forward reasoning graph? What's the advantages/disadvantages of this representation using a graph compared to other frameworks, e.g., dILP?
Also, Section 3.3,  which seems to me to be a key component that can have a huge impact on BlendRL, is lacking in details. For instance, I suggest the authors to provide more information about step (i). In addition, doesn't using the NUDGE policies to generate examples for the LLM provide a strong inductive bias to BlendRL?

The experimental validation is missing some details and maybe a bit light. The authors should explain how they chose their hyperparameters (e.g., Entropy coefficient for blending regularization). Since BlendRL requires a more complex policy model than Neural PPO or NUDGE), I believe it's possible via (potentially costly) hyperparameter tuning to obtain better results for BlendRL.
The authors only evaluates on three Atari games, which are moreover different from those used in NUDGE. Why is that so? Would BlendRL still outperform NUDGE on the tasks where it has demonstrated a good performance?

Although I haven't followed closely the latest developments in neuro-symbolic approaches, I believe that the literature in this direction is quite rich. For instance, I think that there are other works trying to combine System 1 and System 2 capabilities in AI agents and there are other neuro-symbolic RL approaches. The paper could be improved by better situating BlendRL in the existing literature. 

Minor points:
There are a number of typos that should be corrected, e.g.,
- line 107: I believe that function symbols are not used in this work. This should be clarified.
- line 162: What's called "state" actually corresponds to "observation"
- line 177: Regarding the dimension of the logic policy, should F appear?
- lines 852-855: the learning rates are missing?
Yes
No ethics review needed.
1. Figure 6 references textual interpretation; could you kindly provide a more detailed explanation?
2. In cases where the logic chain becomes quite long, would the policy still be considered explainable?
The paper is clearly written, and the proposed framework is compelling in its attempt to bridge the gap between symbolic reasoning and neural policy learning. The dual use of symbolic logic for high-level reasoning and deep neural networks for low-level reaction is well-motivated and timely. The experiments convincingly show that BlendRL agents perform better than existing methods, particularly in environments that demand both reasoning and quick reflexes. For instance, in Seaquest, the neural policy was able to effectively manage shooting enemies, while the symbolic component handled pathfinding and resource collection, demonstrating how the hybrid model synergistically enhances performance.
The paper presents BlendRL, a neuro-symbolic reinforcement learning framework that integrates symbolic reasoning and neural policies, aiming to improve agent performance by mixing neural and symbolic policies. The authors demonstrate BlendRL’s efficacy in classic Atari games, highlighting that the framework outperforms purely neural baselines and neuro-symbolic baselines like NUDGE.
Overall, I think this work will have a positive impact on the community, but I still have some concerns:
1. The paper evaluates the proposed method in a limited set of experimental environments. It would strengthen the validation of the method if the authors could evaluate BlendRL in a wider variety of environments, especially those with different characteristics (such as the environments BeamRiderNoFrameskip-v4, EnduroNoFrameskip-v4, and QbertNoFrameskip-v4).
2. While the paper demonstrates strong performance, it would be more impactful if the method were tested in more realistic environments such as IsaacGym[1] or MetaDrive[2]. These environments pose more challenging and realistic scenarios, which could further validate the generalizability and robustness of BlendRL. It would also be valuable for the authors to discuss any challenges or adjustments required to apply BlendRL to these more complex environments, and to comment on how they expect its performance and benefits to scale in such domains.
3. The explanation of how symbolic policies contribute to the overall behavior of mixed policies could benefit from more clarity. It would be helpful if the authors provided a detailed analysis or specific examples showing how the symbolic explanations correspond to the mixed policy's actions. Including case studies that illustrate how explanations evolve as the blending between neural and symbolic components changes could enhance understanding of the interpretability and functionality of the method.

[1] Makoviychuk, Viktor, et al. "Isaac gym: High performance gpu-based physics simulation for robot learning." arXiv preprint arXiv:2108.10470 (2021).

[2] Li, Quanyi, et al. "Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning." IEEE transactions on pattern analysis and machine intelligence 45.3 (2022): 3461-3475.
We propose a framework that jointly learns symbolic and neural policies for reinforcement learning.
@inproceedings{
anonymous2024blendrl,
title={Blend{RL}: A Framework for Merging Symbolic and Neural Policy Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=60i0ksMAhd},
note={under review}
}
Humans can leverage both symbolic reasoning and intuitive responses. In contrast, reinforcement learning policies are typically encoded in either opaque systems like neural networks or symbolic systems that rely on predefined symbols and rules. This disjointed approach severely limits the agents’ capabilities, as they often lack either the flexible low-level reaction characteristic of neural agents or the interpretable reasoning of symbolic agents. 

To overcome this challenge, we introduce *BlendRL*, a neuro-symbolic RL framework that harmoniously integrates both paradigms. 
We empirically demonstrate that BlendRL agents outperform both neural and symbolic baselines in standard Atari environments, and showcase their robustness to environmental changes. Additionally, we analyze the interaction between neural and symbolic policies, illustrating how their hybrid use helps agents overcome each other's limitations.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Neuro-Symbolic AI Differentiable Reasoning Reinforcement Learning Interpretable AI First-order logic
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/398ab09626ff303b17655f007b4eaa24f0b62474.pdf
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/5c46c8353e8307a28cd707a8d93d9c705ad7e30b.zip
BlendRL: A Framework for Merging Symbolic and Neural Policy Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission