Brain Bandit: A Biologically Grounded Neural Network for Efficient Control of Exploration
explore-exploit;stochastic Hopfield network;Thompson sampling;decision under uncertainty;brain-inspired algorithm;reinforcement learning
Yes
No ethics review needed.
The pseudo code for the MDP tasks includes P_{sas'}. Does this mean that it is assumed that the agent has access to the true dynamics of the environment?
This paper extends the concept of Boltzmann machine to continuous valued recurrent neural networks. I could not follow all the details of the mathematics, but the analysis seems to be new and helpful.
The simulation results are compared with human behavioral data.
This paper presents a simple recurrent neural network model of exploratory action selection.
The implementation of bandit assumes the availability of the memory of the all past experience in the memory buffer, which may be impractical as a cognitive model.
Yes
No ethics review needed.
# Suggestions for improvement:
* L169 -- L174: What is MFPT? The lead up to these equations needs better development, esp. for someone not already familiar with this literature
* L202/208: Define escape efficiency
* L322/323: Define relative and total uncertainty
* L438 and elsewhere: Explain the rationale behind the choice of T steps for BBN simulation.
* Citation format seems wrong
# Strengths
* Novelty: As far as I can tell, this is a novel combination of Stochastic Hopfield Networks applied to decision making / explore-exploit tasks. The connections with neuroscience are interesting.
* Well written/Thorough: Paper is quite clear and well written, and as such has high pedagogical value. The authors present extensive experiments comparing BBN against baseline methods on multiple tasks, providing empirical support for their claims. (Minor concerns addressed below)
# Summary
The authors present a biologically-inspired neural network model called Brain Bandit Network (BBN) that addresses the explore-exploit problem in sequential decision making tasks. 
The model itself is a stochastic continuous Hopfield network, which they show can implement a form of Bayesian posterior sampling and a tunable input uncertainty bias. 
They demonstrate the BBN's effectiveness in MAB tasks, showing that it approximates human and animal choice patterns, and on some MDP exploration benchmarks.
# Weaknesses
* Niche/limited applicability: The method itself is quite niche, needs an SDE solver, and has several hyperparameters that require niche/expert knowledge to tune. 
* Tuning the numerous hyperparameters requires specialized knowledge, making it difficult for non-experts to utilize effectively. Are there principled ways to determine optimal BBN parameters for new environments? 
* The performance of this model in high-dimensions needs more exposition. Section 3.4 provides a start, but more extensive experiments and theoretical analysis are needed to understand the behavior and performance of BBN in high-dimensional settings. (ideally supported by some theoretical footing).
Yes
No ethics review needed.
How were the states and Q-values parameterized in the task? Please provide a Methods-style description of the model. This may be particularly important in the light of the next question.

One thing that I couldn’t understand is: What makes the model explore more than the baselines? In particular, how would the exploration in this task compare to the random walk? How would that result change based on the parameterization of the stare and the Q-function (e.g., tabular vs. deep learning)?

Finally, I didn’t quite understand why is the higher state coverage considered a positive thing in this task. Specifically, as the task is not rewarded (Figure 8 b-c), the exploration seems unnecessary. When rewards are present, their utility may be different depending on the reward and the task.

Overall, I think this is an interesting paper relevant to the ICLR community. I believe that it offers sufficient novelty and significance through the remarkable match between the model’s predictions and human behavior. Text-related issues can be addressed during the remaining time.
The work deals with an interesting and important problem of the exploration-exploitation tradeoff, particularly focusing on the aspect of human decision-making. As human decision-making, shaped by evolution, is expected to be optimal, at least in certain ways, understanding its mechanics may be useful for the design of future ML algorithms. This work combines the study of exploration-exploitation tradeoff with recurrent neural networks and bandit tasks, which are all relevant and interesting to the ICLR community.

The quality of the research in this work is high: it starts with the thorough derivations of the analytic predictions for the system’s behavior and then continues with numerical experiments where the model’s predictions are first validated and then comparisons are performed with human and animal data. The text is mostly well-written, following the standard structure of machine-learning papers and introducing concepts in a logical sequence, making it easy to follow.

The result regarding the similarity between the model’s predictions and human/mouse data on exploration/exploitation in multi-armed bandit tasks is novel and interesting. It is particularly strong as it spans multiple datasets from multiple works.
In this work, the Authors propose a framework addressing the exploration-exploitation tradeoff using a stochastic continuous Hopfield network. They provide a derivation to show that the proposed framework implements a form of Bayesian inference while also displaying an uncertainty-related bias. The comparison of the proposed model with human and mouse data on multi-armed bandit task shows the similarity between the two; additional considerations and experiments are provided for two MDP tasks.
While the derivations for the analytic results are thorough and lengthy, most of them appear to be either existing or intuitive. The attractor structure of the Hopfield network with two neurons and inhibitory connections is known. Hopfield networks have been shown to be able to implement Bayesian inference, although the derivations have been provided for the discrete case. The fact that the probability of the network’s activity to stay in an attractor depends on the relative orientation of its Hessian matrix and the noise direction is clear and intuitive. Of course, none of that diminishes the novelty of the result regarding the similarity between the model’s behavior and human decision-making.

The MDP part may need a fuller description (ideally a Methods section; e.g. in the Appendix) to clarify what gives rise to the observed phenomena. The uncertainty Bellman equation is also worth mentioning in the text for the clarity. Please see the specific questions regarding the Gridworld task below.
We demonstrate that a brain-inspired stochastic Hopfield network can achieve efficient, human-like, uncertainty-aware exploration in bandit and MDP tasks.
@inproceedings{
anonymous2024brain,
title={Brain Bandit: A Biologically Grounded Neural Network for Efficient Control of Exploration},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=RWJX5F5I9g},
note={under review}
}
How to balance between exploration and exploitation in an uncertain environment is a central challenge in reinforcement learning. In contrast, humans and animals have demonstrated superior exploration efficiency in novel conditions. To understand how the brain’s neural network controls exploration under uncertainty, we analyzed the dynamical systems model of a biological neural network that controls explore-exploit decisions during foraging. Mathematically, this type of network (which is named the Brain Bandit Net, or BBN) is a special type of stochastic continuous Hopfield networks. We show through theory and simulation that BBN can perform posterior sampling of action values with a tunable bias towards or against uncertain options. We then demonstrate that, in multi-armed bandit (MAB) tasks, BBN can generate probabilistic choice behavior with an uncertainty bias in a way that resembles human and animal choice patterns. In addition to its high efficiency in MAB tasks, BBN can also be embedded with reinforcement learning algorithms to accelerate learning in MDP tasks. Altogether, our findings reveal the theoretical basis for efficient exploration in biological neural networks and proposes a general, brain-inspired algorithmic architecture for efficient exploration in RL.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
explore-exploit stochastic Hopfield network Thompson sampling decision under uncertainty brain-inspired algorithm reinforcement learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/899bee251c1c1b5eef52e0637090a7ea6f3102d1.pdf
applications to neuroscience & cognitive science
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Brain Bandit: A Biologically Grounded Neural Network for Efficient Control of Exploration
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission