Informed Exploration via Generative Modeling
bandit algorithms;Thompson sampling;bayesian inference;generative models
Yes
No ethics review needed.
* Could the authors clarify the differences between their approach and existing works, highlighting the novelty and contribution of this work?
* Could the authors clarify if the proposed methods are used to address the delayed feedback issue (missing outcomes) rather than used for building a recommendation policy?
* The paper presents a new method for Bayesian uncertainty quantification through autoregressive generative modeling, which provides a scalable and effective alternative to traditional methods that rely on complex posterior approximations.

* By applying this method to a news recommendation task, the paper demonstrates the approach's potential to improve real-world decision-making where uncertainty and user interactions play a crucial role.

* The paper provides a formal regret bound, showing that the approach’s decision-making performance directly relates to the sequence model’s training loss, solidifying the theoretical basis for the proposed method.
The paper explores uncertainty in neural networks for decision-making, focusing on a meta-learning bandit setup applied to news recommendation. It introduces a novel approach using autoregressive generative modeling to quantify Bayesian uncertainty by predicting future rewards based on historical data. This method involves pre-training a generative model to forecast potential rewards, which helps balance exploration and exploitation without directly modeling latent variables. The approach is demonstrated through theoretical bounds and experiments, showing it enhances decision-making accuracy by effectively integrating neural networks.
* A major concern is that this paper lacks a thorough discussion of related work, despite several highly relevant and overlapping studies. There is existing research that uses generative modeling for posterior approximations【1】and in recommendation tasks【2】. Additionally, multiple works have explored connections between autoregressive generative modeling and approximate Bayesian inference【3,4】. Furthermore, meta-learning with autoregressive generative networks has been studied in contexts such as classification【5】, Bayesian Optimization (BO)【6】, and decision-making in sequential prediction【7】.


1. Gal, Yarin, and Zoubin Ghahramani. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. PMLR, 2016.
2. Da Tsai, Yun, and Shou De Lin. "Fast online inference for nonlinear contextual bandit based on generative adversarial network." arXiv preprint arXiv:2202.08867 (2022).
3. Hollmann, Noah, et al. "Tabpfn: A transformer that solves small tabular classification problems in a second." arXiv preprint arXiv:2207.01848 (2022).
4. Müller, Samuel, et al. "Transformers Can Do Bayesian Inference." International Conference on Learning Representations.
5. Bonet, David, et al. "HyperFast: Instant Classification for Tabular Data." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024.
6. Müller, Samuel, et al. "Pfns4bo: In-context learning for bayesian optimization." International Conference on Machine Learning. PMLR, 2023.
7. Lee, Jonathan, et al. "Supervised pretraining can learn in-context reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).
Yes
No ethics review needed.
Please see the above Weaknesses section.
1) The idea of incorporating sequential models into decision-making makes sense to me. Pre-trained on previous data, the autoregressive model has the ability to predict the missing reward values, which is benefit for the downstream algorithm.

2) The authors provide several theory analyses to support the efficiency of the proposed modules.

3) Experiments on both synthetic and real-world settings show the improvements of the proposed model.
This paper aims to address the challenge of modeling uncertainty in decision-making in news recommendation systems. The authors propose a method that leverages autoregressive generative models to predict the missing or future rewards based on the historical, and then use the Thompson sampling for final decision-making with the previous and generated rewards. This paper proposes several theory analyses of the advantages of using sequential models (like transformers). The authors demonstrate their approach through both synthetic and semi-realistic news recommendation experiments, showing that their method can effectively incorporate pretrained language models and achieve strong performance in terms of regret minimization and uncertainty quantification.
1) One of the main concerns comes from the experiments. The authors only test their models with the base models (such as PS Neural Linear, PS Beta-Bernoulli), which may reduce the convincingness of the proposed method.

2) The authors claim that they use the language foundation model to improve the uncertainty quantification, which is not precise. Because they only pre-trained sequential models on historical data, the trained models are not actually foundation models.

3) The Thompson sampling at the inference stage relies heavily upon the generated missing reward values, which may limit its application when sequence models cannot precisely predict missing rewards.
Yes
No ethics review needed.
See weakness.
- This paper tackles the important issue of exploration in general neural networks, a challenge with broad implications for real-world applications like recommendation systems, education, and even LLM generation. Informed exploration are critical for these complex decision-making systems.
- The authors effectively bridge the gap between sequence modeling and Bayesian inference, introducing the concept of "generative Thompson sampling". This provides a strong theoretical justification for their approach and nicely clarifies the role of pre-training as a form of empirical Bayes.
- The proposed method is easy to implement, and the experimental results on recommendation tasks demonstrate the effectiveness of the proposed approach, in terms of both reducing regret and providing valid confidence intervals.
This paper studies exploration under general neural networks. They propose the method kind of like "generative Thompson Sampling", which does not explicitly model the unknown latent variables, but aims at modeling the missing rewards. The procedure of the algorithm starts with implicitly learns the Bayesian model via pre-training by minimizing a sequence prediction loss on successive reward using historical data, then at inference time, it uses the both the observed and imputed reward to make decisions. This basically reduces 
 the sequential decision-making problem to a sequence prediction training with low loss. Experiments are done on synthetic and news recommendation dataset.
- Lack of Clarity and Potential Inconsistencies: The paper would benefit from increased clarity in its methodological details and problem formulation. Specifically:

1.  While the motivating example of news recommendation emphasizes no overlap in the action set across time steps, this constraint seems not to be the case  in the formal problem definition and Algorithm 1.
2. The role of different users in Equation 2 and Algorithm 1 remains unclear. It appears to estimate a population-level mean reward for each action, over all users. This then seems a multi-arm bandit problem than the contextual bandit one? The authors should explicitly state whether they are addressing a multi-arm bandit or a contextual bandit problem and revise the relevant sections accordingly. It is hard to estimate the context-dependent reward based on the current alg.

- Computational Cost: The proposed method's computational complexity scales linearly with both the action set size and the "overlap" between new and historical actions, and time-step T. This could become prohibitive in applications with large action spaces or significant small overlap, even disjoint support. The authors should discuss more on the scaling of the method.

- Limited Novelty: The proposed approach is very similar to reward models [1] employed in the offline bandit and RL literature. While subtle differences may exist, the authors need to provide a more thorough comparison with existing reward modeling techniques, explicitly highlighting their novel contributions here.

Ref: https://arxiv.org/pdf/1103.4601
We use generative sequence modeling to implement a principled version of Thompson Sampling with neural networks in a meta-learning bandit setting; our approach forms approximate posterior draws by generating plausible sequences of future rewards.
@inproceedings{
anonymous2024informed,
title={Informed Exploration via Generative Modeling},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=JNhU9NeOFr},
note={under review}
}
Conventionally trained neural networks excel at prediction but often struggle to model uncertainty in their own predictions. We explore this challenge in a meta-learning bandit decision-making problem for news recommendations; this setting require decision-making algorithms to incorporate pretrained language models to process text data for the best performance. We present a scalable approach to Bayesian uncertainty quantification by posing it as a problem of autoregressive generative modeling of future rewards. First, we use historical data on previously released news articles to pre-train a generative model to predict sequences of future potential rewards. At inference time, our algorithm makes decisions based on limited previous rewards and autoregressively generated future rewards. Far from a heuristic, we synthesize insights from the literature to show our method is a novel implementation of Thompson (posterior) sampling, a prominent bandit algorithm. We prove our pretraining loss directly controls online decision-making performance, and we demonstrate our framework on a news recommendation task where we integrate end-to-end fine-tuning of a pretrained language model to process news article headline text to improve performance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
bandit algorithms Thompson sampling bayesian inference generative models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cb06135fb26af4e36b7e403eee7807fc3af1c23c.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/847cf65e9a6e3d2a0b4faaeb6ebaf676d05dace9.zip
Informed Exploration via Generative Modeling
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission