Is Offline Decision Making Possible with Only Few Samples? Reliable Decisions in Data-Starved Bandits via Trust Region Enhancement
Multi-armed bandit;high dimensional decision making;reinforcement learning.
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
We propose an algorithm that could learn a stochastic policy in offline Multi-armed bandit in the data-starved case, where there are only a few samples for each arm.
@misc{
zhang2024is,
title={Is Offline Decision Making Possible with Only Few Samples? Reliable Decisions in Data-Starved Bandits via Trust Region Enhancement},
author={Ruiqi Zhang and Yuexiang Zhai and Andrea Zanette},
year={2024},
url={https://openreview.net/forum?id=63pceN3fOg}
}
What can an agent learn in a stochastic Multi-Armed Bandit (MAB) problem from a dataset that contains just a single sample for each arm? Surprisingly, in this work, we demonstrate that even in such a data-starved setting it may still be possible to find a policy competitive with the optimal one. This paves the way to reliable decision-making in settings where critical decisions must be made by relying only on a handful of samples.

Our analysis reveals that \emph{stochastic policies can be substantially better} than deterministic ones for offline decision-making. Focusing on offline multi-armed bandits, we design an algorithm called Trust Region of Uncertainty for Stochastic policy enhancemenT (TRUST) which is quite different from the predominant value-based lower confidence bound approach. Its design is enabled by localization laws, critical radii, and relative pessimism. We prove that its sample complexity is comparable to that of LCB on minimax problems while being substantially lower on problems with very few samples.

Finally, we consider an application to offline reinforcement learning in the special case where the logging policies are known.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Ruiqi_Zhang2 ~Yuexiang_Zhai1 ~Andrea_Zanette1
Ruiqi Zhang Yuexiang Zhai Andrea Zanette
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multi-armed bandit high dimensional decision making reinforcement learning.
I certify that there is no acknowledgement section in this submission for double blind review.
zhang|is_offline_decision_making_possible_with_only_few_samples_reliable_decisions_in_datastarved_bandits_via_trust_region_enhancement
/pdf/ecf219901d399702fcff1911847a314fadcf5451.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/1aad0366c8830f7338f8e12e22c10bec0cc9828c.pdf
Is Offline Decision Making Possible with Only Few Samples? Reliable Decisions in Data-Starved Bandits via Trust Region Enhancement
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission