Regret measure in continuous time limit for a stochastic Multi-armed bandit problem
Stochastic multi-armed bandit;Risk-sensitive regret;Hamilton-Jacobi-Bellman equation;Continuous time-limit
Yes
No ethics review needed.
Please refer to the weaknesses.
I am unable to provide a comprehensive scientific review of the paper, and thus I cannot identify specific strengths. Please refer to the weaknesses  below.
This paper aims to analyze multi-armed bandit problems using differential equations and introduces a new risk measure for the analysis.
The paper has significant issues with presentation. Not only are there numerous grammatical errors, typos, and punctuation mistakes, but many sentences are incomplete and seem disconnected from the surrounding context. Additionally, the writing lacks a clear logical flow, making it difficult to follow the argument.

Furthermore, it appears that the authors have not adhered to the official ICLR style guidelines.

Due to these issues, I am unable to provide a more detailed review.
Yes
No ethics review needed.
1. what is the main contribution of the paper?

2. what is exactly the problem studied?

3. why studying the continuous-time limit is relevant for bandit problems?

4. How should we interpret the main result Theorem 1 and understand its practical relevance?
Considering continuous-time limit of regret measures in continuous time.
This paper studies a class of stochastic multi-armed bandit problems with a risk-sensitive regret measure within a continuous limit setting
The presentation is not clear. 

The paper's contribution and the significance of the problem are not clearly articulated in the Introduction and the main text. 

The English in the paper could benefit from some further refinement or editing to enhance clarity and coherence.
Yes
No ethics review needed.
See Weaknesses.
The way to convert the MAB problem to a PDE problem is interesting and meaningful. The work compares different concepts, like frequentist and Bayesian settings making it easy to understand the applicability of the method.
This paper considers the traditional multi-armed bandit problem with a new risk measure. The authors continuize the time through rescaling and use PDE to find the optimal policy. In the meantime, the authors use some simulations to verify their results.
1. The writing needs to be improved. There are a lot of typos which make it hard to understand the paper. 

2. There are no real-world applications provided by the author regarding why this new risk measure is important, reducing the credibility and impact of the paper.

3. The usage of MDP seems improper. In your setting, $\nu$ seems to be fixed and only $s$ and $q$ are changing. However, there is no need to learn the transition kernel as if you choose an action $a$, corresponding $q$ will be increased by 1. Then, it reduces to learning the reward function which is the same as in traditional MAB literature and so people usually don't call it MDP. It's more reasonable to use your framework to consider the case that $\nu$ is varying and say it's MDP.

4. The notations are messy. For example, why $V_{i+1}$ only relies on $R_i$? And you use a very strong assumption but only hide it in the Lemma 1.

5. The Theorem 1 is unclear. What is zero? Why do you use a bracket but link it to nothing?

6. In your numerical study, how do you implement UCB and TS? Do you adjust their definitions of regrets to your new risk measure? If not, they are not comparable. Otherwise, it's better to mention how you set the baseline in detail.
@inproceedings{
anonymous2024regret,
title={Regret measure in continuous time limit for a stochastic Multi-armed bandit problem},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=4jzjexvjI7},
note={under review}
}
We study a class of stochastic multi-armed bandit problems with a risk-sensitive regret measure within a continuous limit setting. This problem is interesting when optimizing the expected reward is not the foremost objective, and the problem horizon is long. Through scaling the state parameters, including the number of pulls and cumulative reward for each arm we study the  bandit problem with infinite horizon, we delineate such risk using a Hamilton-Jacobi-Bellman equation with quadratic growth. Using this approach, we establish an explicit form of the optimal policy associated with the considered risk. As an application, we present examples where the results obtained in continuous time offer insights into the optimal policy for each case. Finally, numerical experiments confirm the theoretical results are presented.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Stochastic multi-armed bandit Risk-sensitive regret Hamilton-Jacobi-Bellman equation Continuous time-limit
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cf0559a93e3cce480271d887b4e3a5098d667b29.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Regret measure in continuous time limit for a stochastic Multi-armed bandit problem
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission