Time efficient way to implement Multi-Armed-Bandits?
Store the values in a priority queue.  Typically, in each iteration you will update the value for only a single arm, so you need to change the key of a single value in the priority queue, which can be done in $O(\log n)$ time, where $n$ is the number of arms.  You can also find the argmax in $O(\log n)$ time.