Multi or n-armed bandit reinforcement learning in R
There were quite a few things wrong with your implementation:


In both the if and else parts, choice ended up being a numeric (a non-integer, e.g. 0.6532) when it should be an integer. The problem was that a) you where using max instead of which.max inside bestArm and b) randomly picking a value from arms instead of its index.
Inside the else part, you were using k and rwd without setting them.
your computation of the running mean was not using weights as the python implementation does
This won't have a significant impact, but I think the python implementation had an issue: the observed mean rewards av should be initialized to zero, not one.


Here are other improvements I would suggest:


The if/else should be limited to the picking of the choice index. Everything else should be moved after the if/else as to not repeat code. This was a problem from the original python implementation you pointed to.
Instead of writing a script, you should write a function. It is a good way to isolate the inputs and outputs of your program, it gives you an easy way to test with different input parameters, to share your code, etc.
use spaces to make your code easier on the eyes. Spaces also avoid ambiguous situations for people who are not experts in R's syntax rules; for example, when someone sees a&lt;-b, is it assigning b to a, or is it asking if a is less than -b? Maybe another way to convince you is to make you notice that R's compiled code does contain spaces (run print(lapply) for example); it shows that R's authors agreed that code readability was more important than compactness.
If you are slowly building a vector (runMean) and know in advance its final length, initialize it with that exact length: runMean &lt;- rep(NA, iter). Otherwise, at each iteration R will have to create a longer object and this will significantly slow down your code for any large number of iterations.


So the code could look like this:

n.armed.bandit.sim &lt;- function(iter = 500, n = 10, eps = 0.1, max.reward = 100) {

  arms &lt;- runif(n)
  av &lt;- rep(0, n)
  counts &lt;- rep(0L, n)

  reward  &lt;- function(prob) sum(runif(max.reward) &lt; prob)
  bestArm &lt;- which.max

  runMean &lt;- rep(NA, iter)
  for (i in seq_len(iter)) {
    choice &lt;- if(runif(1) &gt; eps) bestArm(av) else sample(n, 1) 
    counts[choice] &lt;- counts[choice] + 1L
    k &lt;- counts[choice]
    rwd &lt;- reward(arms[choice])    
    old_avg &lt;- av[choice]
    new_avg &lt;- old_avg + (1 / k) * (rwd - old_avg)
    av[choice] &lt;- new_avg
    runMean[i] &lt;- weighted.mean(av, counts / sum(counts))
  }
  plot(runMean)
}