Unable to replicate Figure 2.1 from "Reinforcement Learning: An Introduction"
You are calculating the average reward for each action (i.e. bandit arm) incorrectly. You cannot calculate this simply with a list comprehension, and you need to keep a second list storing the number of times each action was taken.

The correct calculation is to divide the total reward obtained from each action by the number of times that action was taken. You are dividing all actions by the total number of actions taken. Adding:

...
counts = [0 for i in range(10)]
...
else:
    averages = []
    for i in range(0, 10):
        averages.append(Q[i]/counts[i] if counts[i] &gt; 0 else 0)
    ...
...
counts[action] += 1


results in working code, and I can generate this graph using 200 samples per method, which looks like a noisier version of the one in the figure you referenced: