Why do we explore after we have an accurate estimate of the value function?
Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small probability for exploration?

It will depend on the goal of the work:

If the learning algorithm is off-policy (e.g. Q learning), it is normal to continue to explore at a moderate-to-low rate because it can accurately estimate an optimal deterministic target policy from a close-to-optimal stochastic behaviour policy.

Perhaps it is engineered with a low tolerance and will keep going even when you don't need it to.

Perhaps the code is for education and run so long that convergence is easily visible. Or for comparison with other methods which really do take that long to converge, and you would like data on the same axis.

For comparison with other methods for sample efficiency whilst learning and measuring regret (i.e. how much the exploration is costing you).

When environment is dynamic and could change, then continuous exploration is potentially useful to discover the changes, as you suggest in the question.


If you do really have an ideal agent, then of course you could just stop and say &quot;job done&quot;. In practice for more interesting problems, you won't usually get small state spaces and perfect solutions inside 2000 episodes (or ever) - as a result if you are reading tutorials in reinforcement learning, they may just skip this point.