Does an RL agent learn during exploitation?
It depends on the game the agent is playing. If there are rewards all over the environment, the agent learns only when the coefficient of exploration is greater than zero. That is, if you are only allowing it to exploit, the agent may aswell just exploit the first reward it meets which ends a game. 

In this case, it will find the first reward that ends the game, and will not change its algorithm (will not learn any other way of playing). On the other hand, if you allow it to explore, it may eventually find another better strategy (learn). 

It is always good to conciliate the ratio of exploration and exploitation. It should always be capable of exploring, even if the coefficient is low. That is the whole advantage of Reinforcement Learning.