Q-Ensembles are a model-free approach where input images are fed into different Q-networks and exploration is driven by the assumption that uncertainty is proportional to the variance of the output Q-values obtained. They have been shown to perform relatively well compared to other exploration strategies. Further, model-based approaches, such as encoder-decoder models have been used successfully for next frame prediction given previous frames. This paper proposes to integrate the model-free Q-ensembles and model-based approaches with the hope of compounding the benefits of both and achieving superior exploration as a result. Results show that a model-based trajectory memory approach when combined with Q-ensembles produces superior performance when compared to only using Q-ensembles.
@cite_24: We show how an ensemble of @math -functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the @math -learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experiments show significant gains on the Atari benchmark.
This ensemble approach for uncertainty estimation in Neural Networks motivated its use for uncertainty estimation for exploration in the case of Deep Reinforcement Learning in the form of Q-Ensembles @cite_24 . Specifically, an ensemble voting algorithm is proposed where the agent takes action based on a majority vote over the Q-ensemble. The exploration strategy described uses the estimate of the confidence interval to then optimistically explore in the direction of the largest confidence interval (highest uncertainty). This approach was demonstrated to improve significantly over an Atari benchmark. The Q-Ensemble approach is an example of a model-free reinforcement learning approach where we do not need to infer the environment in the learning process. Model-free approaches are however generally high in sample complexity. Sampling from a learned model of the environment can help us mitigate this problem.