ViZDoom: DRQN with Prioritized Experience Replay, Double-Q Learning, &
  Snapshot Ensembling
  ViZDoom is a robust, first-person shooter reinforcement learning environment,
characterized by a significant degree of latent state information. In this
paper, double-Q learning and prioritized experience replay methods are tested
under a certain ViZDoom combat scenario using a competitive deep recurrent
Q-network (DRQN) architecture. In addition, an ensembling technique known as
snapshot ensembling is employed using a specific annealed learning rate to
observe differences in ensembling efficacy under these two methods. Annealed
learning rates are important in general to the training of deep neural network
models, as they shake up the status-quo and counter a model's tending towards
local optima. While both variants show performance exceeding those of built-in
AI agents of the game, the known stabilizing effects of double-Q learning are
illustrated, and priority experience replay is again validated in its
usefulness by showing immediate results early on in agent development, with the
caveat that value overestimation is accelerated in this case. In addition, some
unique behaviors are observed to develop for priority experience replay (PER)
and double-Q (DDQ) variants, and snapshot ensembling of both PER and DDQ proves
a valuable method for improving performance of the ViZDoom Marine.
