Guided Exploration with Proximal Policy Optimization using a Single
  Demonstration
  Solving sparse reward tasks through exploration is one of the major
challenges in deep reinforcement learning, especially in three-dimensional,
partially-observable environments. Critically, the algorithm proposed in this
article uses a single human demonstration to solve hard-exploration problems.
We train an agent on a combination of demonstrations and own experience to
solve problems with variable initial conditions. We adapt this idea and
integrate it with the proximal policy optimization (PPO). The agent is able to
increase its performance and to tackle harder problems by replaying its own
past trajectories prioritizing them based on the obtained reward and the
maximum value of the trajectory. We compare different variations of this
algorithm to behavioral cloning on a set of hard-exploration tasks in the
Animal-AI Olympics environment. To the best of our knowledge, learning a task
in a three-dimensional environment with comparable difficulty has never been
considered before using only one human demonstration.
