Look Before You Leap: Safe Model-Based Reinforcement Learning with Human
  Intervention
  Safety has become one of the main challenges of applying deep reinforcement
learning to real world systems. Currently, the incorporation of external
knowledge such as human oversight is the only means to prevent the agent from
visiting the catastrophic state. In this paper, we propose MBHI, a novel
framework for safe model-based reinforcement learning, which ensures safety in
the state-level and can effectively avoid both "local" and "non-local"
catastrophes. An ensemble of supervised learners are trained in MBHI to imitate
human blocking decisions. Similar to human decision-making process, MBHI will
roll out an imagined trajectory in the dynamics model before executing actions
to the environment, and estimate its safety. When the imagination encounters a
catastrophe, MBHI will block the current action and use an efficient MPC method
to output a safety policy. We evaluate our method on several safety tasks, and
the results show that MBHI achieved better performance in terms of sample
efficiency and number of catastrophes compared to the baselines.
