Closed-Loop Policies for Operational Tests of Safety-Critical Systems
  Manufacturers of safety-critical systems must make the case that their
product is sufficiently safe for public deployment. Much of this case often
relies upon critical event outcomes from real-world testing, requiring
manufacturers to be strategic about how they allocate testing resources in
order to maximize their chances of demonstrating system safety. This work
frames the partially observable and belief-dependent problem of test scheduling
as a Markov decision process, which can be solved efficiently to yield
closed-loop manufacturer testing policies. By solving for policies over a wide
range of problem formulations, we are able to provide high-level guidance for
manufacturers and regulators on issues relating to the testing of
safety-critical systems. This guidance spans an array of topics, including
circumstances under which manufacturers should continue testing despite
observed incidents, when manufacturers should test aggressively, and when
regulators should increase or reduce the real-world testing requirements for an
autonomous vehicle.
