Title: On Testing Machine Learning Programs,
Abstract: Nowadays, we are witnessing a wide adoption of Machine learning (ML) models
in many safety-critical systems, thanks to recent breakthroughs in deep
learning and reinforcement learning. Many people are now interacting with
systems based on ML every day, e.g., voice recognition systems used by virtual
personal assistants like Amazon Alexa or Google Home. As the field of ML
continues to grow, we are likely to witness transformative advances in a wide
range of areas, from finance, energy, to health and transportation. Given this
growing importance of ML-based systems in our daily life, it is becoming
utterly important to ensure their reliability. Recently, software researchers
have started adapting concepts from the software testing domain (e.g., code
coverage, mutation testing, or property-based testing) to help ML engineers
detect and correct faults in ML programs. This paper reviews current existing
testing practices for ML programs. First, we identify and explain challenges
that should be addressed when testing ML programs. Next, we report existing
solutions found in the literature for testing ML programs. Finally, we identify
gaps in the literature related to the testing of ML programs and make
recommendations of future research directions for the scientific community. We
hope that this comprehensive review of software testing practices will help ML
engineers identify the right approach to improve the reliability of their
ML-based systems. We also hope that the research community will act on our
proposed research directions to advance the state of the art of testing for ML
programs.