Title: Can Deep Clinical Models Handle Real-World Domain Shifts?,
Abstract: The hypothesis that computational models can be reliable enough to be adopted
in prognosis and patient care is revolutionizing healthcare. Deep learning, in
particular, has been a game changer in building predictive models, thereby
leading to community-wide data curation efforts. However, due to the inherent
variabilities in population characteristics and biological systems, these
models are often biased to the training datasets. This can be limiting when
models are deployed in new environments, particularly when there are systematic
domain shifts not known a priori. In this paper, we formalize these challenges
by emulating a large class of domain shifts that can occur in clinical
settings, and argue that evaluating the behavior of predictive models in light
of those shifts is an effective way of quantifying the reliability of clinical
models. More specifically, we develop an approach for building challenging
scenarios, based on analysis of \textit{disease landscapes}, and utilize
unsupervised domain adaptation to compensate for the domain shifts. Using the
openly available MIMIC-III EHR dataset for phenotyping, we generate a large
class of scenarios and evaluate the ability of deep clinical models in those
cases. For the first time, our work sheds light into data regimes where deep
clinical models can fail to generalize, due to significant changes in the
disease landscapes between the source and target landscapes. This study
emphasizes the need for sophisticated evaluation mechanisms driven by
real-world domain shifts to build effective AI solutions for healthcare.