In summary, machine learning is a valuable and important tool to predict physicochemical properties, but can suffer from error sources uncommon to other fields of research. Increased attention should be paid to noise and bias from data coverage, model architecture and representation to identify and remedy shortcomings of chemistry-related deep learning models concerning their performance and uncertainty calibration. ACKNOWLEDGEMENT E.H. acknowledges support from the Austrian Science Fund (FWF), project J-4415. The authors acknowledge the Machine Learning for Pharmaceutical Discovery and Synthesis Consortium (MLPDS) for funding. Parts of the data reported within this paper were generated on the MIT SuperCloud Lincoln Laboratory Supercomputing Center. We also thank our colleagues Lagnajit Pattanaik and Kevin Greenman for helpful review and discussion of this study.