Advances in computing architectures for microscope laboratories are one side of the coin, but integrated or hybrid machine learning based algorithms need to be deployed alongside to enable automatic analysis of large-scale data. Recent developments in machine-learning and in particular deep-learning approaches in electron microscopy hold great promise for laying the foundation for autonomous data-analysis and electron-microscope operation , . Ultimately, the aim is to enable the discovery of new material phenomena and to probe the physical properties of materials and their evolution with atomic precision. Since the physical nature of electron wave propagation and interaction in a crystalline material is well understood, ground truth training data for a deep learning model can be efficiently generated . However, a large deep-learning model would need to contain information not only of all known crystal structures and phases, but more importantly of different point, line, or planar defect types. Recognizing defects from supervised learning, however, is nearly impossible to achieve at the day of writing, since the atomic configurations existing in nature are not necessarily known or understood. Instead, a convolutional neural network can be trained on simulated images of pristine crystal structures, while still localizing and obtaining information on material imperfections . Figure shows the neural network representations obtained after dimension reduction of the fully connected layer before the classification and the corresponding uncertainty of the prediction. Although the model was trained on pristine crystal structures, it is capable to distinguish the different types of interfaces (here: grain boundaries) and the model uncertainty provides an indirect way to locate material imperfections. Approaches combining supervised, unsupervised and active learning are needed to further explore regions in data sets with high uncertainty, which may represent an unknown interface structure or surface configuration. Furthermore, the classification tasks have to be extended to also consider local composition and electronic structure to fully exploit the data and yield a holistic picture of the physical nature of a material on the atomic level. Future models should enable live feedback at high time resolution to facilitate autonomous steering of the experiment and consider active re-training to include disturbed or unknown atomic structures.