Usually, ML models are treated as black boxes, which makes it challenging to understand any physical principles learned by the models. However, we find that obtaining SHAP values from our ML models on chemical descriptors and MFFs makes our models not only accurate but also interpretable. By extracting the most important chemical features that predict gas permeability, we draw physical insights into the molecular design of polymer membranes. Here we decide to focus our analysis on the DNN ensemble because of its better performance, but other model types can also be explained using the same method. average SHAP values-their relative impacts on the six gas permeabilities under study. A summary of the definitions of these descriptors is provided in Table of Supporting Information. The most important descriptor is VSA_EState8, which is a hybrid electronic state and van der Waals surface area (VSA) descriptor , based on precalculated surface area values derived from a list of functional groups. While some of these descriptors do not have obvious, intuitive physical meaning, the permeability of polymer membranes is determined by the solubility and diffusivity of gas molecules , which is affected by the electrostatic interactions and free volume elements respectively. Therefore, these identified chemical descriptors should play important roles in gas permeability of polymer membranes. In Fig. (b), we show how the values of each of the top descriptors impacts the model's CH4 permeability prediction. If higher feature values result in more positive SHAP values, then the feature has a positive effect on permeability: the feature is directly correlated with gas permeability.