Predicting the band gap of binary compounds from machine-learning regression methods
Density functional theory (DFT) is a ubiquitous first-principles method, but the approximate nature of the exchange-correlation functional poses an inherent limitation for the accuracy of various computed properties. In this context, surrogate models based on machine learning have the potential to provide a more efficient and physically meaningful understanding of electronic properties, such as the band gap. Here, we construct a gradient boosting regression (GBR) model for prediction of the band gap of binary compounds from simple physical descriptors, using a dataset of over 4000 DFT-computed band gaps. Out of 27 features, electronegativity, periodic group, and highest occupied energy level exhibit the highest importance score, consistent with the underlying physics of the electronic structure. We obtain a model accuracy of 0.81 and root mean squared error of 0.26 eV using the top five features, achieving accuracy comparable to previously reported values but employing less number of features. Our work presents a rapid and interpretable prediction model for solid-state band gap with high fidelity to DFT and can be extended beyond binary materials considered in this study.
Introduction<!>Gradient boosting regression<!>𝐹 ̂= argmin 𝐹 𝔼 𝑥,𝑦 [𝐿(𝑦, 𝐹(𝑥))].<!>Dataset and features<!>Feature importance and efficacy<!>Conclusions
<p>According to the Schrödinger equation, a constrained particle occupies several eigenstates with quantized energy levels. For example, the ground state energy of the electron in a hydrogen atom is -13.6 eV, and the higher energy levels can be accessed by absorbing a photon whose wavelength is determined by the Rydberg equation. In solids with multiple atoms, the electronic states turn into continuous bands. The Fermi level divides the energy range into the valence and conduction bands. Insulators and semiconductors have bandgaps, the energy range where electrons are forbidden to occupy. To conduct electricity, electrons need to be excited across the bandgap from the valence band into the conduction band. Bandgaps, especially that of semiconductors, play a central role in many device applications, such as transistors, 1 light emitting diodes (LED), 2,3 photovoltaics, 4 and thermoelectric materials. 5 Therefore, accurate evaluation of bandgaps from computation is crucial to rational materials design.</p><p>Density functional theory (DFT) is widely used to calculate the electronic structure of solids. In the Kohn-Sham equation, the exact form of the exchange-correlation energy, corresponding to many-body quantum interactions, remains unknown in terms of the electron density. As such, an approximation is needed, such as local density approximation (LDA) and generalized gradient approximation (GGA). Conventional DFT based on these approximations fail to calculate accurate bandgaps, and many insulators and semiconductors are predicted to be metallic. 6,7 As a result, more computationally demanding methods such as hybrid functionals 8 and GW method are needed. 9 A much cheaper and popular alternative is DFT+U, where Hubbard U on-site repulsion correction is applied to increase the bandgap. However, such corrections can lead to unphysical distortions of the electronic structure profile. 10 Recently, machine learning has seen an increase in applications to overcome many of these limitations of conventional DFT, e.g. machine-learning density functionals 11 , as well as datadriven prediction of electronic properties such as band gaps at higher accuracy. 12 For example, regression methods such as support vector machine, 12 k-nearest neighbors, 13 and logistic regression, 14 have been used to successfully predict bandgaps using experimentally measured values.</p><p>In recent years, machine-learning assisted screening of functional materials has garnered much attention. 12,13,15 Machine learning methods have been used to predict a variety of materials properties, such as hardness, electrical conductivity, 16 melting temperatures, 17 phase stability, 18 and ionic conductivity. 19 The conventional trial-and-error method relies on the intuition and experience of researchers and requires a lot of time and resources. Although computational screening based on firstprinciples methods greatly facilitates the screening process, the computational cost remains very high. Machine learning is a viable alternative that can enable faster and more efficient exploration of the large materials space.</p><p>Here, we construct a gradient boosting regression model for prediction of the band gap of binary compounds from simple physical descriptors, using a dataset of more than 4000 DFT-computed band gaps. We build on a previous study of double perovskites that used random forest regression and linear ridge regression to predict DFT-computed band gaps with an improved accuracy compared to prior literature. 13 Out of 27 features, electronegativity, periodic group, and highest occupied energy level have the highest importance score, which provide a physically interpretable understanding of the band gap. Moreover, our model is capable of achieving accuracy comparable to previously reported values but employing less number of features.</p><!><p>In regression, the prediction function 𝐹 ̂ is optimized by minimizing a loss function 𝐿(𝑦, 𝐹(𝑥)) for the true value 𝑦 and the prediction 𝐹(𝑥): 20</p><!><p>The gradient boosting method assumes a real-valued 𝑦 and seeks an approximation 𝐹 ̂ in the form of a weighted sum of functions ℎ(𝑥), called base (or weak) learners, with weights 𝛾(𝑥):</p><p>The loss function is iteratively minimized using the steepest descent. The maximum-descent direction of the loss function 𝐹 𝑚 at iteration 𝑚 is given by the function at the previous step 𝑚 − 1 subtracted by the loss function gradient:</p><!><p>We choose the dataset of DFT band gaps of 4096 binary compounds, 21 computed using the Perdew-Berke-Ernzerhof (PBE) parametrization 22 of the generalized gradient approximation (GGA) of the exchange-correlation functional. The dataset is available through the matminer package. 23 27 atomic features are obtained using the Pymatgen (Python Materials Genomics) package, 24 including Pauling electronegativity, ionization potential, highest occupied atomic level, lowest unoccupied atomic level, and s-, p-and d-valence orbital radii of isolated neutral atoms. 25 Each element in a given compound is classified as either host or guest based on their relative composition from which weighted average of each feature is computed. All regression analysis is performed using the scikit-learn package. 26  We first compare linear vs. nonlinear regression. Linear ridge regression (LRR) and gradient boosting regression (GBR) models provide test set score of 0.50 and 0.85, respectively, as can be seen by data points lying closer to the parity line with GBR in Fig. 1a. This observation suggests that nonlinear regression is better suited to describe our dataset. GBR uses an ensemble of weak prediction models, typically decision trees, where boosting is used for optimization on a suitable cost function. 27 The main advantage of GBR is its resilience against overfitting. As such, we employ GBR to generate all results presented in the following discussions.  Next, we optimize three main parameters of the GBR model: (1) learning_rate for shrinking the contribution of each weak prediction trees; (2) subsample for the fraction of samples to be used for fitting the individual base learners; and (3) n_estimators for the number of boosting iterations. Fig. 2 shows test set deviance while varying either learning _rate or subsample parameter at a time. Better model performance is seen with decreasing learning_rate and increasing subsample. Across both cases, the model remains fairly robust to overfitting and the deviance decreases exponentially, plateauing after around 200 th iteration. As such, better prediction can be obtained with larger values of n_estimators. From further grid search in Fig. 3, optimal parameters are chosen as learning_rate=0.1, subsample=1, and n_estimators=1000.  We investigate the effect of different training and test set partitions on the model performance, ranging from test/training ratio of 10/90 to 75/25. (Fig. 4). Increasing the relative proportion of the test set leads to a decrease in the test set score with concomitant increase of the root mean squared error (RMSE) and the training set score (Table 1). Based on this observation, we choose the test/training ratio of 20/80, which provides an optimal compromise of the training and test set score of 0.97 and 0.84, respectively, while maintaining a reasonable RMSE of 0.19 eV. Both test and training set deviances decrease across boosting iterations, the latter more so than the former, consistent with the observation of Fig. 2.</p><!><p>Table 2. Training and test set scores and test set root mean square error (RMSE) for models with different number of features with the highest importance score as enumerated in Fig. 5 and 6   We characterize the feature importance in terms of two metrics: (1) mean decrease in impurity (MDI) quantifies the mean and standard deviation of the accumulation of the impurity decrease within each tree, computed on the training set; (2) permutation importance quantifies the decrease in a model score when a single feature value is randomly shuffled, computed on the test set. Sequential reduction of the feature dimension is conducted to determine the optimal number of features with the highest importance score, ranging from 17 down to 3 features (Fig. 5). Overall, the model performance does not vary appreciably when the number of futures is reduced from 27 to 5 (Table 2), with the training and test set scores varying 0.95-0.97 and 0.78-0.84, respectively, and the RMSE varying 0.22-0.28 eV. The performance drops significantly when only three features are used, with the respective metrics of 0.88, 0.66, and 0.46 eV. Based on this observation, we conclude that 5 features are sufficient to retain high accuracy at levels comparable to previously reported model score of 0.919 using 12 features; 15 and cross-validation score of 96% and RMSE of 0.28 eV using 21 features. 27 Three main types of features exhibited the highest importance score: electronegativity, periodic group, and highest occupied energy level. In terms of physics, the periodic potential of the constituent ions would influence the electronic structure and the overall conductivity as determined by the band gap. Electronegativity quantifies how strongly an atom attracts an electron within a chemical bond. In solid-state systems, electronegativity would correspond to the extent of electron localization around the constituent atoms, which influences the insulating character of the material. Such atomic feature has a close correlation with the periodic group, which is the column of the periodic table that an element belongs to. Transition metals toward the right side of the periodic table have higher number of protons, which increases the electronegativity, as well as higher number of electrons, which stabilizes the electrons and lowers the d-band center and the valence band maximum corresponding to the highest occupied energy level.</p><!><p>First-principles modeling based on density functional theory (DFT) is ubiquitous in computational materials science but remains limited in its accuracy and cost. Machine learning has the potential to provide efficient and physically interpretable model of crucial electronic structure properties, such as the band gap, while maintaining high fidelity to DFT. In this work, we construct a gradient boosting regression (GBR) model for prediction of the band gap of binary compounds from simple physical descriptors, using a dataset of over 4000 DFT-computed band gaps. Out of 27 features, electronegativity, periodic group, and highest occupied energy level exhibit the highest importance score, consistent with the underlying physics of the electronic structure. Using the top five features, we obtain a model accuracy of 0.81 and root mean squared error of 0.26 eV, which is comparable to previously reported values but employing fewer features. Our approach is broadly applicable for rapid and interpretable prediction of solid-state band gap for other types of materials beyond binary systems considered in this study.</p>