{'fraction_non_alphanumeric': 0.02033467485702182, 'fraction_numerical': 0.008525736072865919, 'mean_word_length': 5.503099173553719, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Element selection for functional materials discovery by integrated machine learning of atomic contributions to properties

At the high level, the fundamental differences between materials originate from the unique nature of the constituent chemical elements. Before specific differences emerge according to the precise ratios of elements (composition) in a given crystal structure (phase), the material can be represented by its phase field defined simply as the set of the constituent chemical elements. Classification of the materials at the level of their phase fields can accelerate materials discovery by selecting the elemental combinations that are likely to produce desirable functional properties in synthetically accessible materials. Here, we demonstrate that classification of the materials phase field with respect to the maximum expected value of a target functional property can be combined with the ranking of the materials synthetic accessibility. This end-to-end machine learning approach (PhaseSelect) first derives the atomic characteristics from the compositional environments in all computationally and experimentally explored materials and then employs these characteristics to classify the phase field by their merit. PhaseSelect can quantify the materials potential at the level of the periodic table, which we demonstrate with significant accuracy for three avenues of materials applications: high-temperature superconducting, high-temperature magnetic and targetted energy band gap materials.


Introduction
Conceptualization of novel materials begins at the level of the periodic table with selection of chemical elements for synthetic investigation. There is a variety of possible ratios or compositions that can be formed from a set of chemical elements leading to different materials (phases); the field of these potential realizations can be defined as a material's phase field. The choice of a phase field to investigate ultimately determines the outcome of the synthetic work and the functional properties of the prospective materials.
The fundamental differences between atoms result in the variance in the materials' properties in thousands of compositions accumulated in materials databases [1][2][3] . Harvesting these statistical data, there has been a surge of machine learning (ML) methods aiming to predict the materials' properties from the knowledge of their structures and compositions 4,5 . Ranging from formation enthalpy 6 to energy band gap 7 to superconducting transition temperature 8 , ML predictions enable fast screening of functional inorganic materials at scale, overcoming the otherwise forbidding combinatorial challenge for precise, but significantly more resource-demanding high-throughput quantum-mechanical calculations. At the same time, most of these high-performance ML models are based on the deep learning 9 or ensembles 10 methods that lack interpretability 11 , hence they are not readily adopted by experimental teams. Improvement of interpretability of ML approaches without compromises on performance could bridge powerful ML methods with experimental workflows to form trusted MLexpert systems in material sciences.
Codification of the materials for statistical treatment involves description of the constituent chemical elements, often represented as vectors of their chemical and physical characteristics, that are combined linearly to describe a compound 6 . This approach relies on the expert selection of a number of exploited chemical characteristics as well as the relevance of these characteristics and the corresponding weights for the atomic descriptions in materials representations. This selection determines the quality of the model 12 . The composition-based models are predisposed to data leakage between training and validation datasets via compositionally close datapoints, that impedes the extrapolation of patterns in materials-properties relationships onto unexplored materials that have distinct chemistries from those in the training set 13 .
In this work, our goal is to assess the attractiveness of candidate functional materials at the high level of the periodic table by identifying the most promising phase fields that are likely to contain these candidates. This circumvents the combinatorial challenge of individual assessment of all possible compositions built from the chosen elements and aligns with the experimental challenge of identifying new functional materials from previously uncharted chemistry. We demonstrate that unsupervised learning of chemical elements combined with the attention technique for learning elemental contributions can be used for the accurate classification of the materials' functional performance at the level of the phase fields, while improving interpretability of the ML reasoning. This end-to-end integrated machine learning (PhaseSelect) of the materials databases can prioritise the materials with respect to both probability of a merit (maximum achievable value of the target property) and synthetic accessibility of the phase fields, while the existing vast chemical knowledge is learnt each time in the context of the specific target material function.
In our approach, the machine learns all atomic elements and their specific characteristics responsible for materials formation. This is achieved by exploring possible compositional combinations in all theoretically and experimentally studied materials 14 , similarly to the concept in reference 15 . For each atom, a machine learns a vector, that encodes atomic characteristics learnt from the co-existence of atoms within some compositional environments and the absence of such co-existence with others.
Thus built atomic vectors are then combined linearly to form a phase field representation, whereas attention mechanism 16 is trained to derive the weights to the atomic vectors that magnify the most prominent atomic contributions specific to a particular property. This offers a statistically-derived alternative to the expert knowledge-based manual selection of relevant chemical characteristics and their contributions, and enables the high-level ranking and classification of materials for functional applications. Furthermore, by aggregating compositions into the phase fields in the input data, this high-level approach eliminates concerns of data leakage at the compositional level as all compositions within a phase field represent a single data entry.
We demonstrate a significant accuracy of PhaseSelect in classification of the materials with respect to three different properties: superconducting transition temperature, Curie temperature, and energy band gap, when learning the relevant property from SuperCon 3 and Materials Platform for Data Science (MPDS) 1 databases. Within these training and test sets, each phase field is labelled according to the maximum reported value of all materials within it. This maximum value is compared to the chosen thresholds (10K, 300K, 4.5eV) that reflect practical interests in high-temperature superconducting, magnetic materials and dielectrics respectively, and a class label is allocated accordingly.
In these applications, PhaseSelect demonstrates 80.4, 86.2, 75.6 % accuracy and 72.9, 84.2, 75.3% F1 score respectively. Furthermore, the phase field representations derived during properties classification are exploited to recognise patterns in elemental combinations that afford stable compositions in material databases and produce the ranking of synthetic accessibility for unexplored phase fields. The arising metrics of the phase fields -the merit probability (probability of achieving a high value of a property) and synthetic uncertainty (accessibility ranking) -can be orthogonally applied to any combination of elements at scale, creating a map of potentially attractive phase fields that can provide guidance to human researchers in the consequential and costly choice of phase fields for investigations and discovery of functional materials.

PhaseSelect model architecture
At the level of the phase fields, relationships between elemental combinations and their synthetic accessibility have been studied with unsupervised machine learning and validated experimentally 12 .
Here, we employ an integrated statistical description of atomic elements and their combinations to learn what elemental combinations have high probabilities of both synthetic realization and high values of target properties. The architecture of the model is illustrated in Figure 1. show the information flow between the various components described in this paper: 1) experimentally confirmed compositions are aggregated into the phase fields; the maximum values of the properties in the phase fields are selected; 2) compositional environments (elemental co-occurrence in materials) are aggregated from all theoretically and experimentally studied materials; 3) unsupervised learning of atomic representation from data collected in 2); 4) supervised classification of phase fields by maximum achievable values of the properties; the predicted probability of entering the high-value class is used as a merit probability; 5) unsupervised ranking of the phase fields by synthetic uncertainty; metrics derived in 4) and 5) result in a map of the phase fields' likelihood to form stable compounds with desired properties. The model is trained end-to-end so the losses of learning the atomic representation (3) and classification (4) are minimised simultaneously.
PhaseSelect consists of several connected modules (depicted as the sharp-corner rectangles in Figure   1) that pass information from the databases, while transforming the data (different data representations are depicted as the rounded-corner windows in Figure 1) and are trained simultaneously, while minimising the compound loss. We describe the data processing and the mechanisms of these modules in the following sections.

Aggregation of compositions into phase fields
For the classification and accessibility ranking of the phase fields (See bottom stream in Figure 1 Across the three property datasets, the phase fields are formed from up to 12 constituent elements, with the majority of data represented by ternary, quaternary and quinary phase fields (See Figure 2d). The abundance of chemical elements among the explored materials in the databases is illustrated in Figure   2e. All datasets have similar trends with peaks for materials containing, e.g., carbon, oxygen, sulphur, with an especially pronounced match between elemental distribution in datasets with materials for superconducting and magnetic applications (See inset in Figure 2e). The data distributions across different chemical elements observed in Figure 2e, reflect the biases in the input data: e.g., magnetism is associated with Fe predominantly, while superconductivity with Cu, etc.

Atomic representation and phase field representation
To learn atomic characteristics from the compositional environments -explored chemical compositions, where the atoms are found to form the variety of stable and metastable materials -we build a module for atomic representation based on a large materials database that includes both experimental and theoretical materials 14,15 . For each chemical element one can build a one-hot encoding vector from its instances in the database. The database is expanded into a table similarly to the approach proposed in reference 15 (depicted as a matrix of coexisting elements and compositional environments in the materials in Figure 1, 2)). The rows of the table correspond to the chemical elements, the columns are the remainders of the compositional formulas of the reported compounds, which we define here as compositional environments. For example, from stability of Li3PO4 we can learn about its constituent elements, Li, P, O and their compositional environments, "()3PO4", "()Li3O4" and "()4Li3P" respectively. In this notation, empty parentheses denote an element that by combining with the compositional environment forms a composition. Similarly, all alkali metals form the tri-"element" phosphates with "()3PO4", while trivalent elements do not, as they form the one-"element" phosphates with "()PO4" instead. In the proposed matrix representation 15 , the intersections of the rows for elements with the columns for compositional environments are filled with ones if the resulting composition is reported in 14 and with zeros otherwise. The resulting sparse matrix represents coexistence of the chemical elements and compositional environments in the materials. We then employ a shallow autoencoder neural network -an unsupervised ML technique -to reduce the dimensionality of this matrix, and to condense the information into the rich latent space of dimensionality k, in which similar atomic vectors (of length k) are grouped close to each other. We study the effects of the size of dimensionality k of thus derived atomic vectors on the classification accuracy to select the most efficient atomic description ( Supplementary Fig. 1). We use the vectors of the most efficient latent space as atomic representations to build up the phase fields descriptions   Unexplored ternary phase fields that are classified to exhibit superconductivity at T > 10 K with more than 70% probability and that have high likelihood of forming stable compounds with synthetic uncertainty (accessibility ranking) < 0.2, demonstrate trends in constituent elements: most of the top 50 phase fields are predicted to contain Mg, Fe, Nb and N. b Unexplored ternary phase fields that are classified to exhibit an energy band gap > 4.5 eV with more than 75% probability and that have high likelihood of forming stable compounds (with synthetic accessibility score < 0.1) demonstrate trends in distribution by constituent elements: different combinations of Hg-, F-, Bi-, Hfand Pb-based phase fields have the highest probabilities. c Unexplored ternary phase fields that are classified to exhibit magnetic properties at Curie T > 300 K with more than 71% probability and that have high likelihood of forming stable compounds (with synthetic accessibility score < 0.1) demonstrate trends in constituent elements: all top-ranked phase fields are Fe-based, with many phase fields containing Co and Y. While these predictions may align well with the human experts' understanding of chemistry, hence emphasizing the models' ability to infer complex atomic characteristics and phase fields-properties relationship from historical data, the models can also be used to identify unconventional and rare prospective elemental combinations as well as to rank the attractive candidate materials for experimental investigations.     a Accuracy and F1 score for classification of materials with respect to the maximum of superconducting transition temperature threshold 10 K: the best performing model has 80 atomic features; b Accuracy and F1 score for classification of materials with respect to the maximum of Curie transition temperature threshold 300 K: the best performing model has 50 atomic features; c Accuracy and F1 score for classification of materials with respect to the maximum of energy band gap threshold 4.5 eV: the best performing model has 60 atomic features.

Attention to atomic contributions maximizing the properties
In the end-to-end classification models, we employ an attention mechanism 3 to emphasize those atomic contributions that minimize the combined loss, and hence maximize classification metrics.
To incorporate information about atomic bonding interplay from all available data, the variance in size of the phase fields is alleviated by zero-padding in the phase fields representation module that further allows extrapolation of the patterns derived from the explored materials onto the candidate phase fields of arbitrary number of elements. We extract the attention scores obtained during the training of the models that illustrate atomic contributions to the properties manifested by the phase fields ( Supplementary Figures 2-6). For visualisation, the attention scores are averaged across the attention heads and across all instances of the atomic pairs in the corresponding datasets. In Supplementary   Figure 11, distributions of the averaged attention scores are plotted for the atoms that contribute the most to identify phase fields that manifest particular properties. The atomic contributions weights are also used for building a model for an arbitrary number of elements in a phase field. For this, we create all phase fields representations vectors of an equal size l, corresponding to the largest phase field in a database, and pad the smaller phase field vectors, of size s, with l -s zeros, that will have zero attention weights, but will further enable formation of a neural network layer for processing of all input data in a single model. The described construction of a phase field representation with local attention weights also makes the model insensitive to the order in which atomic elements are listed in a phase field, without the need to take into account all possible permutation of the elements. The performance metrics from the 5 models for each dataset are then averaged to describe a general ability of the models' architecture to learn from the available data. For validation of the unsupervised models for the phase fields ranking with respect to synthetic accessibility, we employ an approach developed in 5 . We perform 5-fold cross validation, in which the validation error is defined as the percentage of entries in the test set that evaluated with normalized reconstruction errors in the 20% of the maximum Supplementary Table 2. Additionally, we compare the predicted reconstruction errors for the validation sets with the ground truth reconstruction errors obtained for the same entries in unsupervised training, when the entries are included in the training data ( Supplementary Fig. 8) and calculate the mutual information score adjusted against chance 6 (Supplementary Table 2). The typical training process of the ranking autoencoder neural network for different datasets are depicted in Supplementary Fig. 9.

Prediction of superconducting behaviour for reported phase fields in ICSD-v2021
We apply PhaseSelect ensembles of classification models to identify likely candidates for novel superconducting materials among the phase fields that have been reported to form stable compounds in ICSD-v2021, but were not investigated from the perspectives of superconducting applications and reported in MPDS and SuperCon (hence were not included into the training dataset). The excerpt of these predictions is presented in Supplementary Table 7; classification of all binary, ternary and quaternary phase field in ICSD with respect to the maximum accessible value of superconducting critical temperature is uploaded in 10 .