Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Chemical reactions can be described as the stepwise redistribution of electrons in molecules. As such, reactions are often depicted using "arrow-pushing" diagrams which show this movement as a sequence of arrows. We propose an electron path prediction model (ELECTRO) to learn these sequences directly from raw reaction data. Instead of predicting product molecules directly from reactant molecules in one shot, learning a model of electron movement has the benefits of (a) being easy for chemists to interpret, (b) incorporating constraints of chemistry, such as balanced atom counts before and after the reaction, and (c) naturally encoding the sparsity of chemical reactions, which usually involve changes in only a small number of atoms in the reactants. We design a method to extract approximate reaction paths from any dataset of atom-mapped reaction SMILES strings. Our model achieves excellent performance on an important subset of the USPTO reaction dataset, comparing favorably to the strongest baselines. Furthermore, we show that our model recovers a basic knowledge of chemistry without being explicitly trained to do so. The ability to reliably predict the products of chemical reactions is of central importance to the manufacture of medicines and materials, and to understand many processes in molecular biology. Theoretically, all chemical reactions can be described by the stepwise rearrangement of electrons in molecules (b). This sequence of bond-making and breaking is known as the reaction mechanism. Understanding the reaction mechanism is crucial because it not only determines the products (formed at the last step of the mechanism), but it also provides insight into why the products are formed on an atomistic level. Mechanisms can be treated at different levels of abstraction. On the lowest level, quantum-mechanical simulations of the electronic structure can be performed, which are prohibitively computationally expensive for most systems of interest. On the other end, chemical reactions can be treated as rules that "rewrite" reactant molecules to products, which abstracts away the individual electron redistribution steps into a single, global transformation step. To combine the advantages of both approaches, chemists use a powerful qualitative model of quantum chemistry colloquially called "arrow pushing", which simplifies the stepwise electron shifts using sequences of arrows which indicate the path of electrons throughout molecular graphs (b).Recently, there have been a number of machine learning models proposed for directly predicting the products of chemical reactions BID2;; a;; ), largely using graph-based or machine translation models. The task of reaction product prediction is shown on the left-hand side of FIG0.In this paper we propose a machine learning model to predict the reaction mechanism, as shown on the right-hand side of FIG0, for a particularly important subset of organic reactions. We argue that our The reaction product prediction problem: Given the reactants and reagents, predict the structure of the product. (Right) The reaction mechanism prediction problem: Given the reactants and reagents, predict how the reaction occurred to form the products.model is not only more interpretable than product prediction models, but also allows easier encoding of constraints imposed by chemistry. Proposed approaches to predicting reaction mechanisms have often been based on combining hand-coded heuristics and quantum mechanics BID0;; b;;; ), rather than using machine learning. We call our model ELECTRO, as it directly predicts the path of electrons through molecules (i.e., the reaction mechanism). To train the model we devise a general technique to obtain approximate reaction mechanisms purely from data about the reactants and products. This allows one to train our a model on large, unannotated reaction datasets such as USPTO . We demonstrate that not only does our model achieve impressive , surprisingly it also learns chemical properties it was not explicitly trained on. We begin with a brief from chemistry on molecules and chemical reactions, and then review related work in machine learning on predicting reaction outcomes. We then describe a particularly important subclass of chemical reactions, called linear electron flow (LEF) reactions, and summarize the contributions of this work. Organic (carbon-based) molecules can be represented via a graph structure, where each node is an atom and each edge is a covalent bond (see example molecules in FIG0). Each edge (bond) represents two electrons that are shared between the atoms that the bond connects. Electrons are particularly important for describing how molecules react with other molecules to produce new ones. All chemical reactions involve the stepwise movement of electrons along the atoms in a set of reactant molecules. This movement causes the formation and breaking of chemical bonds that changes the reactants into a new set of product molecules (a). For example, FIG0 (Right) shows how electron movement can break bonds (red arrows) and make new bonds (green arrows) to produce a new set of product molecules. In general, work in machine learning on reaction prediction can be divided into two categories: Product prediction, where the goal is to predict the reaction products, given a set of reactants and reagents, shown in the left half of FIG0; and Mechanism prediction, where the goal is to determine how the reactants react, i.e., the movement of electrons, shown in the right of FIG0.Product prediction. Recently, methods combining machine learning and template-based molecular rewriting rules have been proposed BID2 a;;; ). Here, a learned model is used to predict which rewrite rule to apply to convert one molecule into another. While these models are readily interpretable, they tend be brittle. Another approach, introduced by , constructs a neural network based on the Weisfeiler-Lehman algorithm for testing graph isomorphism. They use this algorithm (called WLDN) to select atoms that will be involved in a reaction. They then enumerate all chemically-valid bond changes involving these atoms and learn a separate network to Table 1: Work on machine learning for reaction prediction, and whether they are (a) end-to-end trainable, and (b) predict the reaction mechanism.rank the ing potential products. This method, while leveraging new techniques for deep learning on graphs, cannot be trained end-to-end because of the enumeration steps for ensuring chemical validity. represents reactants as SMILES strings and then uses a sequence to sequence network (specifically, the work of) to predict product SMILES. While this method (called Seq2Seq) is end-to-end trainable, the SMILES representation is quite brittle as often single character changes will not correspond to a valid molecule. These latter two methods, WLDN and Seq2Seq, are state-of-the-art on product prediction and have been shown to outperform the above template-based techniques . Thus we compare directly with these two methods in this work. Mechanism prediction. The only other work we are aware of to use machine learning to predict reaction mechanisms are; Kayala and Baldi (2011; 2012);. All of these model a chemical reaction as an interaction between atoms as electron donors and as electron acceptors. They predict the reaction mechanisms via two independent models: one that identifies these likely electron sources and sinks, and another that ranks all combinations of them. These methods have been run on small expert-curated private datasets, which contain information about the reaction conditions such as the temperature and anion/cation solvation potential (, §2). In contrast, in this work, we aim to learn reactions from noisy large-scale public reaction datasets, which are missing the required reaction condition information required by these previous works. As we cannot yet apply the above methods on the datasets we use, nor test our models on the datasets they use (as the data are not yet publicly released), we cannot compare directly against them; therefore, we leave a detailed investigation of the pros and cons of each method for future work. As a whole, this related work points to at least two main desirable characteristics for reaction prediction models:1. End-to-End: There are many complex chemical constraints that limit the space of all possible reactions. How can we differentiate through a model subject to these constraints? 2. Mechanistic: Learning the mechanism offers a number of benefits over learning the products directly including: interpretability (if the reaction failed, what electron step went wrong), sparsity (electron steps only involve a handful of atoms), and generalization (unseen reactions also follow a set of electron steps). Table 1 describes how the current work on reaction prediction satisfies these characteristics. In this work we propose to model a subset of mechanisms with linear electron flow, described below. Reaction mechanisms can be classified by the topology of their "electron-pushing arrows" (the red and green arrows in FIG0 . Here, the class of reactions with linear electron flow (LEF) topology is by far the most common and fundamental, followed by those with cyclic topology (a). In this work, we will only consider LEF reactions that are heterolytic, i.e., they involve pairs of electrons. If reactions fall into this class, then a chemical reaction can be modelled as pairs of electrons moving in a single path through the reactant atoms. In arrow pushing diagrams representing LEF reactions, this electron path can be represented by arrows that line up in sequence, differing from for example pericyclic reactions in which the arrows would form a loop (a).Further for LEF reactions, the movement of the electrons along the linear path will alternately remove existing bonds and form new ones. We show this alternating structure in the right of FIG0. The reaction formally starts by (step 1) taking the pair of electrons between the Li and C atoms and moving them to the C atom; this is a remove bond step. Next (step 2) a bond is added when electrons are moved from the C atom in reactant 1 to a C atom in reactant 2. Then (step 3) a pair of electrons are removed between the C and O atoms and moved to the O atom, giving rise to the products. Predicting the final product is thus a byproduct of predicting this series of electron steps. Contributions. We propose a novel generative model for modeling the reaction mechanism of LEF reactions. Our contributions are as follows:• We propose an end-to-end generative model for predicting reaction mechanisms, ELECTRO, that is fully differentiable. It can be used with any deep learning architecture on graphs.• We design a technique to identify LEF reactions and mechanisms from purely atom-mapped reactants and products, the primary format of large-scale reaction datasets.• We show that ELECTRO learns chemical knowledge such as functional group selectivity without explicit training. In this section we define a probabilistic model for electron movement in linear electron flow (LEF) reactions. As described above (§2.1) all molecules can be thought of as graphs where nodes correspond to atoms and edges to bonds. All LEF reactions transform a set of reactant graphs, M 0 into a set of product graphs M T +1 via a series of electron actions P 0:T = (a 0, . . ., a T). As described, these electron actions will alternately remove and add bonds (as shown in the right of FIG0). This reaction sometimes includes additional reagent graphs, M e, which help the reaction proceed, but do not change themselves. We propose to learn a distribution p θ (P 0:T | M 0, M e) over these electron movements. We first detail the generative process that specifies p θ, before describing how to train the model parameters θ. To define our generative model, we describe a factorization of p θ (P 0:T | M 0, M e) into three components: 1. the starting location distribution p start θ (a 0 | M 0, M e); 2. the electron movement distribution p θ (a t | M t, a t−1, t); and 3. the reaction continuation distribution p cont θ (c t | M t). We define each of these in turn and then describe the factorization (we leave all architectural details of the functions introduced to the appendix).Starting Location. At the beginning the model needs to decide on which atom a 0 starts the path. As this is based on (i) the initial set of reactants M 0 and possibly (ii) a set of reagents M e, we propose to learn a distribution p DISPLAYFORM0 To parameterize this distribution we propose to use any deep graph neural network, denoted h A (·), to learn graph-isomorphic node features from the initial atom and bond features 2 (; ; ;). We choose to use a 4 layer Gated Graph Neural Network (GGNN) , for which we include a short review in the appendix. Given these atom embeddings we also compute graph embeddings (, §B.1) (also called an aggregation graph transformation (, §3)), which is a vector that represents the entire molecule set M that is invariant to any particular node ordering. Any such function g(·) that computes this mapping can be used here, but the particular graph embedding function we use is inspired by , and described in detail in Appendix B. We can now parameterize p We represent the characteristic probabilities the model may have over these next actions as colored circles over each atom. Some actions are disallowed on certain steps, for instance you cannot remove a bond that does not exist; these blocked actions are shown as red crosses. DISPLAYFORM1 where f start is a feedforward neural network which computes logits x; the logits are then normalized into probabilities by the softmax function, defined as softmax DISPLAYFORM2 Electron Movement. Observe that since LEF reactions are a single path of electrons (§2.3), at any step t, the next step a t in the path depends only on (i) the intermediate molecules formed by the action path up to that point M t, (ii) the previous action taken a t−1 (indicating where the free pair of electrons are), and (iii) the point of time t through the path, indicating whether we are on an add or remove bond step. Thus we will also learn the electron movement distribution p θ (a t | M t, a t−1, t).Similar to the starting location distribution we again make use of a graph-isomorphic node embedding function h A (M). In contrast, the above distribution can be split into two distributions depending on the parity of t: the remove bond step distribution p remove θ (a t | M t, a t−1) when t is odd, and the add bond step distribution p add θ (a t | M t, a t−1) when t is even. We parameterize the distributions as p DISPLAYFORM3 DISPLAYFORM4 The vectors β remove, β add are masks that zero-out the probability of certain atoms being selected. Specifically, β remove sets the probability of any atoms a t to 0 if there is not a bond between it and the previous atom a t−1 3. The other mask vector β add masks out the previous action, preventing the model from stalling in the same state for multiple time-steps. The feedforward networks f add (·), f remove (·) and other architectural details are described in Appendix C.Reaction Continuation / Termination. Additionally, as we do not know the length of the reaction T, we introduce a latent variable c t ∈ {0, 1} at each step t, which describes whether the reaction continues (c t = 1) or terminates (c t = 0) 4. We also define an upper bound T max on the number of reaction steps.3 One subtle point is if a reaction begins with a lone-pair of electrons then we say that this reaction starts by removing a self-bond. Thus, in the first remove step β remove it is possible to select a1 = a0. But this is not allowed via the mask vector in later steps. 4 An additional subtle point is that we do not allow the reaction to stop until until it has picked up an entire pair (ie c1 = 1). The generative steps of ELECTRO (given that the model chooses to react, ie c 0 = 1). Input: Reactant molecules M 0 (consisting of atoms A), reagents M e, atom embedding function h A (·), graph embedding functions g reagent (·) and g cont (·), additional logit functions DISPLAYFORM0 The molecule does not change until complete pair picked up 4: c 1 1You cannot stop until picked up complete pair 5: for t = 1,..., T max do DISPLAYFORM1 if t is odd then 7: DISPLAYFORM2 electrons remove bond between a t and a t−1 9: DISPLAYFORM3 11: DISPLAYFORM4 electrons add bond between a t and a t−1 12:end if13: DISPLAYFORM5 M t+1 ← M t, a t modify molecules based on previous molecule and action 15: DISPLAYFORM6 16: The final distribution we learn is the continuation distribution p cont θ (c t | M t). For this distribution we learn a different graph embedding function g cont (·) to decide whether to continue or not: DISPLAYFORM7 DISPLAYFORM8 where σ is the sigmoid function σ(a) = 1/(1 + e −a).Path Distribution Factorization. Given these distributions we can define the probability of a path P 0:T with the distribution p θ (P 0:T | M 0, M e), which factorizes as FIG1 gives a graphical depiction of the generative process on a simple example reaction. Algorithm 1 gives a more detailed description. DISPLAYFORM9 DISPLAYFORM10 Training We can learn the parameters θ of all the parameterized functions, including those producing node embeddings, by maximizing the log likelihood of a full path log p θ (P 0: DISPLAYFORM11 This is evaluated by using a known electron path a t and intermediate products M t extracted from training data, rather than on simulated values. This allows us to train on all stages of the reaction at once, given electron path data. We train our models using Adam and an initial learning rate of 10 −4, with minibatches consisting of a single reaction, where each reaction often consists of multiple intermediate graphs. Prediction Once trained, we can use our model to sample chemically-valid paths given an input set of reactants M 0 and reagents M e, simply by simulating from the conditional distributions until sampling a continue value equal to zero. We instead would like to find a ranked list of the top-K predicted paths, and do so using a modified beam search, in which we roll out a beam of width K until a maximum path length T max, while recording all paths which have terminated. This search procedure is described in detail in Algorithm 2 in the appendix. To evaluate our model, we use a collection of chemical reactions extracted from the US patent database . We take as our starting point the 479,035 reactions, along with the training, validation, and testing splits, which were used by , referred to as the USPTO dataset. This data consists of a list of reactions. Each reaction is a reaction SMILES string and a list of bond changes. SMILES is a text format for molecules that lists the molecule as a sequence of atoms and bonds. The bond change list tells us which pairs of atoms have different bonds in the the reactants versus the products (note that this can be directly determined from the SMILES string). Below, we describe two data processing techniques that allow us to identify reagents, reactions with LEF topology, and extract an underlying electron path. Each of these steps can be easily implemented with the open-source chemo-informatics software RDKit (RDKit, online).Reactant and Reagent Seperation Reaction SMILES strings can be split into three parts -reactants, reagents, and products. The reactant molecules are those which are consumed during the course of the chemical reaction to form the product, while the reagents are any additional molecules which provide context under which the reaction occurs (for example, catalysts), but do not explicitly take part in the reaction itself; an example of a reagent is shown in FIG0.Unfortunately, the USPTO dataset as extracted does not differentiate between reagents and reactants. We elect to preprocess the entire USPTO dataset by separating out the reagents from the reactants using the process outlined in Schwaller et al. FORMULA1, where we classify as a reagent any molecule for which either (i) none of its constituent atoms appear in the product, or (ii) the molecule appears in the product SMILES completely unchanged from the pre-reaction SMILES. This allows us to properly model molecules which are included in the dataset but do not materially contribute to the reaction. To train our model, we need to (i) identify reactions in the USPTO dataset with LEF topology, and (ii) have access to an electron path for each reaction. FIG3 shows the steps necessary to identify and extract the electron paths from reactions exhibiting LEF topology. We provide further details in Appendix D.Applying these steps, we discover that 73% of the USPTO dataset consists of LEF reactions (349,898 total reactions, of which 29,360 form the held-out test set). Table 2: Results when using ELECTRO for mechanism prediction. Here a prediction is correct if the atom mapped action sequences predicted by our model match exactly those extracted from the USPTO dataset. 1st Choice 2nd Choice 3rd ChoiceWe now evaluate ELECTRO on the task of (i) mechanism prediction and (ii) product prediction (as described in FIG0). While generally, it is necessary to know the reagents M e of a reaction to faithfully predict the mechanism and product, it is often possible to make inferences from the reactants alone. Therefore, we trained a second version of our model that we call ELECTRO-LITE, which ignores reagent information. This allows us to gauge the importance of reagents in determining the mechanism of the reaction. Example of how we turn a SMILES reaction string into an ordered electron path, for which we can train ELECTRO on. This consists of a series of steps: Identify bonds that change by comparing bond triples (source node, end node, bond type) between the reactants and products. FORMULA3 Join up the bond changes so that one of the atoms in consecutive bond changes overlap (for reactions which do not have linear electron flow topology, such as multi-step reactions, this will not be possible and so we discard these reactions). Order the path (ie assign a direction). A gain of charge (or analogously the gain of hydrogen as H + ions without changing charge, such as in the example shown) indicates that the electrons have arrived at this atom; and vice-versa for the start of the path. When details about both ends of the path are missing from the SMILES string we fall back to using an element's electronegativity to estimate the direction of our path, with more electronegative atoms attracting electrons towards them and so being at the end of the path. The extracted electron path deterministically determines a series of intermediate molecules which can be used for training ELECTRO. Paths that do not consist of alternative add and removal steps and do not in the final recorded product do not exhibit LEF topology and so can be discarded. An interesting observation is that our approximate reaction mechanism extraction scheme implicitly fills in missing reagents, which are caused by noisy training data -in this example, which is a Grignard-or Barbier-type reaction, the test example is missing a metal reagent (e.g. Mg or Zn). Nevertheless, our model is robust enough to predict the intended product correctly . For mechanism prediction we are interested in ensuring we obtain the exact sequence of electron steps correctly. We evaluate accuracy by checking whether the sequence of integers extracted from the raw data as described in Section 4 is an exact match with the sequence of integers output by ELECTRO. We compute the top-1, top-2, top-3, and top-5 accuracies and show them in Table 2, with an example prediction shown in FIG2. Reaction mechanism prediction is useful to ensure we form the correct product in the correct way. However, it underestimates the model's actual predictive accuracy: although a single atom mapping is provided as part of the USPTO dataset, in general atom mappings are not unique (e.g., if a molecule contains symmetries). Specifically, multiple different sequences of integers could correspond to chemically-identical electron paths. The first figure in the appendix shows an example of a reaction with symmetries, where different electron paths produce the exact same product. Recent approaches to product prediction have evaluated whether the major product reported in the test dataset matches predicted candidate products generated by their system, independent of mechanism. In our case, the top-5 accuracy for a particular reaction may include multiple different electron paths that ultimately yield the same product molecule. To evaluate if our model predicts the same major product as the one in the test data, we need to solve a graph isomorphism problem. To approximate this we (a) take the predicted electron path, (b) apply these edits to the reactants to produce a product graph (balancing charge to satisfy valence Table 3 : Results for product prediction, following the product matching procedure in Section 5.2. For the baselines we compare against models trained (a) on the full USPTO training set (marked FTS) and only tested on our subset of LEF reactions, and (b) those that are also trained on the same subset as our model. We make use of the code and pre-trained models provided by. For the Seq2Seq approach, as neither code nor more fine grained are available, we train up the required models from scratch using the OpenNMT library . Figure 5: (Left) Nucleophilic substitutions S N 2-reactions, (right) Suzuki-coupling (note that in the "real" mechanism of the Suzuki coupling, the reaction would proceed via oxidative insertion, transmetallation and reductive elimination at a Palladium catalyst. As these details are not contained in training data, we treat Palladium implicitly as a reagent). In both cases, our model has correctly picked up the trend that halides lower in the period table usually react preferably (I > Br > Cl).constraints), (c) remove atom mappings, and (d) convert the product graph to a canonical SMILES string representation in Kekulé form (aromatic bonds are explicitly represented as double-bonds). We can then evaluate whether a predicted electron path matches the ground truth by a string comparison. This procedure is inspired by the evaluation of. To obtain a ranked list of products for our model, we compute this canonicalized product SMILES for each of the predictions found by beam search over electron paths, removing duplicates along the way. These product-level accuracies are reported in Table 3.We compare with the state-of-the-art graph-based method Jin et al. FORMULA1; we use their evaluation code and pre-trained model 5, re-evaluated on our extracted test set. We also use their code and re-train a model on our extracted training set, to ensure that any differences between our method and theirs is not due to a specialized training task. We also compare against the Seq2Seq model proposed by ; however, as no code is provided by Schwaller et al. FORMULA1, we run our own implementation of this method based on the OpenNMT library . Overall, ELECTRO outperforms all other approaches on this task, with 87% top-1 accuracy and 95.9% top-5 accuracy. Omitting the reagents in ELECTRO degrades top-1 accuracy slightly, but maintains a high top-3 and top-5 accuracy, suggesting that reagent information is necessary to provide context in disambiguating plausible reaction paths. Complex molecules often feature several potentially reactive functional groups, which compete for reaction partners. To predict the selectivity, that is which functional group will predominantly react in the presence of other groups, students of chemistry learn heuristics and trends, which have been established over the course of three centuries of experimental observation. To qualitatively study whether the model has learned such trends from data we queried the model with several typical text book examples from the chemical curriculum (see Figure 5 and the appendix). We found that the model predicts most examples correctly. In the few incorrect cases, interpreting the model's output reveals that the model made chemically plausible predictions. In this section we briefly list a couple of limitations of our approach and discuss any pointers towards their resolution in future work. LEF Topology ELECTRO can currently only predict reactions with LEF topology (§2.3). These are the most common form of reactions (b), but in future work we would like to extend ELECTRO's action repertoire to work with other classes of electron shift topologies such as those found in pericyclic reactions. This could be done by allowing ELECTRO to sequentially output a series of paths, or by allowing multiple electron movements at a single step. Also, since the approximate mechanisms we produce for our dataset are extracted only from the reactants and products, they may not include all observable intermediates. This could be solved by using labelled mechanism paths, obtainable from finer grained datasets containing also the mechanistic intermediates. These mechanistic intermediates could also perhaps be created using quantum mechanical calculations following the approach in.Graph Representation of Molecules Although this shortcoming is not just restricted to our work, by modeling molecules and reactions as graphs and operations thereon, we ignore details about the electronic structure and conformational information, ie information about how the atoms in the molecule are oriented in 3D. This information is crucial in some important cases. Having said this, there is probably some balance to be struck here, as representing molecules and reactions as graphs is an extremely powerful abstraction, and one that is commonly used by chemists, allowing models working with such graph representations to be more easily interpreted. In this paper we proposed ELECTRO, a model for predicting electron paths for reactions with linear electron flow. These electron paths, or reaction mechanisms, describe how molecules react together. Our model (i) produces output that is easy for chemists to interpret, and (ii) exploits the sparsity and compositionality involved in chemical reactions. As a byproduct of predicting reaction mechanisms we are also able to perform reaction product prediction, comparing favorably to the strongest baselines on this task. In the main text we described the challenges of how to evaluate our model, as different electron paths can form the same products, for instance due to symmetry. Figure 6 is an example of this. Figure 6: This example shows how symmetry can affect the evaluation of electron paths. In this example, although one electron path is given in the USPTO dataset, the initial N that reacts could be either 15 or 12, with no difference in the final product. This is why judging purely based on electron path accuracy can sometimes be misleading. In this section we briefly review existing work for forming node and graph embeddings, as well as describing more specific details relating to our particular implementation of these methods. FIG7 provides a visualization of these techniques. We follow the main text by denoting a set of molecules as M, and refer to the atoms in these molecules (which are represented as nodes in a graph) as A.We start with Gated Graph Neural Networks (GGNNs) , which we use for finding node embeddings. We denote these functions as h A: M → R |A|×d, where we will refer to the output as the node embedding matrix, H M ∈ R |A|×d. Each row of this node embedding matrix represents the embedding of a particular atom; the rows are ordered by atom-mapped number, a unique number assigned to each atom in a SMILES string. The GGNN form these node embeddings through a recurrent operation on messages, m v, with v ∈ A, so that there is one message associated with each node. At the first time step these messages, m Table 4. GGNNs then update these messages in a recursive nature: DISPLAYFORM0 Where GRU is a Gated Recurrent Unit BID1, the functions N e1 (v), N e2 (v), N e3 (v) index the nodes connected by single, double and triple bonds to node v respectively and f single (·), f double (·) and f triple (·) are linear transformations with learnable parameters. This process continues for S steps (where we choose S = 4). In our implementation, messages and the hidden layer of the GRU have a dimensionality of 101, which is the same as the dimension of the raw atom features. The node embeddings are set as the final message belonging to a node, so that indexing a row of the node embeddings matrix, H M, gives a transpose of the final message vector, ie ). These networks consist of a series of iterative steps where the embeddings for each node are updated using the node's previous embedding and a message from its neighbors. Graph embeddings are q-dimensional vectors, representing a set of nodes, which could for instance be all the nodes in a particular graph . They are formed using a function on the weighted sum of node embeddings. Table 4: Atom features we use as input to the GGNN. These are calculated using RDKit. DISPLAYFORM1 Atom type 72 possible elements in total, one hot Degree One hot Explicit Valence One hot (0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14 Although we found choosing the first entry in the electron path is often the most challenging decision, and greatly benefits from reagent information, we also considered a version of ELECTRO where we fed in the reagent information at every step. In other words, the modules for f add (·) and f remove (·) also received the reagent embeddings calculated by r reagent (·) concatenated onto their inputs. On the mechanism prediction task (Table 2) this gets a slightly improved top-1 accuracy of 78.4% (77.8% before) but a similar top-5 accuracy of 94.6% (94.7% before). On the reaction product prediction task (Table 3) we get 87.5%, 94.4% and 96.0% top-1, 3 and 5 accuracies (87.0%, 94.5% and 95.9% before). The tradeoff is this model is somewhat more complicated and requires a greater number of parameters. We train everything using Adam and an initial learning rate of 0.0001, which we decay after 5 and 9 epochs by a factor of 0.1. We train for a total of 10 epochs. For training we use reaction minibatch sizes of one, although these can consist of multiple intermediate graphs. This section provides further details on how we extract reactions with linear electron flow topology, complementing FIG3 in the main text. We start from the USPTO SMILES reaction string and bond changes and from this wish to find the electron path. The first step is to look at the bond changes present in a reaction. Each atom on the ends of the path will be involved in exactly one bond change; the atoms in the middle will be involved in two. We can then line up bond change pairs so that neighboring pairs have one atom in common, with this ordering forming a path. For instance, given the pairs "11-13, 14-10, 10-13" we form the unordered path "14-10, 10-13, 13-11". If we are unable to form such a path, for instance due to two paths being present as a of multiple reaction stages, then we discard the reaction. For training our model we want to find the ordering of our path, so that we know in which direction the electrons flow. To do this we examine the changes of the properties of the atoms at the two ends of our path. In particular, we look at changes in charge and attached implicit hydrogen counts. The gain of negative charge (or analogously the gain of hydrogen as H + ions without changing charge) indicates that electrons have arrived at this atom, implying that this is the end of the path; vice-versa for the start of the path. However, sometimes the difference is not available in the USPTO data, as unfortunately only major products are recorded, and so details of what happens to some of the reactant molecules' atoms may be missing. In these cases we fall back to using an element's electronegativity to estimate the direction of our path, with more electronegative atoms attracting electrons towards them and so being at the end of the path. The next step of filtering checks that the path alternates between add steps (+1) and remove steps (-1). This is done by analyzing and comparing the bond changes on the path in the reactant and product molecules. Reactions that involve greater than one change (for instance going from no bond between two atoms in the reactants to a double bond between the two in the products) can indicate multi-step reactions with identical paths, and so are discarded. Finally, as a last sanity check, we use RDKit to produce all the intermediate and final products induced by our path acting on the reactants, to confirm that the final product that is produced by our extracted electron path is consistent with the major product SMILES in the USPTO dataset. At predict time, as discussed in the main text, we use beam search to find high probable chemicallyvalid paths from our model. Further details are given in Algorithm 2. For ELECTRO this operation takes 0.337s per reaction, although we do not parallelize the molecule manipulation across the different beams, and so the majority of this time (0.193s) is used within RDKit to make intermediate We filter down to the top K most promising actions. for all (ρ, p path) ∈ B t−1 do M ρ = calc_intermediate_mol(M 0, ρ) p c = calc_prob_continue(M ρ) 17:P =P ∪ {(ρ, p path + log(1 − p c))} for all v ∈ A do 19: DISPLAYFORM0 New proposed path is concatenation of old path with new node. v t−1 = last element of ρ 21:B =B ∪ {(ρ, p path + log p c + log calc_prob_action(v, M ρ, v t−1, F remove))} F remove = F remove + 1 mod 2.If on add step change to remove and vice versa. 26: end for 27: 28:P = sort_on_prob(P) Output: Valid completed paths and their respective probabilities, sorted by the latter,P molecules and extract their features. At test time we take advantage of the embarrassingly parallel nature of the task to parallelize across test inputs. To compute the log likelihood of a reaction (with access to intermediate steps) it takes ELECTRO 0.007s. This section provides further examples of the paths predicted by our model. In FIG11, we wish to show how the model has learnt chemical trends by testing it on textbook reactions. In FIG0 we show further examples taken from the USPTO dataset. Figure 9: Additional typical selectivity examples: Here, the expected product is shown on the right. The blue arrows indicate the top ranked paths from our model, the red arrows indicate other possibly competing but incorrect steps, which the model does not predict to be of high probability. In all cases, our model predicted the correct products. In b) and c), our model correctly recovers the regioselectivity expected in electrophilic aromatic substitutions.

represent this text to find a much much longer article about it with say hundreds-thousands of words

E.g.:
We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models == Comparing the inferences of diverse candidate models is an essential part of model checking and escaping local optima. To enable efficient comparison, we introduce an amortized variational inference framework that can perform fast and reliable posterior estimation across models of the same architecture. Our Any Parameter Encoder (APE) extends the encoder neural network common in amortized inference to take both a data feature vector and a model parameter vector as input. APE thus reduces posterior inference across unseen data and models to a single forward pass. In experiments comparing candidate topic models for synthetic data and product reviews, our Any Parameter Encoder yields comparable posteriors to more expensive methods in far less time, especially when the encoder architecture is designed in model-aware fashion. We consider the problem of approximate Bayesian inference for latent variable models, such as topic models , embedding models , and dynamical systems models . An important step in using such probabilistic models to extract insight from large datasets is model checking and comparison. While many types of comparison are possible , we focus on a problem that we call within-model comparison. Given several candidate parameter vectors θ 1, θ 2,..., all from the same space Θ ⊆ R D, our goal is to efficiently determine which parameter θ m is best at explaining a given dataset of N examples {x n} N n=1. Multiple ways exist to rank candidate parameters, including performance on heldout data or human-in-the-loop inspection. A principled choice is to select the parameter that maximizes the data's marginal likelihood: N n=1 log p(x n |θ m). For our latent variable models of interest, computing this likelihood requires marginalizing over a hidden variable h n: p(x n |θ m) = p(x n |h n, θ m)p(h n |θ m)dh n. This integral is challenging even for a single example n and model m. One promising solution is variational inference (VI). Using VI, we can estimate an approximate posterior q(h n |x n, θ m) over hidden variables. Approximate posteriors q can be used to compute lower bounds on marginal likelihood, and can also be helpful for human inspection of model insights and uncertainties. However, it is expensive to estimate a separate q at each example n and model m. In this paper, we develop new VI tools 1 that enable rapid-yet-effective within-model comparisons for large datasets. The need for within-model comparison (and our methods) is present in many practical modeling tasks. Here we discuss two possible scenarios, with some details specialized to our intended topic modeling applications . First, in human-in-the-loop scenarios, a domain expert may inspect some estimated parameter θ and then suggest an alternative parameter θ that improves interpretability. In topic modeling, this may mean removing "intruder words" to make topics more coherent . Second, in automated parameter learning scenarios, many algorithms propose data-driven transformations of the current solution θ into a new candidate θ, in order to escape the local optima common in non-convex optimization objectives for latent variable models , Examples include split-merge proposal moves or evolutionary algorithms . Across both these scenarios, new candidates θ arise repeatedly over time, and estimating approximate posteriors for each is essential to assess fitness yet expensive to perform for large datasets. Our contribution is the Any Parameter Encoder (APE), which amortizes posterior inference across models θ m and data x n. We are inspired by efforts to scale a single model to large datasets by using an encoder neural network (NN) to amortize posterior inference across data examples . Our key idea is that to additionally generalize across models, we feed model parameter vector θ m and data feature vector x n as input to the encoder. APE is applicable to any model with continuous hidden variables for which amortized inference is possible via the reparameterization trick. We consider a general family of probabilistic models that use parameter vector θ m to generate a dataset of N continuous hidden variables h n and observations x n via a factorized distribution: N n=1 p(h n |θ m)p(x n |h n, θ m). Our goal is fast-yet-accurate estimation of each example's local posterior p(h n |x n, θ m) for a range of model parameters θ 1, θ 2,... ∈ Θ. Topic Models. As a sample application, we focus on the Logistic Normal topic model from. Given known vocabulary size V, we observe N documents represented by count vectors x n (vector of size V counting the types of all T n words in document n). We model each x n as a mixture of K possible topics. Let hidden variable h nk be the probability that a word in document n is produced by topic k. Thus, h n ∈ ∆ K is a non-negative vector of size K that sums to one. We model h n with a Logistic Normal prior, with mean and covariance set to be similar to a sparse Dirichlet(0.01) prior for interpretability. Given h n, we model the observed word-count vector for document n with a Multinomial likelihood:. This is a document-specific mixture of topics, where each topic k is defined by a word probability vector θ k ∈ ∆ V. Our parameter of interest is the topic-word probability vector θ = {θ k} K k=1. VI Approximations for the Single Example Posterior. While the true posterior p(h n |x n, θ m) is usually intractable, we use variational inference (VI) to approximate it. We choose a simpler density q(h n |λ n) and optimize parameter λ n to minimize KL divergence from the true posterior. Inference reduces to the well-known evidence lower bound (ELBO) optimization problem given example x n and model θ m: Inference: Given several parameters of interest, we can perform model comparison by solving the above optimization problem separately for each θ m. However, this is expensive. Solving Eq. for a model θ m requires dozens of iterative updates of gradient ascent for each example. VI Amortized across Data Examples. and have sped up inference by setting per-example variational parameters λ n to the output of an encoder neural network (NN) instead of an iterative optimization procedure. The "Standard" encoder, with weights parameters φ, takes input data x n and produces variational parameters λ NN φ (x n). Inference for example n reduces to one fast forward pass: λ n ← λ NN φ (x n). While encoders often produce λ n with worse ELBO scores than optimal solutions to Eq. , they are preferred for their speed. However, for our model comparison goals the standard encoder is expensive, because for each parameter θ m of interest we must train separate specialized NN weights φ m. Contribution: VI Amortized over Model Parameters. Our goal is to enable rapid estimation of posteriors p(h n |x n, θ m) for many possible parameters θ 1, θ 2,... ∈ Θ (not all known in advance). We thus consider a family of approximating densities q that explicitly conditions on both a given data vector x n and the query parameter vector θ m. Again, we use a neural network to transform these inputs into the variational parameters, λ n ← λ NN φ (x n, θ m). We call this the Any Parameter Encoder. Unlike the earlier Standard Encoder, which trains φ for one specific θ, our approach can directly generalize to many θ. Encoder Architecture Design for Topic Models. Given the difficulty of posterior inference even for a single parameter θ, developing an effective Any Parameter Encoder requires careful selection of a NN architecture that can transform its two inputs, data x n and model θ, to produce accurate approximate posteriors. Following previous work , we use multi-layer perceptrons. We further suggest that an architecture designed to capture structure in the generative model should improve further. Our baseline "naive" architecture defines the input of the neural net as simply the concatenation of vector x n and vector θ. While simple, we suggest this will be difficult to train effectively given the size of the input ((K + 1)V for the topic model) and lack of inductive bias to prioritize the model's needed interactions between entries of x n and θ. As an improvement, we consider a model-aware encoder architecture. Our design is motivated by a view of posterior inference as roughly moment-matching when data is plentiful. For our topic model, each document's Multinomial likelihood has a mean vector equal to k h nk θ k = θh n, writing θ as a V × K matrix. This mean vector should be (roughly) equal to the observed word-frequency vector 1 Tn x n. If µ n is the mean of q(h n) and used as a plug-in estimate for h n, then we want to satisfy 1 Tn x n ≈ θµ n. Solving for µ n via least squares, we get µ n ≈ 1 Tn (θ T θ) −1 θ T x n, which we might simplify to a non-linear function of θ T x n. Thus, we suggest using the following model-aware encoder architecture: This model-aware architecture has encoder input dimension K, which is much smaller than (K + 1)V for the naive approach (and thus hopefully easier to train). Furthermore, this should provide desirable inductive bias to produce useful mean and covariance estimates. We emphasize that this design is specialized to the topic model, and further work is needed to develop model-aware architecture design strategies for general latent variable models. Training the Encoder. Training our encoder parameters φ requires an available set of M parameter vectors {θ m} M m=1 of interest. We choose these to be representative of the subset of Θ we wish to generalize well to. We then maximize ELBO across all M models: We use stochastic gradient ascent to solve for φ, using the reparameterization trick to estimate gradients for a minibatch of examples and models at each step. We can interpret this objective as an expectation over samples θ m from a target distribution over parameters. We compare our proposed Any Parameter Encoder (APE) to several other inference methods on two topic modeling tasks. For all VI methods, we choose q to be a Logistic Normal parameterized by a mean and a diagonal covariance. The appendix has complete details. APE. We consider both naive and model-aware encoder architectures described above. Both use MLPs with 2 layers with 100 units per hidden layer, selected via grid search. Baselines. We consider three baselines implemented in Pyro and PyTorch . First, Variational Inference (VI) uses gradient ascent to optimize Eq.. Second, we use Standard encoder VAEs for topic models . This encoder is specialized to a single parameter θ, with architecture size selected via grid search (similar to APE). Finally, we run Pyro's off-the-shelf implementation of Hamiltonian Monte Carlo with the No U-Turn Sampler (NUTS) , though we expect specialized implementations to be more performant. Synthetic Data Experiments. We consider a V = 100 vocabulary dataset inspired by the "toy bars" of. Using K = 20 true topics θ *, we sample Error from VI ELBO Product Reviews Figure 1: Left: ELBO vs. elapsed time for VI on a test set of 300 document-θ m combinations on synthetic data. We show a randomly-initialized run (black) and a warm-started run (green) initialized via our Any-Parameter Encoder (red "X"). The randomly-initialized VI would require over 400 milliseconds (vertical line) to reach the quality our APE achieved in <20 ms. Right: Kernel Density Estimation of absolute difference between encoder ELBO and VI ELBO across different topics. APE (red) are closer to VI (i.e. less error). a 500-document dataset. We consider M = 50, 000 possible model parameters {θ m} M m=1, sampled from a symmetric, sparse Dirichlet prior over the vocabulary. Typical θ m look unlike the true topics θ *, as shown in the supplement, so inference must handle diversity well. We train our APE on 25 million possible document-θ pairs for two epochs, then evaluate on unseen document-θ pairs drawn from the same generative process. Product Reviews. We model 6,343 text documents of consumer product reviews . We use the V = 3000 most frequent vocabulary terms and K = 30 topics. We generate training topics in the same way as in the synthetic data experiments, and we evaluate on test topics found via Gibbs sampling with several separately initialized runs. Results: Encoder Design. Results comparing naive and model-aware encoder architectures are in Table 1. Our proposed model-aware input layer yields better heldout likelihoods than the naive alternative, which we suggest is due to its more effective inductive bias. Results: Quality-vs-Time Tradeoff. Comparing across Table 1 and Fig. 1, we see that while the Standard Encoder understandably fails to generalize across models, our Any Parameter Encoder achieves quality close to non-amortized VI and NUTS with a speed up factor of over 100-1000x. APE can also provide a useful warm start initialization to VI. Results: Agreement in model comparison. Motivated by the need to rapidly assess proposal moves that escape local optima, we gather 10 different models and measure whether each encoder's ranking of a pair θ, θ on the test set agrees with VI's ranking. Table 1 shows that APE agrees with VI in 75% of 45 cases in the real data scenario, while Standard Encoder agrees just 29% of the time. This suggests APE may be trustworthy for accept/reject decisions, though further work is needed to improve this number further. Across two datasets and many model parameters, our Any Parameter Encoder produces posterior approximations that are nearly as good as expensive VI, but over 100x faster. Future opportunities include simultaneous training of parameters and encoders, and handling Bayesian nonparametric models where θ changes size during training (n)) For encoder methods, the parameters {µ n, log σ 2 n} are the output of a shared encoder NN. For VI, these are free parameters of the optimization problem. Variational Inference (VI). We perform using gradient ascent to maximize the objective in Eq., learning a per-example mean and variance variational parameter. We run gradient updates until our moving average loss (window of 10 steps) has improved by less than 0.001% of its previous value. For our VI runs from random initializations, we use the Adam optimizer with an initial learning rate of.01, decaying the rate by 50% every 5000 steps. For our warm-started runs, we use an initial learning rate of 0.0005. In practice, we ran VI multiple times with different learning rate parameters and took the best one. Table 1 only reports the time to run the best setting, not the total time which includes various restarts. Standard encoder. We use a standard encoder that closely matches the VAE for topic models in. The only architectural difference is the addition of a temperature parameter on the µ n vector before applying the softmax to ensure the means lie on the simplex. We found that the additional parameter sped up training by allowing the peakiness of the posterior to be directly tuned by a single parameter. We use a feedforward encoder with two hidden layers, each 100 units. We chose the architecture via hyperparameter sweeps. The total number of trainable parameters in the model is 24,721 on the synthetic data and 316,781 on the real data; this is compared to 16,721 and 19,781 parameters for model-aware APE. NUTS. For the Hamiltonian Monte Carlo (HMC) with the No U-Turn Sampler (NUTS) , we use a step size of 1 adapted during the warmup phase using Dual Averaging scheme. Upon inspection, we find that the method's slightly lower posterior predictive log likelihood relative to VI is due to its wider posteriors. We also find that the Pyro implementation is (understandably) quite slow and consequently warm-start the NUTS sampler using VI to encourage rapid mixing. We are aware that there exist faster, more specialized implementations, but we decided to keep our tooling consistent for scientific purposes. We generate a set of different models {θ 0, θ 1, ...θ M} from a symmetric Dirichlet prior with α = 0.1. We train our Any-Parameter Encoder in random batches of document-topic combinations. With 500 documents and 50,000 topics (i.e. D = 500, M = 50, 000), we have 25 million combinations in total. The topics used to generate the synthetic data represent "toy bars", inspired by . See Figure 2 for a visualization. We use this same toy bars-biased prior to generate all our topics in the holdout set, though the order of the topics is random. See For training both APE and the Standard encoder on the synthetic data, we use Adam with an exponential decay learning schedule, a starting learning rate of 0.01, and a decay rate of.8 every 50,000 steps. We find that this schedule tends to be fairly robust; these hyperparameters were used for both APE and the Standard encoder on both the synthetic and real data. We chose our initial learning rate via a learning rate finder posed in , and we train for 2 epochs with a batch size of 100. We train our standard VAE encoder on a single model with parameters θ drawn randomly from a symmetric Dirichlet prior with α = 0.1. To train the standard encoder, we pass in our model of interest to the decoder, holding its weights fixed as we perform stochastic backpropagation to update the encoder weights. The same thing happens for APE, though the same topics are additionally included as part of the input into the encoder. We develop VAEs where the encoder takes a model parameter vector as additional input, so we can do rapid inference for many models

A generative model for reaction prediction that learns the mechanistic electron steps of a reaction directly from raw reaction data.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent text

Question: When machine learning models are used for high-stakes decisions, they should predict accurately, fairly, and responsibly. To fulfill these three requirements, a model must be able to output a reject option (i.e. say "``I Don't Know") when it is not qualified to make a prediction. In this work, we propose learning to defer, a method by which a model can defer judgment to a downstream decision-maker such as a human user. We show that learning to defer generalizes the rejection learning framework in two ways: by considering the effect of other agents in the decision-making process, and by allowing for optimization of complex objectives. We propose a learning algorithm which accounts for potential biases held by decision-makerslater in a pipeline. Experiments on real-world datasets demonstrate that learning to defer can make a model not only more accurate but also less biased. Even when operated by highly biased users, we show that deferring models can still greatly improve the fairness of the entire pipeline. Recent machine learning advances have increased our reliance on learned automated systems in complex, high-stakes domains such as loan approvals BID6, medical diagnosis BID12, and criminal justice BID22. This growing use of automated decisionmaking has raised questions about the obligations of classification systems. In many high-stakes situations, machine learning systems should satisfy (at least) three objectives: predict accurately (predictions should be broadly effective indicators of ground truth), predict fairly (predictions should be unbiased with respect to different types of input), and predict responsibly (predictions should not be made if the model cannot confidently satisfy the previous two objectives).Given these requirements, we propose learning to defer. When deferring, the algorithm does not output a prediction; rather it says "I Don't Know" (IDK), indicating it has insufficient information to make a responsible prediction, and that a more qualified external decision-maker (DM) is required. For example, in medical diagnosis, the deferred cases would lead to more medical tests, and a second expert opinion. Learning to defer extends the common rejection learning framework (; BID9 in two ways. Firstly, it considers the expected output of the DM on each example, more accurately optimizing the output of the joint DM-model system. Furthermore, it can be used with a variety of training objectives, whereas most rejection learning research focuses solely on classification accuracy. We believe that algorithms that can defer, i.e., yield to more informed decision-makers when they cannot predict responsibly, are an essential component to accountable and reliable automated systems. In this work, we show that the standard rejection learning paradigm (learning to punt) is inadequate, if these models are intended to work as part of a larger system. We propose an alternative decision making framework (learning to defer) to learn and evaluate these models. We find that embedding a deferring model in a pipeline can improve the accuracy and fairness of the pipeline as a whole, particularly if the model has insight into decision makers later in the pipeline. We simulate such a pipeline where our model can defer judgment to a better-informed decision maker, echoing real-world situations where downstream decision makers have more resources or information. We propose different formulations of these models along with a learning algorithm for training a model that can work optimally with such a decision-maker. Our experimental on two real-world datasets, from the legal and health domains, show that this algorithm learns models which, through deferring, can work with users to make fairer, more responsible decisions. Notions of Fairness. One of the most challenging aspect of machine learning approaches to fairness is formulating an operational definition. Several works have focused on the goal of treating similar people similarly (individual fairness) and the ing necessity of fair-awareness -showing that it may be essential to give the algorithm knowledge of the sensitive variable BID10; BID11.Some definitions of fairness center around statistical parity BID19 BID20, calibration BID14 BID14 or disparate impact/equalized odds BID7 BID16 BID23 BID13. It has been shown that disparate impact and calibration cannot be simultaneously satisfied BID7 BID23. BID16 present the related notion of "equal opportunity". In a subsequent paper, argue that in practice fairness criteria should be part of the learning algorithm, not post-hoc. BID13 and BID3 develop and implement learning algorithms that integrate equalized odds into learning via regularization. Incorporating IDK. While we are the first to propose learning to defer, some works have examined the "I don't know" (IDK) option (cf. rejection learning, see BID9 for a thorough survey), beginning with Chow (1957; BID8 who studies the tradeoff between error-rate and rejection rate. BID9 develop a framework for integrating IDK directly into learning. KWIK (Knows-What-It-Knows) learning is proposed in BID25 as a theoretical framework. BID0 discuss the difficulty of a model learning what it doesn't know (particularly rare cases), and analyzes how human users can audit such models. propose a cascading model, which can be learned from end-to-end; higher levels can say IDK and pass the decision on to lower levels. Similarly, BID24; BID9; BID2 provide algorithms for saying IDK in classification and provides a statistical overview of the problem, including both "don't know" and "outlier" options. However, none of these works look at the fairness impact of this procedure. A few papers have addressed topics related to both fairness and IDK. BID5 describe fair sequential decision making but do not have an IDK concept, nor do they provide a learning procedure. In BID18, the authors show theoretical connections between KWIK-learning and a proposed method for fair bandit learning. BID13 discuss fairness that can arise out of a mixture of classifiers. However, they do not provide a learning procedure, nor do they address sequential decision making, which we believe is of great practical importance. propose "safety reserves" and "safe fail" options which combine learning with rejection and fairness/safety, but do not suggest how such options may be learned or analyze the effect of a larger decision-making framework. AI Safety. Finally, our work also touches on aspects of the AI safety literature -we provide a method by which a machine learns to work optimally with a human. This is conceptually similar to work such as; , which discuss the situations in which a robot should be compliant/cooperative with a human. The idea of a machine and human jointly producing a fair classifier also relates to BID15, which describes algorithms for machines to align with human values. Previous works in rejection learning (see Sec. 2) have proposed models that can choose to not classify (say IDK/reject). In these works, the standard method is to optimize the accuracy-IDK tradeoff: how much can a model improve its accuracy on the cases it does classify by saying IDK to some cases?We find this paradigm inadequate. In many of the high-stakes applications this type of work is aimed at, an IDK is not the end of the story. Rather, a decision must be made eventually on every example, whether the model chooses to classify it or not. When the model predicts, the system outputs the model's prediction; when the model says IDK, the system outputs the decision-maker's (DM's) prediction. Rejection learning considers the "IDK Model" to be the system output. Say our model is trained to detect melanoma, and when it says IDK, a human doctor can run an extra suite of medical tests. The model learns that it is very inaccurate at detecting amelanocytic (non-pigmented) melanoma, and says IDK if this might be the case. However, suppose that the doctor is even less accurate at detecting amelanocytic melanoma than the model is. Then, we may prefer the model to make a prediction despite its uncertainty. Conversely, if there are some patients that the doctor knows well, then they may have a more informed, nuanced opinion than the model. Then, we may prefer the model to say IDK more frequently relative to its internal uncertainty. Saying IDK on the wrong examples can also have fairness consequences. If the doctor's decisions bias against a certain group, then it is probably preferable for our model (if it is less biased) to defer less frequently on the cases of that group. In short, the model we train is part of a larger pipeline, and we should be training and evaluating the performance of the pipeline with this model included, rather than solely focusing on the model itself. To enable this, we define a general two-step framework for decision-making FIG0. The first step is an automated model whose parameters we want to learn. The second step is some external decision maker (DM) which we do not have control over; this could be a human user or a more resourceintensive tool. The decision-making flow is done as a cascade, where the first-step model can either predict (positive/negative) or say IDK. If it predicts, the DM trusts the model completely, and outputs that prediction. However, if it says IDK, the DM makes its own decision. We assume that the DM is more powerful than the model we train -reflecting a number of practical scenarios where decision makers later in the chain have more resources for efficiency, security, or contextual reasons.. In Appendix A, we prove that learning to defer is equivalent to rejection learning for a broad class of loss functions, including classification error, if the DM is an oracle. Since our DM is rarely an oracle, learning to defer therefore yields a modeling advantage over rejection learning. Furthermore, learning to defer allows us to optimize a variety of objectives L(Y, Y sys), whereas most rejection learning research focuses on classification error. In the rest of this work, we show how to learn fair IDK models in this framework. The paper proceeds as follows: in Sec. 4 we give some on the fairness setup; in Sec. 5 we describe two methods of learning IDK models and how we may learn them in a fair way; and in Sec. 6 we give a learning algorithm for optimizing models to succeed in this framework. In Sec. 7 we show on two real-world datasets. In fair binary classification, we have data X, labels Y, predictionsŶ, and sensitive attribute A, assuming for simplicity that Y,Ŷ, A ∈ {0, 1}. In this work we assume that A is known (fair-aware) and that it is a single binary attribute (e.g., gender, race, age, etc.); extensions to more general settings are straightforward. The aim is twofold: firstly, that the classifier is accurate i.e., Y i =Ŷ i; and secondly, that the classifier is fair with respect to A i.e.,Ŷ does not discriminate unfairly against examples with a particular value of A. Classifiers with fairness constraints provably achieve worse error rates (cf. Chouldechova FORMULA0 ; BID23). We thus define a loss function which trades off between these two objectives, relaxing the hard constraint proposed by models like BID16 and yielding a regularizer, similar to BID20 BID3. We use disparate impact (DI) as our fairness metric BID7, as it is becoming widely used and also forms the legal basis for discrimination judgements in the U.S. BID1. Here we define a continuous relaxation of DI, using probabilistic output DISPLAYFORM0 Note that constraining DI = 0 is equivalent to equalized odds BID16. If we constrain p ∈ {0, 1}, we say we are using hard thresholds; allowing p ∈ is soft thresholds. We include a hyperparameter α to balance accuracy and fairness; there is no "correct" way to weight these. When we learn such a model, p is a function of X parametrized by θ. Our regularized fair loss function (L F air, or L F) combines cross-entropy for accuracy with this fairness metric: DISPLAYFORM1 5 SAYING IDK: LEARNING TO PUNTWe now discuss two model formulations that can output IDK: ordinal regression, and neural networks with weight uncertainty. Both of these models build on binary classifiers by allowing them to express some kind of uncertainty. In this section, we discuss these IDK models and how to train them to be fair; in the following section we address how to train them to take into account the downstream decision-maker. We extend binary classifiers to include a third option, yielding a model that can classify examples as "positive", "negative" or "IDK". This allows the model to punt, i.e., to output IDK when it prefers not to commit to a positive or negative prediction. We base our IDK models on ordinal regression with three categories (positive, IDK, and negative). These models involve learning two thresholds τ = (t 0, t 1) (see FIG8). We can train with either hard or soft thresholds. If soft, each threshold t i, i ∈ {0, 1} is associated with a sigmoid function σ i, where DISPLAYFORM0 1+e −x. These thresholds yield an ordinal regression, which produces three values for every score x: DISPLAYFORM1 Using hard thresholds simply restricts P, I, N ∈ {0, 1} 3 (one-hot vector). We can also calculate a score p(x) ∈, which we interpret as the model's prediction disregarding uncertainty. These values are: DISPLAYFORM2 P represents the model's bet that x is "positive", N the bet that x is "negative", and I is the model hedging its bets; this rises with uncertainty. Note that p is minimized at P = N; this is also where I is maximized. At test time, we use the thresholds to partition the examples. On each example, the model outputs a score x ∈ R (the logit for the ordinal regression), and a prediction p. If t 0 < x < t 1, then we replace the model's prediction p with IDK. If x ≤ t 0 or x ≥ t 1, we leave p as is. To encourage fairness, we can learn a separate set of thresholds for each group: (t 0,A=0, t 1,A=0) and (t 0,A=1, t 1,A=1); then apply the appropriate set of thresholds to each example. We can also regularize IDK classifiers for fairness. When training this model, P, I, and N are parametrized functions, with model parameters θ and thresholds τ. Using soft thresholds, the regularized loss DISPLAYFORM3 Note that we add a term penalizing I(X), to prevent the trivial solution of always outputting IDK. In addition, we regularize the disparate impact for P (X) and N (X) separately. This was not necessary in the binary case, since these two probabilities would have always summed to 1. We learn soft thresholds end-to-end; for hard thresholds we use a post-hoc thresholding scheme (see Appendix D). We can also take a Bayesian approach to uncertainty by learning a distribution over the weights of a neural network BID4. In this method, we use variational inference to approximate the posterior distribution of the model weights given the data. When sampling from this distribution, we can obtain an estimate of the uncertainty. If sampling several times yields widely varying , we can state the model is uncertain on that example.. The reciprocal of this (π = 1/S) allows high values to be more uncertain, while π = σ(log(1/S)) (where σ is the logistic function) yields uncertainty values in a range. At test time, the system can threshold this uncertainty; any example with uncertainty beyond a threshold is punted to the DM.We can regularize this Bayesian model to improve fairness as in the standard binary classifier. In the likelihood term for the variational inference, we can simply add the disparate impact regularizer (Eq. 1), making solutions of low disparate impact more likely. With weights w and variational parameters θ, our variational lower bound L V is then: DISPLAYFORM0 6 LEARNING TO DEFER IDK models come with a consequence: when a model punts, the prediction is made instead by some external, possibly biased decision maker (DM) e.g., a human expert. In this work we assume that this DM is possibly biased, but is more accurate than the model; perhaps the DM is a judge with detailed information on repeat offenders, and with more information about the defendant than the model has, or a doctor who can conduct a suite of complex medical tests. Here we introduce a distinction between learning to punt and learning to defer. In learning to punt, the goal is absolute: the model's aim is to identify the examples where it has a low chance of being correct. In learning to defer, the model has some information about the DM and takes this into account in its IDK decisions. Hence the goal is relative: the model's aim is to identify the examples where the DM's chance of being correct is much higher than its own. If the model punts mostly on cases where the DM is very inaccurate or unfair, then the joint predictions made by the model-DM pair may be poor, even if the model's own predictions are good. We can think of learning to punt as DM-unaware learning, and learning to defer as DM-aware learning. To conduct DM-aware learning, we can modify the model presented in Section 5 to take an extra input: the DM's scores on every case in the training set. The model is then optimized for some loss function L(Y, A, X); for our purposes, this loss will be a combination of accuracy and fairness. We propose the following general method, drawing inspiration from mixture-of-experts BID17. We introduce a mixing parameter π for each example x, which is the probability of deferral; that is, the probability that the DM makes the final decision on the example x, rather than the model. Then, 1 − π is the probability that the model's decision becomes the final output of the system. Let s ∼ Ber(π). Our mixing parameter π corresponds to our model's uncertainty estimate -I(x) in ordinal regression, σ(log( 1 S)) in the Bayesian neural network. Let p be the first stage model's predictions andỸ be the DM's predictions. We can express the joint system's predictionsp aŝ DISPLAYFORM1 In learning this model, we can parametrize p and π by θ = (θ p, θ π), which may be shared parameters. We can then define our loss function (L Def er, or L D) as an expectation over the Bernoulli variables DISPLAYFORM2 We call this learning to defer. When optimizing this function, the model learns to recognize when there is relevant information that is not present in the data it has been given by comparing its own predictions to the DM's predictions. Full details of how this loss function is calculated are provided in Appendix E. Experimental Setup. To evaluate our models, we measure three quantities: classification error, disparate impact, and deferral rate. We train an independent model to simulate predictions made by an external DM. This DM is trained on a version of the dataset with extra attributes, simulating the extra knowledge/accuracy that the DM may have. However, the DM is not trained to be fair. When our model outputs IDK we take the output of the DM instead (see FIG0).Datasets and Experiment Details. We show on two datasets: the COMPAS dataset BID22, where we predict a defendant's recidivism (committing a crime while on bail) without discriminating by race, and the Heritage Health dataset, where we predict a patient's Charlson Index (a comorbidity indicator related to likelihood of death) without discriminating by age. For COMPAS, we give the DM the ground truth for a defendant's violent recidivism; for Health, we give the DM the patient's primary condition group. Appendix C contains additional details on both datasets. We trained all models using a one-hidden-layer fully connected neural network with a logistic or ordinal regression on the output, where appropriate. We used 5 sigmoid hidden units for COMPAS and 20 sigmoid hidden units for Health. We used ADAM BID21 for gradient descent. We split the training data into 80% training, 20% validation, and stopped training after 50 consecutive epochs without achieving a new minimum loss on the validation set. In the ordinal regression model, we trained with soft thresholds since we needed the model to be differentiable end to end. In the post-hoc model, we searched threshold space in a manner which did not require differentiability, so we used hard thresholds. This is equivalent to an ordinal regression which produces one-hot vectors i.e. P, I, N ∈ {0, 1}. See Appendices D and E for additional details on both of these cases. Displaying Results. Each model contains hyperparameters, such as the coefficients (α, γ) for training and/or post-hoc optimization. We show the of several models, with various hyperparameter settings, to illustrate how they mediate the tradeoff of accuracy and fairness. Each plotted point is a median of 5 runs at a given hyperparameter setting. We only show points on the Pareto front of Bottom left hand corner is optimal. The purple star is a baseline model, trained only to optimize accuracy; green squares is a model also optimizing fairness; the red diamond optimizes accuracy while allowing IDK; and blue circles are the full model with all three terms. Yellow star shows the second stage model DM alone. Each point is the median of 5 runs on the test set at a given hyperparameter setting. FIG1; the new here show the performance of a DM-aware model (defer-fair), depicted by the green triangles. Of particular note is the improvement of this model relative to the punting model (the blue circles).the , i.e., those for which no other point had both better error and DI. Finally, all are calculated on a held-out test set. In FIG1, we compare punting models to binary models, with and without fairness regularization. These IDK models have not learned to defer, i.e. they did not receive access to the DM scores during training (see FIG2). The show that, on both datasets, the IDK models achieve a stronger combination of fairness and accuracy than the binary models. Graphically, we observe this by noting that the line of points representing IDK model are closer to the lower left hand corner of the plot than the line of points representing binary model . Some of this improvement is driven by the extra accuracy in the DM. However, we note that the model-DM combination achieves a more effective accuracy-fairness tradeoff than any of the three baselines: the accurate but unfair DM; the fair but inaccurate binary model with DI regularization; and the unfair and inaccurate unregularized binary model. Learning to punt can therefore be a valuable tool for anyone who designs or oversees a many-part system -a simple first stage capable of expressing uncertainty can improve the fairness of a more accurate DM. FIG2 demonstrates a clear improvement over the punting models (DM-unaware, Sec. 5). If we have access to examples of past DM behavior, learning to defer provides an effective way to improve the fairness of the entire system. For insight here, we can inspect the different roles IDK plays in their respective loss functions. In the DM-unaware IDK model, punting is penalized at a constant rate, determined by γ. However, in the DM-aware model, deferring penalized in a way which is dependent on the output of the DM on that example. We can consider the unaware model to be optimizing the expected DM-aware loss function for a DM with constant expected loss on each examples, such as an oracle (see Appendix A). Then, we can see any improvement by the DM-aware model as effective identification of the examples on which the expected loss of the DM is unusually high; in other words, identifying the inconsistencies or biases of the DM. One advantage of deferring is that it can account for specific characteristics of a DM. To test this, we considered the case of a DM which is extremely biased FIG3. We find that the advantage of a deferring model holds in this case, as it compensates for the DM's extreme bias. We can further analyze where the model chooses to defer. Recall that the DM is given extra information; in this case the violent recidivism of the defendant (true for about 7% of the dataset), which is difficult to predict from the other attributes. FIG4 compares the IDKs predicted by a punting model and a deferring model -split by group (race) on the left, and by group and violent recidivism on the right. Both models achieved roughly 27% error; the deferring model had 2% DI and the punting model had 4%. On the left, we see that the deferring model says IDK to more black people (the pink bar). On the right however, we see that the deferring model says IDK to a higher percentage of violently recidivating non-black people, and a lower percentage of violently recidivating black people. This improves DI -the extra information the DM has received is more fully used on the non-protected group. The punting model cannot adjust this way; the deferring model can, since it receives noisy access to this information through the DM scores in training. In this work, we propose the idea of learning to defer. We propose a model which learns to defer fairly, and show that these models can better navigate the accuracy-fairness tradeoff. We also consider deferring models as one part of a decision pipeline. To this end, we provide a framework for evaluating deferring models by incorporating other decision makers' output into learning. We give an algorithm for learning to defer in the context of a larger system, and show how to train a deferring model to optimize the performance of the pipeline as a whole. This is a powerful, general framework, with ramifications for many complex domains where automated models interact with other decision-making agents. A model with a low deferral rate could be used to cull a large pool of examples, with all deferrals requiring further examination. Conversely, a model with a high deferral rate can be thought of as flagging the most troublesome, incorrect, or biased decisions by a DM, with all non-deferrals requiring further investigation. Automated models often operate within larger systems, with many moving parts. Through deferring, we show how models can learn to predict responsibly within their surrounding systems. Automated models often operate within larger systems, with many moving parts. Through deferring, we show how models can learn to predict responsibly within their surrounding systems. Building models which can defer to more capable decision makers is an essential step towards fairer, more responsible machine learning. In Section 7.1, we discuss that DM-unaware IDK training is similar to DM-aware training, except with a training DM who treats all examples similarly, in some sense. Here we show experimental evidence. The plots in FIG5 compare these two models: DM-unaware, and DM-aware with an oracle at training time, and the standard DM at test time. We can see that these models trade off between error rate and DI in almost an identical manner. We can show that when for a broad class of objective functions (including classification error and cross entropy), these are provably equivalent. Note that this class does not include our fairness regularizer; for that we show the experimental evidence in FIG5.Let Y be the ground truth label, Y DM be the DM output, Y model be the model output, s be the IDK decision indicator (s = 0 for predict, s = 1 for IDK), and Y sys be the output of the joint DM-model system. Suppose these are all binary variables. The standard rejection learning (DM-unaware) loss (which has no concept of Y DM) is BID9: DISPLAYFORM0 We can describe the learning-to-defer system output as DISPLAYFORM1 If we wish to train the system output Y sys to optimize some loss function L(Y, Y sys), we can simply train s and DISPLAYFORM2. This deferring framework is strictly more expressive than the rejection learning model, as it can be used on many different objectives L, while rejection learning is mostly used with classification accuracy. We now show that if we take the DM to be an oracle (always outputs ground truth), learning to defer reduces to rejection learning for a broad class of objective functions, including classification error, cross entropy, and mean squared error. DISPLAYFORM3 ) be the objective we aim to minimize, where Y i = arg min DISPLAYFORM4 the DM is an oracle, the learning-to-defer and learning-to-punt objectives are equivalent. Proof. As in Eq. 8, the standard rejection learning objective is DISPLAYFORM5 where the first term encourages a low loss for non-IDK examples and the second term penalizes IDK at a constant rate, with γ punt ≥ 0. In rejection learning, is usually classification error (cf. BID9). Note that this objective has no notion of DM output (Y DM).If we include a similar γ def er penalty, the deferring loss function is DISPLAYFORM6 Now, if the DM is an oracle, then DISPLAYFORM7 if we set γ def er = γ punt. The in Figures 8 and 9 roughly replicate the from BID3, who also test on the COMPAS dataset. Their are slightly different for two reasons: 1) we use a 1-layer NN and they use logistic regression; and 2) our training/test splits are different from theirswe have more examples in our training set. However, the main takeaway is similar: regularization with a disparate impact term is a good way to reduce DI without making too many more errors. We show on two datasets. The first is the COMPAS recidivism dataset, made available by ProPublica BID22 1. This dataset concerns recidivism: whether or not a criminal defendant will commit a crime while on bail. The goal is to predict whether or not the person will recidivate, and the sensitive variable is race (split into black and non-black). We used information about counts of prior charges, charge degree, sex, age, and charge type (e.g., robbery, drug possession). We provide one extra bit of information to our DM -whether or not the defendant violently recidivated. This clearly delineates between two groups in the data -one where the DM knows the correct answer (those who violently recidivated) and one where the DM has no extra information (those who did not recidivate, and those who recidivated non-violently). This simulates a real-world scenario where a DM, unbeknownst to the model, may have extra information on a subset of the data. The simulated DM had a 24% error rate, better than the baseline model's 29% error rate. We split the dataset into 7718 training examples and 3309 test examples. The second dataset is the Heritage Health dataset 2. This dataset concerns health and hospitalization, particularly with respect to insurance. For this dataset, we chose the goal of predicting the Charlson Index, a comorbidity indicator, related to someone's chances of death in the next several years. We binarize the Charlson Index of a patient as 0/greater than 0. We take the sensitive variable to be age and binarize by over/under 70 years old. This dataset contains information on sex, age, lab test, prescription, and claim details. The extra information available to the DM is the primary condition group of the patient (given in the form of a code e.g., 'SEIZURE', 'STROKE', 'PNEUM'). Again, this simulates the situation where a DM may have extra information on the patient's health that the algorithm does not have access to. The simulated DM had a 16% error rate, better than the baseline model's 21% error rate. We split the dataset into 46769 training examples and 20044 test examples. We now explain the post-hoc threshold optimization search procedure we used. In theory, any procedure can work. Since it is a very small space (one dimension per threshold = 4 dimensions), we used a random search. We sampled 1000 combinations of thresholds, picked the thresholds which minimized the loss on one half of the test set, and evaluated these thresholds on the other half of the test set. We do this for several values of α, γ in thresholding, as well as several values of α for the original binary model. We did not sample thresholds from the interval uniformly. Rather we used the following procedure. We sampled our lower thresholds from the scores in the training set which were below 0.5, and our upper thresholds from the scores in the training set which were above 0.5. Our sampling scheme was guided by two principles: this forced 0.5 to always be in the IDK region; and this allowed us to sample more thresholds where the scores were more dense. If only choosing one threshold per class, we sampled from the entire training set distribution, without dividing into above 0.5 and below 0.5. This random search was significantly faster than grid search, and no less effective. It was also faster and more effective than gradient-based optimization methods for thresholds -the loss landscape seemed to have many local minima. We go into more detail regarding the regularization term for expected disparate impact in Equation 7. When using soft thresholds, it is not trivial to calculate the expected disparate impact regularizer: DISPLAYFORM0 due to the difficulties involved in taking the expected value of an absolute value. We instead chose to calculate a version of the regularizer with squared underlying terms: DISPLAYFORM1 DISPLAYFORM2 Then, we can expandŶ i asŶ DISPLAYFORM3 whereỸ i ∈ and S(x i) ∈ are the DM and machine predictions respectively. For brevity we will not show the rest of the calculation, but with some algebra we can obtain a closed form expression for DI sof t (Y, A,Ŷ) in terms of Y, A,Ỹ and S.F : LEARNING TO DEFER, BY DEFERRAL RATE Models which rarely defer behave very differently from those which frequently defer. In FIG0, we break down the from Section 7.1 by deferral (or punting) rate. First, we note that even for models with similar deferral rates, we see a similar fairness/accuracy win for the DM-aware models. Next, we can look separately at the low and high deferral rate models. We note that the benefit of DM-aware training is much larger for high deferral rate models. This suggests that the largest benefit of learning to defer comes from a win in fairness, rather than accuracy. Figure 10: Comparison of DM-aware and -unaware learning. Split into 3 bins, low, medium, and high deferral rate for each dataset. Bins are different between datasets due to the differing distributions of deferral rate observed during hyperparameter search.

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

Incorporating the ability to say I-don't-know can improve the fairness of a classifier without sacrificing too much accuracy, and this improvement magnifies when the classifier has insight into downstream decision-making.

Represent the TLDR for finding the corresponding scientific article

We can identify prototypical and outlier examples in machine learning that are quantifiably very different, and make use of them to improve many aspects of neural networks.