Question: Represent the article for finding a single-sentence TLDR

Question: We introduce the masked translation model (MTM) which combines encoding and decoding of sequences within the same model component. The MTM is based on the idea of masked language modeling and supports both autoregressive and non-autoregressive decoding strategies by simply changing the order of masking. In experiments on the WMT 2016 Romanian-English task, the MTM shows strong constant-time translation performance, beating all related approaches with comparable complexity. We also extensively compare various decoding strategies supported by the MTM, as well as several length modeling techniques and training settings. Neural machine translation (NMT) has been developed under the encoder-decoder framework with an intermediary attention mechanism . The encoder learns contextualized representations of source tokens, which are used by the decoder to predict target tokens. These two components have individual roles in the translation process, and they are connected via an encoder-decoder attention layer . Many advances in NMT modeling are based on changes in the internal layer structure (; ; ; ;, tweaking the connection between the layers (; ; ; a), or appending extra components or latent variables (; ; ;) -all increasing the overall architectural complexity of the model while keeping the encoder and decoder separated. Our goal is to simplify the general architecture of machine translation models. For this purpose, we propose the masked translation model (MTM) -a unified model which fulfills the role of both the encoder and decoder within a single component. The MTM gets rid of the conventional decoder as well as the encoder-decoder attention mechanism. Its architecture is only a sequence encoder with self-attention layers, trained with an objective function similar to masked language modeling . In order to model the translation problem, the MTM is given the concatenation of the source and target side from a parallel sentence pair. This approach is similar to the translation language model presented by , but focuses on the target side, i.e. the masking is applied to some selected positions in the target sentence. The MTM is trained to predict the masked target words relying on self-attention layers which consider both the source sentence and a masked version of the target sentence. Trained in this way, the model is perfectly suitable for non-autoregressive decoding since the model learned to predict every position in parallel, removing the dependency on decisions at preceding target positions. Within its extremely simple architecture, one can realize various decoding strategies, e.g., using left-to-right, non-autoregressive, or iterative decoding by merely adjusting the masking schemes in search. We present a unified formulation of the MTM for different decoding concepts by factorizing the model probability over a set of masked positions. The MTM has several advantages over the conventional encoder-decoder framework: • A simpler architecture • A variety of decoding strategies including constant-time approaches (Section 3.3.1) On the WMT 2016 Romanian→English translation task, our MTM achieves a better performance than comparable non-autoregressive/constant-time methods while keeping its simple architecture. Using our general formulation of the MTM, we compare the translation performance of various decoding strategies. Moreover, we show that this model allows for decoding speed-up by merely adjusting the number of iterations at the small cost of translation performance. There have been some attempts to combine the encoder and decoder into a single component for simplified translation modeling. share the encoder and decoder parameters of a Transformer translation model and allow the encoder-decoder attention to access the inner layers of the encoder as well. extend this idea by adding locality constraints in all attention layers. train a Transformer decoder on a large monolingual corpus as a language model and use it as an unsupervised translation model on pairs of source and target sentences. Similarly to our work, all these approaches couple the encoding and decoding on the self-attention level. However, their decoding considers only left-side target context, enabling only left-to-right autoregressive translation. Furthermore, their encoding of a source sentence is limited to the source side itself, while our MTM can refine the source representations according to a partial target hypothesis represented in the decoder states. Both aspects hinder their methods from making the best use of the bidirectional representation power of the combined model. Non-autoregressive NMT, which predicts all target words in parallel, potentially exploits full bidirectional context in decoding. To make the parallel decoding produce a reasonable hypothesis, reuse the source words as inputs to the decoder and insert an additional attention module on the positional embeddings. use a separate decoder to revise the target hypotheses iteratively, where train a single decoder with MLM objectives for both the first prediction and its refinements. To improve the integrity of the hypotheses, one could also employ an autoregressive teacher to guide the states of the non-autoregressive decoder b), apply sentence-level rewards in training , or integrate generative flow latent variables . The self-attention layers of their decoders attend to all target positions, including past and future contexts. However, all these methods still rely on the encoder-decoder framework. In this work, we collapse the boundary of the encoding and decoding of sequences and realize non-autoregressive NMT with a unified model. Regardless of the encoding or decoding, the self-attention layers of our MTM attend to all available source and target words for flexible information flow and a model of simplicity. A common problem in non-autoregressive sequence generation is that the length of an output should be predefined beforehand. The problem has been addressed by averaging length difference (b), estimating fertility , dynamically inserting blank outputs via connectionist temporal classification (CTC) (Libovickỳ &), or directly predicting the total length from the encoder representations (; ;). In this work, we train a separate, compact length model on given bilingual data. The training of an MTM is based on the MLM objective , developed for pretraining representations for natural language understanding tasks. concatenate source and target sentences and use them together as the input to an MLM, where both source and target tokens are randomly masked for the training. This improves cross-lingual natural language inference in combination with the original MLM objective, but has not been applied to translation tasks. We use the concatenated input sentences but selectively mask out only target tokens to implement source→target translation. As for inference with an and propose to build up an output sequence iteratively by adjusting the input masking for each iteration. In this work, the generation procedures of both works are tested and compared within our MTM, along with other autoregressive/non-autoregressive decoding strategies (Section 4.1). We introduce the masked translation model (MTM) focusing on three aspects: model architecture, training, and decoding. In the corresponding sections, we show how the MTM 1) relaxes the con-......... ventional architectural constraint of the encoder-decoder framework, 2) learns to perform both nonautoregressive translation and its refinements, and 3) does translation with various decoding strategies in a variable number of iterations. Given a source sentence f.., f J (source input) the goal of an MTM p θ is to generate the correct translation e I 1 = e 1,..., e i,..., e I (target output) by modeling: 1, e I i+1 ) (target input) is a corrupted version -a subset -of the surrounding context in the target sentence e I 1. Therefore, the MTM models the true target sentence e I 1 independently for each position, given both the true source and a noisy version of the target context. Figure 1 illustrates the MTM network architecture which is the same as the MLM presented by or the encoder of a Transformer network with an output softmax layer on top of all corrupted positions. In particular, an MLM consists of N transformer encoder layers each containing two blocks: a self-attention layer with full bidirectional context as well as two linear layers with a RELU activation in between. Layer normalization is applied before every block, and a residual connection is used to combine the output of a block with its input . The source f, whose space is shared over the source and target languages by a joint subword vocabulary. A positional encoding vector is added where the position index is reset at the first target token. Similarly to and , we add language embeddings to distinguish source and target vocabularies efficiently. The embedded representations pass through N transformer encoder layers, where the attention has no direction constraints; this allows the model to use the full bidirectional context beyond the language boundary. Note that the hidden representations of source words can also attend to those of (partially hypothesized) target tokens, which is impossible in the encoder-decoder architecture. During training the target side inputẽ I 1 may be 1) fully masked out, resembling the initial stage of translation before hypothesizing any target words, 2) partially corrupted, simulating intermediate hypotheses of the translation process that need to be corrected, 3) or even the original target sentence e I 1, representing the final stage of translation which should not be refined further. The different levels of corruptions are used to model all plausible cases which we encounter in the decoding process -from primitive hypotheses to high-quality translations (Section 3.3). Given bilingual training data D = {(f Note that a model for length prediction is trained separately and the details of which can be found Appendix A. We cannot expect a single step of parallel decoding to output an optimal translation from only the source context. Therefore, we consider the second scenario of refining the hypothesis, where we simulate a partial hypothesis in training by artificially corrupting the given target sentence in the input. For training an MTM, we formulate this corruption by probabilistic models to 1) select the positions to be corrupted (p s) and 2) decide how such a position is corrupted (p c). The goal of the MTM is now to reconstruct the original target sentence e I 1. This leads to the training loss: where C is a set of positions to be corrupted. The corrupted target inputẽ I 1 is generated in two steps of random decisions : 1. Target positions i ∈ C for the corruption are sampled from a uniform distribution until ρ s · I samples are drawn, with hyperparameter ρ s ∈. We denote this selection process by C ∼ p s (C|e I 1) = p s (C|I) as a simplified notation. 2. The specific type of corruption for each selected position i ∈ C is chosen independently by samplingẽ i from p c (ẽ i |e i). Note that we train the model to reconstruct the original token e i only for the corrupted positions i ∈ C. For the remaining positions i / ∈ C, the corrupted sentence is filled with the original word, i.e.ẽ i = e i. These uncorrupted positions provide the context to the network for the denoising, and no loss is applied for these positions to prevent a bias towards copying. We optimize this criterion in a stochastic way, where C and theẽ i are sampled anew for each epoch and each training instance. In principle, the MTM is a denoising autoencoder of the target sentence conditioned on a source sentence. The MTM training can be customized by selecting p s and p c appropriately. , the probability p s is defined as a uniform distribution over all target positions, without considering the content of the sentence. For the corruption model p c, we define a set of operations and assign a probability mass ρ o ∈ to each operation o. We use the three operations presented by: • Replace with a mask token: • Replace with a random word e * uniformly sampled from the target vocabulary V e: Random: • Keep unchanged: Original: e Figure 2 shows an example of corrupting a target input sentence in the MTM training. Here all positions except 1 are corrupted, i.e. C = {2, 3, 4}. At Position 2 the original word is kept by the Keep operation of p c but in contrast to Position 1 (No Operation) there is a training loss added for Position 2. 3.3 DECODING As described above, the MTM is designed and trained to deal with intermediate hypotheses of varying quality as target input. Accordingly, decoding with an MTM consists of multiple iterations τ = 1,..., T: A non-autoregressive generation of the initial hypothesis (for τ = 1) and several steps of iterative refinements (inspired by) of the hypothesis (for τ > 1). In the context of this work, an iteration of the MTM during decoding refers to one forward pass of the model based on a given source and target input, followed by the selection of a target output based on the predictions of the MTM. To simplify the notation, we denote the given source sentence by F = f and the generated target translation byÊ =ê I 1. Similar to traditional translation models, the goal of decoding with an MTM is to find a hypothesisÊ that maximizes the translation probability: {p(E, I|F)} = arg max We approximate this by a two stage maximization process where we first determine the most likely target lengthÎ:= arg max I {p(I|F)} followed by: Instead of a left-to-right factorization of the target sentence which is common in autoregressive decoding, we perform a step-wise optimization on the whole sequence. For this we define the sequenceÊ (τ) starting fromÊ:=</M>,...,</M> by selecting the best hypothesis given the predecessorÊ (τ −1), i.e.:Ê (τ):= arg max Introducing an intermediate representationẼ allows for a more fine-grained control of the decoding process: where the probability p θ is modeled by a neural network with parameters θ and the masked sequencẽ is modelled by p m, which defines a specific search strategy (Section 3.3.1). In most scenarios, the search strategy is defined to be deterministic, which has the effect that all probability mass of p m is concentrated on one masked sequenceẼ (τ −1) and we can reformulate Equation as: and thus the score is defined solely by the MTM network. The iterative procedure presented in Equation describes a greedy optimization, as it selects the currently best hypothesis in each iteration. This does not provide any guarantee for the quality of Sample a masked sequence: 1) ) Compute the model output greedily for every position: e the final output. However, if we assume that the model score improves in each iteration, i.e. if we can show that: then we know that the maximum score is obtained in the last iteration T. To the best of our knowledge, it is not possible to provide a theoretical proof for this property, yet we will show empirical evidence that it holds in practice (see Section 4.1). Thus, in order to find the best hypothesis, it is sufficient to follow the recursive definition presented in Equation, which can be computed straight forward ing in an iterative decoding scheme. Algorithm 1 describes this process of MTM decoding. In short, 1) generate a hypothesis (Ê), 2) select positions to be masked, and 3) feed the masked hypothesis (Ẽ) back to the next iteration. Note that the output for each position e (τ) i is computed in the same forward pass without a dependency on other words e (τ) i (i = i) from the same decoding step. This means that the first iteration (τ = 1) is non-autoregressive decoding (; Libovickỳ &). Nonautoregressive models tend to suffer from the multimodality problem , where conditionally independent models are inadequate to capture the highly multimodal distribution of target translations. Our MTM decoding prevents this problem making iterative predictionsÊ (τ) each conditioned on the previous sequence. EachÊ (τ) from one forward pass, yielding a complexity of O(T) decoding steps instead of O(I) as in traditional autoregressive NMT. Thus the MTM decoding can potentially be much faster than conditional decoding of a standard encoder-decoder NMT model, as the number of iterations is not dictated by the target output length I. Furthermore, compared to the pure non-autoregressive decoding with only one iteration, our decoding algorithm may collapse the multimodal distribution of the target translation by conditioning on the previous output . The masking probability p m introduced in Equation resembles the two-step corruption of the training (Equation): where C is a set of positions to be masked. Similarly to the training, the corruption is performed only for i ∈ C (τ) and the remaining positions i / ∈ C (τ) are kept untouched. For the corruption model p c in decoding, only the Mask operation is activated, i.e. ρ mask = 1 and ρ o = 0 for o = mask. This leads to the following simple decisions: The ing masked sequenceẼ (τ) is supposed to shift the model's focus towards a selected number of words, chosen by the decoding strategy p s. Given this definition of p c above, a masked (intermediate) hypothesis in decoding is determined solely by the position selection p s, which differs by decoding strategy. Each decoding strategy starts from a fully masked target input, i.e. C = {1, ..., I}, and uncovers positions incrementally in each iteration. The simplest solution is to simply feed back the completely unmasked sequence in each iteration : This method works with the richest context from the output of the previous iteration. This may, however, hurt the model's performance as the first output is often quite poor, and the focus of the model is spread across the whole sentence. Instead of unmasking all positions from the beginning, one can unmask the sequence randomly one position at a time , inspired by Gibbs sampling : Note that this method is nondeterministic and it takes at least I iterations before the output is conditioned on the completely unmasked sequence. A deterministic alternative to the random strategy is to unveil the sequence step-wise, starting from the left-most position in the target sequence. In every decoding iteration τ = 1,..., T, the index i = τ − 1 is removed from the set of masked positions: This corresponds to the traditional autoregressive NMT, but the parallel nature of our MTM decoding inherently enables to update the prediction for any position at any time, e.g. the prediction for the first position can change in the last iteration. Furthermore, it allows for different step-wise strategies -revealing the sequence right-to-left (R2L) or starting from the middle in both directions (middleout) -without re-training the model. L2R decoding ignores a huge advantage of the MTM, which is the property that the fully bidirectional model can predict sequence elements in any given order. This characteristic is leveraged by masking a decreasing number of K(τ) positions in each iteration: At each iteration, K(τ) positions with the lowest model score p θ (confidence) remain masked: where the number of masked positions K(τ) is chosen to be linearly decreasing over the number of iterations T : One can also unmask positions one by one, i.e. K(τ) = I − τ, sacrificing potential improvements in decoding speed. Fertility-based 29.1 -CTC (Libovickỳ &) 24.7 -Imitation learning 28.9 -Reinforcement learning 27.9 -Generative flow 30. We implemented the MTM in the RETURNN framework and evaluate the performance on the WMT 2016 Romanian→English translation task 1. All data used in the experiments are preprocessed using the MOSES tokenizer and frequent-casing . We learn a joint source/target byte pair encoding (BPE) with 20k merge operations on the bilingual training data. Unless mentioned otherwise, we report on the newstest2016 test set, computed with case sensitivity and tokenization using the software SacreBLEU 2 . The MTMs in our experiments follow the base configuration of , however, with a depth of 12 layers and 16 attention heads. They are trained using Adam with an initial learning rate of 0.0001 and a batch size of 7,200 tokens. Dropout is applied with a probability of 0.1 to all hidden layers and word embeddings. We set a checkpoint after every 400k sentence pairs of training data and reduce the learning rate by a factor of 0.7 whenever perplexity on the development set (newsdev2016) does not improve for nine consecutive checkpoints. The final models are selected after 200 checkpoints based on the development set perplexity. During training, we select a certain percentage ρ s of random target tokens to be corrupted. This parameter is selected randomly from a uniform distribution ρ s ∼ U in every training step. We further deviate from , by selecting the hyperparameters for corruption to be ρ mask = 0.6, ρ rand = 0.3, and ρ keep = 0.1, which performed best for MTMs in our preliminary experiments. The main of the MTM are given in Table 1 along with comparative baselines. In total, our MTM, despite its extremely simple architecture, outperforms comparable constant-time NMT methods which do not depend on sequence lengths. Compared to the conventional autoregressive baseline, the MTM falls behind by only -2.4% BLEU with a constant number of decoding steps and the lower model complexity. Furthermore, a control experiment using the gold length instead of a predicted length improves the from the same MTM model by 1.5% BLEU. This minimizes the gap between our MTM and a comparable encoder-decoder model down to 0.9% BLEU, while our model has the ability to improve the decoding speed without retraining by simply reducing the number of iterations, thus trading in performance against speed. Note that all other methods shown in Table 1 are based on the encoder-decoder architecture, which is more sophisticated. Moreover, the performance of , , and relies heavily on knowledge distillation from a well-trained autoregressive model, which involves building an additional NMT model and translating the entire training data with that model. This causes a lot more effort and computation time in training, while the MTM requires no such complicated steps, and its training is entirely end-to-end. et al. demonstrate in their work that even better could be possible by computing multiple translations in parallel for a set of most likely length hypotheses. This approach or even a beam-search variant of our iterative unmasking, will be a focus in our future work. As described in Section 3.3.1, the flexibility of the MTM allows us to easily implement different decoding strategies within a single model. Pure non-autoregressive decoding, i.e., a single forward pass to predict all target positions simultaneously, yields poor translation performance of 13.8% BLEU on the validation set (newsdev2016), which implies that several decoding iterations are needed to produce a good hypothesis. We know this to be true if the inequality in Equation holds, i.e. if we see our model score improving in every iteration. Figure 3 shows that we can actually observe this property by monitoring the average model score throughout all iterations. Outputs for individual sentences might still worsen between two iterations. The overall score, however, shows a steady improvement in each iteration. In Figure 4, we compare various decoding strategies and plot their performance for different number of decoding iterations T. "Fully unmasking", i.e. re-predicting all positions based on the previous hypothesis, improves the hypothesis fast in the early iterations but stagnates at 22% BLEU. L2R, R2L, and confidence-based one-by-one all unmask one position at a time and show a very similar tendency with confidence-based one-by-one decoding reaching the strongest final performance of 31.9% BLEU. Confidence-based fixed-T unmasks several target positions per time step and achieves similar performance. In contrast to position-wise unmasking, the decoding with a fixed number of T (and linear unmasking) only needs ten decoding iterations to reach close to optimal performance. We test "middle-out" a variation of the L2R strategy to see whether the generation order is negligible as long as the sentence is generated contiguously. While this improves the hypothesis faster -most likely due to its doubled rate of unmasking -the final performance is worse than those of L2R or R2L decoding. Random selection of the decoding positions shows comparably fast improvements up to the 10th iterations, keeping up with middle-out, even though it reveals the hypothesis for a single position per iteration, however its performance saturates below most other decoding strategies. Overall the best for a low iteration count is obtained with confidence-based decoding with a fixed number of iterations. This shows that it is possible and sometimes even beneficial to hypothesize several positions simultaneously. We conclude that the choice of the decoding strategy has substantial impacts on the performance and hypothesize that a good decoding strategy relies on the model score to choose which target positions should be unmasked. In this work we simplify the existing Transformer architecture by combining the traditional encoder and decoder elements into a single component. The ing masked translation model is trained by concatenating source and target and applying BERT-style masking to the target sentence. The novel training strategy introduced with the MTM requires a rethinking of the search process and allows for various new decoding strategies to be applied in the theoretical framework we developed in this work. A detailed comparison shows that unmasking the sequence one-by-one gives the overall best performance, be it left-to-right, right-to-left, or confidence-based. Unveiling a constant number of tokens based on confidence in each decoding step, however, can achieve reasonable performance with a fixed, much smaller number of iterations. We show that there is a potential of at least 1.5 % BLEU improvement that can be achieved by more elaborate length models, which yields itself as a good start for further research. Furthermore, we plan to extend the decoding strategies to work with beam search and verify our observations on further language pairs. In this section we present several ablation studies as well as a deeper look into the decoding to further investigate the MTM. Autoregressive models determine the output length by stopping the generation process once a special token that marks the end of the sentence (e.g. '</S>') is predicted. This approach is not compatible with the MTM decoding as target tokens for all positions are predicted in parallel. Therefore we assume a given output length I and a train length model p(I|f J 1) on the bilingual training data. In training, the true length is used, and in decoding, we choose arg max I p(I|f • Count-based Table. For the unseen source lengths in training, we assume I = J. • Poisson Distribution: a more smoothly parametrized model. For J's not appearing in the training data, we back off to a global distribution with the parameter λ that is learned via maximum likelihood overall source/target length pairs in the training data, i.e. • Recurrent Neural Network (RNN): We take the last hidden state as the input to a target length classifier, i.e. a linear projection with a softmax layer over the possible target lengths I ∈ {1, 2, ..., 200}. • Bidirectional RNN: a variation of the above which employs a BiLSTM and uses the last hidden state of the forward and the backward LSTM for length prediction. Table 2 verifies that the translation performance depends highly on the length model. We also report the averaged absolute difference of the predicted length and the reference length. The simplest count-based model shows the worst performance, where the target length is merely determined by a frequency table. The Poisson distribution models the length more systematically, giving +1.1% BLEU against the count-based model. RNN models consider not only the source length but also the whole sentence semantics, and slightly outperform the Poisson distribution. The bidirectional RNN model predicts the target length even better, but the translation performance only improves marginally. Note that using the Reference length improves even further by +1.6% BLEU over our strongest system, which shows that a good length model is a crucial component of the MTM system. As discussed earlier, the MTM architecture is equivalent to a traditional transformer encoder. Nevertheless, the way it is applied to the task at hand differs very much, even compared to an MLM. Thus it was crucial to do a thorough hyperparameter search to obtain an optimal model performance. The baseline MTM configuration we present here is already the product of many preliminary experiments, setting the number of attention heads h = 16, dropout P drop = 0.1, and the learning rate reduction scheme. The ablation study presented in table 3 highlights the importance of both the number of attention heads and especially dropout. It also shows that it was crucial to increase the model depth N compared to the standard transformer encoder, by matching the total number of layers N = 12 as they are used in an encoder-decoder model. In this section, we show a detailed derivation of the decoding process which justifies our iterative optimization procedure and modularizes the unmasking procedure to apply the application of various decoding strategies. Assuming we have a hypothesized target lengthÎ, the goal is to find an optimal target sequenceÊ given lengthÎ and source sentence F: The MTM decoding to find such a sequence is performed in T iterations, whose intermediate hypotheses are introduced as latent variables E,..., E (T −1): = arg max where the last iteration should provide the final prediction, i.e. E (T):= E. Applying the chain rule and a first-order Markov assumption on E (τ) yields: = arg max with E 0:= </M>,..., </M> a sequence of lengthÎ. In a next step, we approximate the sum by a maximization and subsequently apply a logarithm to get: To simplify further derivations, we introduce the score function Q here and do another approximation by considering only the score Q from a single maximum timestep instead of the full sum over τ = 1,..., T. ≈ arg max Even with this approximation, we are still trying to find a value for each E (τ) that optimizes the score of another iterationτ via the connection of dependencies in Q. As this is impractical to compute, we alleviate the problem by focusing on a step-wise maximization. For this we define the sequenceÊ (τ) (withÊ:= E ) as the best hypothesis of iteration τ givenÊ (τ −1), i.e.: −1), F,Î) = arg max If we useÊ (τ −1) instead of E (τ −1) as the dependency in Q, we restrict the optimization to maximizing each step independently, given its predecessors optimum: Ideally, this iterative procedure should improve the score Q in each iteration, i.e.: Q(E (τ) |Ê (τ −1), F,Î) ∀τ = 1,..., T − 1 While Equation is not true for the general case we observe empirically that this the statement is true on average (see Figure 3). This means that the maximum score to be obtained in the last iteration T:Ê (F,Î) ≈ arg max which can be simplified to: = arg max Q(E (T) |Ê (T −1), F,Î) =Ê For completeness we report the strongest for each decoding strategy from Figure 4 in Table 4.

Represent the TLDR for finding the corresponding scientific article

We use a transformer encoder to do translation by training it in the style of a masked translation model.

Represent the short summary for finding the corresponding scientific article (long)

The paper introduces a method of training generative recurrent networks that helps to plan ahead. We run a second RNN in a reverse direction and make a soft constraint between cotemporal forward and backward states.


Question: Represent the article for finding a single-sentence TLDR

Question: We present local ensembles, a method for detecting extrapolation at test time in a pre-trained model. We focus on underdetermination as a key component of extrapolation: we aim to detect when many possible predictions are consistent with the training data and model class. Our method uses local second-order information to approximate the variance of predictions across an ensemble of models from the same class. We compute this approximation by estimating the norm of the component of a test point's gradient that aligns with the low-curvature directions of the Hessian, and provide a tractable method for estimating this quantity. Experimentally, we show that our method is capable of detecting when a pre-trained model is extrapolating on test data, with applications to out-of-distribution detection, detecting spurious correlates, and active learning. As machine learning is deployed in increasingly vital areas, there is increasing demand for metrics that draw attention to potentially unreliable predictions. One important source of unreliability is extrapolation. Extrapolation can be formalized in a number of ways: it can refer to making predictions on inputs outside the support of the training data, making predictions with high Bayesian or Frequentist uncertainty, or making predictions that depend strongly on arbitrary choices outside of the learning problem specification (e.g., a random seed). In this paper, we develop a method for detecting this last form of extrapolation. Specifically, we say that a trained model is extrapolating on a test input if the prediction at this input is underdetermined -meaning that many different predictions are all equally consistent with the constraints posed by the training data and the learning problem specification (i.e., the model architecture and the loss function). Underdetermination is just one form of extrapolation, but it is particularly relevant in the context of overparameterized model classes (e.g. deep neural networks). Recently, simple (but computationally expensive) ensembling methods , which train many models on the same data from different random seeds, have proven highly effective at uncertainty quantification tasks . This suggests that underdetermination is a key threat to reliability in deep learning, and motivates flexible methods that can detect underdetermined predictions cheaply. With this motivation, we present local ensembles, a post-hoc method for measuring the extent to which a pre-trained model's prediction is underdetermined for a particular test input. Given a trained model, our method returns an extrapolation score that measures the variability of test predictions across a local ensemble, i.e. a set of local perturbations of the trained model parameters that fit the training data equally well. Local ensembles are a computationally cheap, post-hoc alternative to fully trained ensembles, and do not require special training procedures of approximate ensembling methods that measure related, but distinct, notions of uncertainty . Local ensembles also address a gap in approximate methods for estimating prediction uncertainty. Specifically, whereas exact Bayesian or Frequentist uncertainty includes underdetermination as one component, approximate methods such as Laplace approximations or influence function-based methods break down when underdetermination is present. In contrast, our method leverages the pathology that makes these methods struggle (an ill-conditioned Hessian). Our contributions in this paper are as follows: • We present local ensembles, a test-time method for detecting underdetermination-based extrapolation in overparameterized models. • We demonstrate theoretically that our method approximates the variance of a trained ensemble with local second-order information. • We give a practical method for tractably approximating this quantity, which is simpler and cheaper than alternative second-order reliability methods. • Through experiments aimed at testing underdetermination, we show our method approximates the behavior of trained ensembles, and can detect extrapolation in a range of scenarios. 2.1 SETUP Let z = (x, y) be an example input-output pair, where x is a vector of features and y is a label. We define a model in terms of a loss function L with parameters θ as a sum over training examples, where is an example-wise loss (e.g., mean-squared error or cross entropy). Let θ be the parameters of the trained model, obtained by, e.g., minimizing the loss over this dataset, i.e., θ = arg min θ L(θ). We write the prediction function given by parameters θ at an input x asŷ(x, θ). We consider the problem of auditing a trained model, where unlabeled test points x arrive one at a time in a stream, and we wish to assess extrapolation on a point-by-point basis. Figure 1: In this quadratic bowl, arrows denote the small eigendirection, where predictions are slow to change. We argue this direction is key to extrapolation. In this section, we introduce our local ensemble extrapolation score E m (x) for an unlabeled test point x (significance of m explained below). The score is designed to measure the variability that would be induced by randomly choosing predictions from an ensemble of models with similar training loss. Our score has a simple form: it is the norm of the prediction gradient g θ (x):= ∇ θŷ (x, θ) multiplied by a matrix of Hessian eigenvectors spanning a subspace of low curvature U m (defined in more detail below). Here, we show that this score is proportional to the standard deviation of predictions across a local ensemble of models with near-identical training loss, and demonstrate that this approximation holds in practice. Our derivation proceeds in two steps. First, we define a local ensemble of models with similar training loss, then we state the relationship between our extrapolation score and the variability of predictions within this ensemble. The spectral decomposition of the Hessian H θ plays a key role in our derivation. Let where U is a square matrix whose columns are the orthonormal eigenvectors of H θ, written (ξ, · · ·, ξ (p) ), and Λ is a square, diagonal matrix with the eigenvalues of H θ, written (λ, · · ·, λ (p) ), along its diagonal. As a convention, we index the eigenvectors and eigenvalues in decreasing order of the eigenvalue magnitude. To construct a local ensemble of loss-preserving models, we exploit the fact that eigenvectors with large corresponding eigenvalues represent directions of high curvature, whereas eigenvectors with small corresponding eigenvalues represent directions of low curvature. In particular, under the assumption that the model has been trained to a local minimum or saddle point, parameter perturbations in flat directions (those corresponding to small eigenvalues λ (j) ) do not change the training loss substantially (see Fig. 1). 1 We characterize this subspace by the span of eigenvectors with 1 To see this, consider a second-order Taylor approximation to the training loss when the parameters are Note that the corresponding small eigenvalues. Formally, let m be the eigenvalue index such that the eigenvalues {λ (j): j > m}, are sufficiently small to be considered "flat". 2 We call the subspace spanned by {ξ (j): j ∈ σ} the ensemble subspace. Parameter perturbations in the ensemble subspace generate an ensemble of models with near-identical training loss. Our score E m (x) characterizes the variance of test predictions with respect to random parameter perturbations ∆ θ that occur in the ensemble subspace. We now show that our extrapolation score E m (x) is, to the first order, proportional to the standard deviation of predictions at a point x. Proposition 1. Let ∆ θ be a random parameter perturbation with mean zero and covariance proportional to the identity · I, projected into the ensemble subspace spanned by {ξ (j): j > m}. Let P ∆ (x) be the linearized change in a test prediction at x induced by perturbing the learned parameters θ by ∆ θ: Proof. First, we characterize the distribution of the loss-preserving parameter perturbation ∆ θ. Let U m be the matrix whose columns {ξ (j): j > m} span the ensemble subspace. Then U m U m is a projection matrix that projects vectors into the ensemble subspace, so the projected perturbation ∆ θ has covariance · U m U m. From this, we derive the variance of the linearized change in prediction We test this hypothesized relationship on several tabular datasets. We train an ensemble of twenty neural networks using the same architecture and training set, varying only the random seed. Then, for each model in the ensemble, we calculate our local ensemble extrapolation score E m (x) for each input x in the test set (see Sec. 3 for details). For each x, we compare, across the ensemble, the mean value of E m (x) to the standard deviationŷ(x). In Fig. 2, we plot these two quantities against each other for one of the datasets, finding a nearly linear relationship. On each dataset, we found a similar, significantly linear relationship (see Table 1 and Appendix A). We note the relationship is weaker with the Diabetes dataset; the standard deviations of these ensembles are an order of magnitude higher than the other datasets, indicating much noisier data. Finally, we note that we can obtain similar for the standard deviation of the loss at test points if we redefine g as the loss gradient rather than the prediction gradient. We now discuss a practical method for computing our extrapolation scores. The key operation is constructing the set of eigenvectors that span a suitably loss-preserving ensemble subspace (i.e. have sufficiently small corresponding eigenvalues). Because eigenvectors corresponding to large eigenvalues are easier to estimate, we construct this subspace by finding the top m eigenvectors and defining the ensemble subspace as their orthogonal complement. Our method can be implemented with any algorithm that returns the top m eigenvectors. See below for discussion on some tradeoffs in the choice of m, as well as the algorithm we choose (the Lanczos iteration ). first-order term drops out because we assume θ lies at a local minimum or saddle point. Perturbations ∆ θ that lie in the subspace spanned by eigenvectors ξ (j) whose corresponding eigenvalues λ (j) are close to zero contribute negligibly to this sum. These eigenvectors define an m-dimensional subspace which is the orthogonal complement to the ensemble subspace. We use these eigenvectors to construct a matrix that projects gradients into the ensemble subspace. Specifically, let U m ⊥ be the matrix whose columns are these large-eigenvalue eigenvectors {ξ (j): j ≤ m}. Then U m ⊥ U m ⊥ is the projection matrix that projects vectors into the "large eigenvalue" subspace, and I − U m ⊥ U m ⊥ projects into its complement: the ensemble subspace. Now, for any test input x, we take the norm of this projected gradient to compute our score The success of this approach depends on the choice of m. Specifically, the extrapolation score E m (x) is the most sensitive to underdetermination in the region of the trained parameters if we set m to be the smallest index for which the training loss is relatively flat in the implied ensemble subspace. If m is set too low, the ensemble subspace will include well-constrained directions, and E m (x) will over-estimate the prediction's sensitivity to loss-preserving perturbations. If m is set too high, the ensemble subspace will omit some under-constrained directions, and E m (x) will be less sensitive. For models where all parameters are well-constrained by the training data, a suitable m may not exist. This will usually not be the case for deep neural network models, which are known to have very ill-conditioned Hessians (see, e.g.,). Whether a direction is "well-constrained" is ultimately a judgment call for the user. One potential heuristic is to consider a small parameter perturbation of a fixed norm in the ensemble subspace, and to set a tolerance for how much that perturbation can change the training loss; for small perturbations, the change in loss is a linear function of the curvature in the direction of the perturbation, which is upper bounded by λ (m). We use the Lanczos iteration to estimate the top m eigenvectors, which presents a number of practical advantages for usage in our scenario. Firstly, it performs well under early stopping, returning good estimates of the top m eigenvectors after m iterations. Secondly, we can cache intermediate steps, meaning that computing the m + 1-th eigenvector is fast once we have computed the first m. Thirdly, it requires only implicit access to the Hessian through a function which applies matrix multiplication, meaning we can take advantage of efficient Hessian-vector product methods . Finally, the Lanczos iteration is simple -it can be implemented in less than 20 lines of Python code (see Appendix B.1). It contains only one hyperparameter, the stopping value m. Fortunately, tuning this parameter is efficient -given a maximum value M, we can try many values m < M at once, by estimating M eigenvectors and then calculating E m by using the first m eigenvectors. The main constraint of our method is space rather than time -while estimating the first m eigenvectors enables easy caching for later use, it may be difficult to work with these eigenvectors in memory as m and model size p increase. This tradeoff informed our choice of m in this paper; we note in some cases that increasing m further could have improved performance (see Appendix E). This suggests that further work on techniques for mitigating this tradeoff, e.g. online learning of sparse representations , could improve the performance of our method. See Appendix B for more details on the Lanczos iteration. 4 RELATED WORK It is instructive to compare our extrapolation score to two other approximate reliability quantification methods that are aimed at Bayesian and Frequentist notions of extrapolation, respectively. Like our extrapolation score, both of these methods make use of local information in the Hessian to make an inference about the variance of a prediction. First, consider the Laplace approximation of the posterior predictive variance. This metric is derived by interpreting the loss function as being equivalent to a Bayesian log-posterior distribution over the model parameters θ, and approximating it with a Gaussian. Specifically, (see, e.g.,) Second, consider scores such as RUE (Resampling Under Uncertainty) designed to approximate the variability of predictions by resampling the training data . These methods approximate the change in trained parameter values induced by perturbing the training data via influence functions . Specifically, the gradient of the parameters with respect to the weight of a given training example z i is given by combine this influence function with a specific random distribution of weights to approximate the variance of predictions under bootstrap resampling; other similar formulations are possible, sometimes with theoretical guarantees . Importantly, both of these methods work well when model parameters are well-constrained by the training data, but they struggle when predictions are (close to) underdetermined. This is because, in the presence of underdetermination, the Hessian becomes ill-conditioned. Practical advice for dealing with this ill-conditioning is available , but we note that this not merely a numerical pathology; by our argument above, a poorly conditioned Hessian is a clear signal of extrapolation. In contrast to these methods, our method focuses specifically on prediction variability induced by underconstrained parameters. Our extrapolation score incorporates only those terms with small eigenvalues, and removes the inverse eigenvalue weights that make inverse-Hessian methods break down. This is clear from its summation representation: Our method also has computational advantages over approaches that rely on inverting the Hessian. Firstly, implicitly inverting the Hessian is a complex and costly process -by finding only the important components of the projection explicitly, our method is simpler and more efficient. Furthermore, we only need to find these components once; we can reuse them for future test inputs. This type of caching is not possible with methods which require us to calculate the inverse Hessian-vector product for each new test input (e.g. conjugate gradient descent). Some recent works explore the relationship between test points, the learned model, and the training set. Several papers examine reliability criteria that are based on distance in some space: within/betweengroup distances , a pre-specified kernel in a learned embedding space , or the activation space of a neural network . We implement some nearest-neighbor baselines inspired by this work in Sec. 5. Additionally, a range of methods exist for related tasks, such as OOD detection (; ; Schölkopf et al., 2001) and calibration . Some work using generative models for OOD detection uses related second-order analysis . A line of work explores the benefits of training ensemble methods explicitly, discussed in detail in. These methods have been discussed for usage in some of the applications we present in Sec. 5, including uncertainty detection , active learning and OOD detection . A number of approximate ensembling methods have also been proposed. For example, MC-Dropout and Bayes by Backprop are approximate Bayesian model averaging methods that leverage special training procedures to represent an ensemble of prediction functions. These target a distinct notion of uncertainty from our loss-preserving ensembles (see Appendix F). Finally, a line of work on "Rashomon sets" explores loss-preserving ensembles more formally in simpler model classes . In this section, we give evidence that local ensembles can detect extrapolation due to underdetermination in trained models. In order to explicitly evaluate underdetermination, we present a range of experiments where a pre-trained model has a "blind spot", and evaluate its ability to detect when an input is in that blind spot. We probe our method's ability to detect a range of extrapolation, exploring cases where the blind spot is: 1. easily visualized, 2. well-defined by the feature distribution, 3. well-defined by a latent distribution, and 4. unknown, but where we can evaluate our model's detection performance through an auxiliary task. See Appendix D for experimental details. (Fig. 3a). We compute extrapolation scores (solid line), which correlate with the standard deviation of the ensemble (dotted line) (Fig. 3b). Our OOD performance achieves high AUC (solid line) even though some of our eigenvector estimates have low cosine similarity to ground truth (dotted line) (Fig. 3c). We note that the first few eigenvalues account for most of the variation, and that our estimates are accurate. We begin with an easily visualized toy experiment. In Fig. 3a, we show our data (y = sin 4x + N (0, 1 4)). We generate in-distribution training data from x ∈ [−1, 0] ∪, but at test time, we consider all x ∈ [−3, 4]. We train 20 neural networks with the same architecture (two hidden layers of three units each). As shown in Fig. 3a, the ensemble disagrees most on x < −1, x > 2. This means that we should most mistrust predictions from this model class on these extreme values, since there are many models within the class that perform equally well on the training data, but differ greatly on those inputs. We should also mistrust predictions from x ∈, although detecting this extrapolation may be harder since the ensemble agrees more strongly on these points. For each model in the ensemble, we test our method on an OOD task: can we flag test points which fall outside the training distribution? Since OOD examples may be uncommon in practice, we use AUC to measure our performance. We show that the extrapolation score is empirically related to the standard deviation of the ensemble's predictions at the input, which in turn is related to whether the input is OOD (Fig. 3b). Examining one model from this ensemble, we observe that by estimating only m = 2 eigenvectors, we achieve > 90% AUC (Fig. 3c). It turns out that m = 10 performs best on this task/model. As we complete more iterations (m > 10) we start finding smaller eigenvalues, which are more important to the ensemble subspace and whose eigenvector we do not wish to project out. We note our AUC improves even with some eigenvector estimates having low cosine similarity to the ground truth eigenvectors (the Lanczos iteration has some stochasticity due to minibatch estimationsee Appendix B for details). We hypothesize this robustness is because the ensemble subspace of this model class is relatively low-dimensional. Even if an estimated vector is noisy, the non-overlapping parts of the projection will likely be mostly perpendicular to the ensemble subspace, due to the properties of high-dimensional space. In this experiment, we create a blind spot in a given dataset by extending and manipulating the data's feature distribution. We induce a collinearity in feature space by generating new features which are a linear combination of two other randomly selected features in the training data. This means there is potential for underdetermination: multiple, equally good learnable relationships exist between those features and the target. However, at test time, we will sometimes sample these simulated features from their marginal distribution instead. This breaks the linear dependence, requiring extrapolation (the model is by definition underconstrained), without making the new data trivially out-of-distribution. We can make this extrapolation detection task easier by generating several features this way, or make it harder by adding some noise ∼ N (0, σ 2) to these features. We run this experiment on four tabular datasets, training a two hidden-layer neural network for each task. We compare to three nearest-neighbour baselines, where the metric is the distance (in some space) of the test point to its nearest neighbour by Euclidean distance in an in-distribution validation set. NN (Inputs) uses input space; NN (Reprs) uses hidden representation space, which is formed by concatenating all the activations of the network together (inspired by , who propose a similar method for adversarial robustness); and NN (Final Layer), uses just the final hidden layer of representations. We note that since our method is post-hoc and can be applied to any twice-differentiable pre-trained model, we do not compare to training-based methods e.g. those producing Bayesian predictive distributions . Our metric is AUC: we aim to assign higher scores to inputs which break the collinearity (where the feature is drawn from the marginal), than those which do not. In Figure 5, we show that local ensembles (LE) outperform the baselines for each number of extra simulated features, and that this performance is fairly robust to added noise. Here, we extend the experiment from Section 5.2 by inducing a blind spot in latent space. The rationale is similar: if two latent factors are strongly correlated at training time, the model may grow reliant on that correlation, and at test-time may be underdetermined if that correlation is broken. We use the CelebA dataset of celebrity faces, which annotates every image with 40 latent binary attributes describing the image (e.g. "brown hair"). To induce the blind spot, we choose two attributes: a label L and a spurious correlate C. We then create a training set where L and C are perfectly correlated: a point is only included in the training set if L = C. We train a convolutional neural network (CNN), with two convolutional layers and a fully-connected layer at the end, as a binary classifier to predict L. Then, we create a test set of held-out data where The test data where L = C is in our model's blind spot; these are the inputs for which we want to output high extrapolation scores. We show in Appendix E that the models dramatically fail to classify these inputs (L = C). We compare to four baseline extrapolation scores: the three nearest-neighbour methods described in Sec. 5.2, as well as MaxProb, where we use 1− the maximum outputted probability of the softmax. We test two values of L (Male and Attractive) and two values of C (Eyeglasses and WearingHat). We chose these specific values of L because they are difficult to predict and holistic i.e.and not localized to particular areas of image space. In Table 2, we present for each of the four L, C settings, showing both the loss gradient and the prediction gradient variant of local ensembles. Note that the loss gradient cannot be calculated at test time since we do not have labels available -instead, we calculate a separate extrapolation score using the gradient for the loss with respect to each possible label, and take the minimum. Our method achieves the best performance on most settings, and is competitive with the best baseline on each. However, the variation between the tasks is quite noteworthy. We note two patterns in particular. Firstly, we note the performance of MaxProb and the loss gradient variant of our method are quite correlated, which we hypothesize relates to ∇Ŷ. Additionally, the effect of increasing m is inconsistent between experiments: we discuss possible relationships to the eigenspectrum of the trained models. See Appendix E for a discussion on these patterns. Table 2: AUC for Latent Factors OOD detection task. Column heading denotes in-distribution definitions: labels are M (Male) and A (Attractive); spurious correlates are E (Eyeglasses) and H (Wearing Hat). Image is in-distribution iff label = spurious correlate. LE stands for local ensembles. Each Lanczos iteration uses 3000 eigenvectors. 500 examples from each test set are used. 95% CI is bolded. Finally, we consider the case where we know our model has blind spots, but do not know where they are. We use active learning to probe this situation. If underdetermination is an important factor in extrapolation, then local ensembles should be useful for selecting the most useful points to add to our training set. We use MNIST and FashionM-NIST for our active learning experiments. We begin the first round with a training set of twenty: two labelled data points from each of the ten classes. In each round, we train our model (a small CNN with two convolutional layers and a fully connected layer at the end) to a minimum validation loss using the current training set. After each round, we select ten new points from a randomly selected pool of 500 unlabelled points, and add those ten points and their labels to our training set. We compare local ensembles (selecting the points with the highest extrapolation scores using the loss-gradient variant) to a random baseline selection mechanism. In Fig. 6, we show that our method outperforms the baseline on both datasets, and this improvement increases in later rounds of active learning. We only used 10 eigenvectors in our Lanczos approximation, which we found to be a surprisingly effective approximation; we did not observe improvement with more eigenvectors. This experiment serves to emphasize the flexibility of our method: by detecting an underlying property of the model, we can use the method for a range of tasks (active learning as well as OOD detection). We present local ensembles, a post-hoc method for detecting extrapolation due to underdetermination in a trained model. Our method uses local second-order information to approximate the variance of an ensemble. We give a tractable implementation using the Lanczos iteration to estimate the largest eigenvectors of the Hessian, and demonstrate its practical flexibility and utility. Although this method is not a full replacement for ensemble methods, which can characterize more complexity (e.g. multiple modes), we believe it fills an important role in isolating one component of prediction unreliability. In future work, we hope to scale these methods to larger models and to further explore the properties of different stopping points m. We also hope to explore applications in fairness and interpretability, where understanding model and training bias is of paramount importance. Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of a high-dimensional distribution. Neural computation, 13 The Lanczos iteration is a method for tridiagonalizing a Hermitian matrix. It can be thought of as a variant of power iteration, iteratively building a larger basis through repeatedly multiplying an initial vector by M, ensuring orthogonality at each step. Once M has been tridiagonalized, computing the final eigendecomposition is relatively simple -a number of specialized algorithms exist which are relatively fast (O(p 2)) . The Lanczos iteration is simple to implement, but presents some challenges. The first challenge is numerical instability -when computed in floating point arithmetic, the algorithm is no longer quarnteed to return a good approximation to the true eigenbasis, or even an orthogonal one. As such, the standard implementation of the Lanczos iteration is unstable, and can be inaccurate even on simple problems. Fortunately, solutions exist: a procedure known as two-step classical Gram-Schmidt orthogonalization -which involves ensuring twice that each new vector is linearly independent of the previous ones -is guaranteed to produce an orthogonal basis, with errors on the order of machine roundoff . A second potential challenge is presented by the stochasticity of minibatch computation. Since we access our Hessian H only through Hessian-vector products, we must use only minibatch computation at each stage. This means that each iteration will be stochastic, which will decrease the accuracy (but not the orthogonality) of the eigenvector estimates provided. However, in practice, we found that even fairly noisy estimates were nonetheless useful -see Sec. 5.1 for more discussion. The Lanczos algorithm is quite simple to implement. Figure 9 shows a a short implementation using Python/Numpy . In Section 5.2, we present for an experiment where we aim to detect broken collinearities in the feature space. In Figures 10, 11, and 12, we show on three more tabular datasets. See the main text for more experimental details. Here we give more experimental details on the datasets and models used. We subtracted the mean and divided by the standard deviation for each feature (as calculated from the training set). Boston and Diabetes . These datasets were loaded from Scikit-Learn . Abalone . This dataset was downloaded from the UCI repository at http://archive.ics.uci.edu/ml/datasets/Abalone. We converted sex to a three-dimensional one-hot vector (M, F, I). WineQuality . This dataset was downloaded from the UCI repository at http://archive.ics.uci.edu/ml/datasets/Wine+Quality. We used both red and white wines. These datasets were loaded using Tensorflow Datasets https://github.com/tensorflow/ datasets. We divided the pixel values by 255 and, for MNIST and FashionMNIST, binarized by thresholding at 0.7. For the first experiment, we train a two-layer neural network with 3 hidden units in each layer and tanh units. We train for 400 optimization steps using minibatch size 32. Our data is generated from y = sin(4x) + N (0, 1 4). We generate 200 training points, 100 test points, and 200 OOD points. We aggregate Y over a grid of 10 points from -1 to 1, with aggregation function min. We run the Lanczos algorithm until convergence. For the second experiment, we train a two-layer neural network with 5 hidden units in each layer and ReLU units. Our data is generated from y = βx 2 + N. Our training set consists of x drawn Table 3: Error rate for in and out of distribution test set with correlated latent factors setup. Column heading denotes in-distribution definitions: labels are M (Male) and A (Attractive); spurious correlate are E (Eyeglasses) and H (Wearing Hat). Image is in-distribution iff label == spurious correlate. uniformly from [−0.5, 0.5] and [2.5, 3.5]. However, at test time, we will consider x ∈ [−3, 6]. We generate 200 training points, 100 test points, and 200 OOD points. We aggregate Y over a grid of 5 points from -6 to 9, with aggregation function min. For each dataset we use the same setup. We use a two-layer MLP with ReLU activations and hidden layer sizes of 20 and 100. We trained all models with mean squared error loss. We use batch size 64, patience 100 and a 100-step running average window for estimating current performance. For the Lanczos iteration, we run up to 2000 iterations. We always report numbers from the final iteration run. For estimating the Hessian in the HVPs in the Lanczos iteration, we use batch size 32 and sample 5 minibatches. To pre-process the data, we first split randomly out 30% of the dataset as OOD. We choose 2 random features i, j and a number β ∼ U, and generate the new featurex. We also normalize this feature by its mean and standard deviation as calculated on the training set. We add random noise to the features after splitting into in-distribution and OOD -meaning we are not redrawing from the same marginal noise distribution. We use 1000 examples from in-distribution and OOD for testing. We use a CNN with ReLU hidden activations. We use two convolutional layers with 50 and 20 filters each and stride size 2 for a total of 1.37 million parameters. We trained all models with cross entropy loss. We use an extra dense layer on top with 30 units. We use batch size 32, patience 100 steps, and a 100-step running average window for estimating current performance. We sample the validation set randomly as 20% of the training set. For the Lanczos iteration, we run 3000 iterations. We always report numbers from the final iteration run. We use 500 examples from in-distribution and OOD for testing. For estimating the Hessian in the HVPs in the Lanczos iteration, we use batch size 16 and sample 5 minibatches. We use a CNN with ReLU hidden activations. We use two convolutional layers with 16 and 32 layers, stride size 5, and a dense layer on top with 64 units. We trained all models with mean squared error loss. We use batch size 32, patient 100 steps, and a 100-step running average window for estimating current performance. In Section 5.3, we discuss an experiment where we correlated a latent label L and confounder C attribute. Table 3 shows the in-distribution and out-of-distribution test error. These are drastically different, meaning that learning to detect this type of extrapolation is critical to maintain model performance. In Fig. 13 and 14, we show that the tasks present differing behaviours as more eigenvectors are estimated. We observe that for the Male/Eyeglasses and Attractive/WearingHat tasks, we get improved performance with more eigenvectors, but for the others we do not necessarily see the same improvements. Interestingly, this upward trend occurs both times that our method achieves a statistically significant improvement over baselines. It is unclear why this occurs for some settings of the task and not others, but we hypothesize that this is a sign that the method is working more correctly in these settings. E.3 RELATIONSHIP BETWEEN LOSS GRADIENT AND MaxProb METHOD As discussed in Sec. 5.3, we have the relationship between the loss, predictionŶ, and parameters θ: ∇ θ = ∇Ŷ · ∇ θŶ. Using min as an aggregation function, we find that min Y ∈{0,1} |∇Ŷ (Y,Ŷ)| has an inverted V-shape (Fig. 15). This is a similar shape to 1 − max(Ŷ, 1 −Ŷ), which is the metric implicitly measured by MaxProb. In Fig. 16, we examine the estimated eigenspectrums of the four tasks we present in the correlated latent factors experiment, to see if we can detect a reason why performance might differ on these tasks. In Fig. 16a, we show that the two tasks with the label Attractive, the eigenvalues are larger. These are also the tasks where the lossgradient-based variant of local ensembles failed, indicating that that variant of the method may be worse at handling larger eigenvalues. In Fig. 16b, we show that the two tasks where the local ensembles methods performed best (M/E, A/H, achieving statistically significant improvements over the baselines at a 95% confidence level, and also showing improvement as more eigenvectors were estimated), the most prominent negative eigenvalue is relatively smaller magnitude compared to the most prominent positive eigenvalue. This could mean that the local ensembles method was less successful in the other tasks (M/H, A/E) Figure 16: We show the estimated eigenspectrum of the four CNNs we train on the correlated latent factors task. In Fig. 16a, we show the absolute estimated eigenvalues sorted by absolute value, on a log scale. In Fig. 16b, we show the estimated eigenvalues divided by the maximum estimated eigenvalues. We only show 3000 estimated eigenvalues because we ran the Lanczos iteration for only 3000 iterations, meaning we did not estimate the rest of the eigenspectrum. simply because those models were not trained close enough to a convex minimum and still had fairly significant eigenvalues. One of the strengths of our method is that it can be applied post-hoc, making it usable in a wide range of situations. Therefore, we choose to compare the local ensembles method to only baselines which can also be applied post-hoc. However, one might still be interested in how some of these methods compare against ours. One such method is the MC-Dropout method , which constructs an ensemble at test time from a model trained with dropout, by averaging over several random dropout inferences. This method is not considered post-hoc, since it is only applicable to methods which have been trained with dropout (and works best for models with dropout on each layer, rather than just a single layer). Both MC-dropout and local ensembles estimate the variance of an ensemble achieved through perturbations of a trianed model. In MC-dropout, those perturbations take the form of stochastically setting some weights of the model to 0; on local ensembles, they are Gaussian perturbation projected into the ensemble subspace. However, despite these similarities, we provide empirical evidence here that our method is categorically different than MCDropout, by showing that our perturbations find a "flatter" (more loss-preserving) ensemble. This means that our method is more directly measuring underdetermination, since we are comparing nearby models which are equally justified by the training data. We use the four tabular datasets discussed in Sec 5.2. For each dataset, we train a neural network with dropout on each weight (as in MC-dropout), to minimize a mean squared error objective. We use a dropout parameter p = 0.1, as suggested in. We then create an MC-dropout ensemble of size 50: that is, we randomly dropout each weight with probability 0.1, and repeat 50 times. We calculate the training loss for each model in the MC-dropout ensemble. To estimate how "flat" this ensemble is (i.e. how loss-preserving on the training set), we simply subtract the training loss of the full model (no weights dropped out) from each model in the dropout- ensemble, and average these differences. We repeat this process 20 times, changing only the random seeds. We can now estimate a similar quantity (average increase in training loss across the ensemble) for our method. For each dataset, we train a neural network with the same architecture as in the dropout experiment. We then choose a standard deviation σ equal to the average magnitude of the perturbations created through MC-dropout on the corresponding task. This is to ensure a fair comparison -for each method, we are using perturbations on the same scale. We run the Lanczos iteration to convergence. We then compute the ensemble subspace for each Lanczos iteration m (recall that p − m is the dimensionality of the ensemble subspace, and m is the number of dimensions projected out). For each ensemble subspace of dimensionality p − m, we sample Gaussian noise from N (0,), project it into the ensemble subspace, and add it to the model's trained parameters. This ensures that when projected into the ensemble subspace, the Gaussian perturbation has the same expected magnitude as the dropout perturbations for the corresponding task. Using these projected Gaussian perturbations, we create an ensemble of size 50 for each m, and calculate the training loss for each model in the local ensemble. We subtract the training loss of the original model (perturbation 0) from each model in the local ensemble, and average across the ensemble. We repeat this process 20 times, changing only the random seeds. In Figure 18, we show the of this comparison for each of the four datasets. The decreasing curve shows that as we project out more eigenvectors (increasing m), the ing ensemble subspace becomes flatter, i.e. perturbations in this subspace increase training loss by a smaller amount. The horizontal dotted line is the average training loss increase for model in an an MC-dropout ensemble. We see that this horizontal line is well above the descending curve representing local ensembles. Recall that we scaled the perturbations to be the same size for both MC-Dropout and all value of m for local ensembles. This means that, for perturbations of the same magnitude, for most values of m, the ensembles found by our method are much flatter than those found by MC-dropout. While MC-dropout ensembles may be useful for other tasks, this shows that our method is more directly measuring underdetermination in the model. Additionally, we run the MC dropout method on the simulated features task from Sec. 5.2. We find (see Fig. 17) that it is unable to match the performance of our method, reinforcing the point that it is concerned mostly with other types of uncertainty.

represent the tldr for finding the corresponding scientific article

We present local ensembles, a method for detecting extrapolation in trained models, which approximates the variance of an ensemble using local-second order information.

Represent the TLDR for finding the corresponding scientific article

For classification problems with k classes, we show that the gradient tends to live in a tiny, slowly-evolving subspace spanned by the eigenvectors corresponding to the k-largest eigenvalues of the Hessian.