Question: Represent the scientific article to find a short TLDR

Question: Human reasoning involves recognising common underlying principles across many examples by utilising variables. The by-products of such reasoning are invariants that capture patterns across examples such as "if someone went somewhere then they are there" without mentioning specific people or places. Humans learn what variables are and how to use them at a young age, and the question this paper addresses is whether machines can also learn and use variables solely from examples without requiring human pre-engineering. We propose Unification Networks that incorporate soft unification into neural networks to learn variables and by doing so lift examples into invariants that can then be used to solve a given task. We evaluate our approach on four datasets to demonstrate that learning invariants captures patterns in the data and can improve performance over baselines. Humans have the ability to process symbolic knowledge and maintain symbolic thought . When reasoning, humans do not require combinatorial enumeration of examples but instead utilise invariant patterns with placeholders replacing specific entities. Symbolic cognitive models embrace this perspective with the human mind seen as an information processing system operating on formal symbols such as reading a stream of tokens in natural language. The language of thought hypothesis frames human thought as a structural construct with varying sub-components such as "X went to Y". By recognising what varies across examples, humans are capable of lifting examples into invariant principles that account for other instances. This symbolic thought with variables is learned at a young age through symbolic play . For instance a child learns that a sword can be substituted with a stick and engage in pretend play. Although variables are inherent in models of computation and symbolic formalisms, as in first-order logic , they are pre-engineered and used to solve specific tasks by means of unification or assignments that bound variables to given values. However, when learning from data only, being able to recognise when and which symbols should take on different values, i.e. symbols that can act as variables, is crucial for lifting examples into general principles that are invariant across multiple instances. Figure 1 shows the invariant learned by our approach: if someone is the same thing as someone else then they have the same colour. With this invariant, our approach can solve all of the training and test examples in task 16 of the bAbI dataset . In this paper we address the question of whether a machine can learn and use the notion of a variable, i.e. a symbol that can take on different values. For instance, given an example of the form "bernhard is a frog" the machine would learn that the token "bernhard" could be someone else and the token "frog" could be something else. If we consider unification a selection of the most appropriate value for a variable given a choice of values, we can reframe it as a form of attention. Attention models (; ;) allow neural networks to focus, attend to certain parts of the input often for the purpose of selecting a relevant portion. Since attention mechanisms are also differentiable they are often jointly learned within a task. This perspective motivates our idea of a unification mechanism that utilises attention and is therefore fully differentiable which we refer to as soft unification. Hence, we propose an end-to-end differentiable neural network approach for learning and utilising the notion of a variable that in return can lift examples into invariants used by the network to perform reasoning tasks. Specifically, we (i) propose a novel architecture capable of learning and using variables by lifting a given example through soft unification, (ii) present the empirical of our approach on four datasets and (iii) analyse the learned invariants that capture the underlying patterns present in the tasks. Our implementation using Chainer is publicly available at [link removed](anonymous link provided with submission). Reasoning with variables involves identifying what variables are, the setting in which they are used as well as the process by which they are assigned values. When the varying components, i.e. variables, of an example are identified, the remaining structure can be lifted into an invariant which then accounts for multiple other instances. Definition 1 (Variable). Given a set of symbols S, a variable X is defined as a pair X (x, s d) where s d ∈ S is the default symbol of the variable and x is a discrete random variable of which the support is S. The representation of a variable φ V (X:s d) is equal to the expected value of the corresponding random variable x given the default symbol s d: where For example, φ could be an embedding and φ V (X:s d) would become a weighted sum of symbol embeddings as in conventional attention models. The default symbol of a variable is intended to capture the variable's bound meaning following the idea of referants by. We denote variables using X, Y, A etc. such as X:bernhard where X is the name of the variable and bernhard the default symbol as shown in Figure 1. Definition 2 (Invariant). Given a structure (e.g. list, grid) G over S, an invariant is a pair I (G, ψ) where G ∈ G is the invariant example such as a tokenised story with tokens as symbols and ψ: S → is a function representing the degree to which the symbol is considered a variable. Thus, the final representation of a symbol s included in G, φ I (s) is: the linear interpolation between its representation φ(s) and its variable bound value with itself as the default symbol φ V (X:s). We adhere to the term invariant and refrain from mentioning rules, unground rules, etc. used in logicbased formalisms, e.g. Muggleton & de , since neither the invariant structure needs to be rule-like nor the variables carry logical semantics. This distinction is clarified in Section 6. Definition 3 (Unification). Given an invariant I and an example K ∈ G, unification binds the variables in I to symbols in K. Defined as a function g: I × G → G, unification binds variables by computing the probability mass functions, P in equation 1, and returns the unified representation using equation 2. The probability mass function of a variable X:s d is: where φ U: S → R d is the unifying feature of a symbol and φ U (K) ∈ R |K|×d is applied element wise to symbols in K. If g is differentiable, it is referred to as soft unification. We distinguish φ from φ U to emphasise that the unifying properties of the symbols might be different from their representations. For example, φ(bernhard) could represent a specific person whereas φ U (bernhard) the notion of someone. Overall soft unification incorporates 3 learnable components: φ, ψ, φ U which denote the base features, variableness and unifying features of a symbol respectively. Given an upstream, potentially task specific, network f: G → S, an invariant I ∈ I and an input example K ∈ G with a corresponding desired output a ∈ S, the following holds: where f now predicts based on the unified representation produced by g. In this work, we focus on g, the invariants it produces together with the interaction of f • g. Since soft unification is end-to-end differentiable, it can be incorporated into existing task-specific upstream architectures. We present 3 architectures that model f • g using multi-layer perceptrons (MLP), convolutional neural networks (CNN) and memory networks to demonstrate the flexibility of our approach. In all cases, the d dimensional representation of symbols are learnable embeddings φ(s) = O[s] T E with E ∈ R |S|×d randomly initialised by N and O[s] the one-hot encoding of the symbol. The variableness of symbols is a learnable weight ψ w (s) = σ(w s) where w ∈ R |S| and σ is the sigmoid function. We consider every symbol independently a variable irrespective of its surrounding context and leave further contextualised formulations as future work. The underlying intuition of this configuration is that a useful symbol for a correct prediction might need to take on other values for different inputs. This usefulness can be viewed as the inbound gradient to the corresponding w s parameter and ψ w (s) acting as a gate. For further model details including the size of the embeddings, please refer to Appendix A. Unification MLP (UMLP) (f : MLP, g: RNN) We combine soft unification into a multi-layer perceptron to process fixed length inputs. In this case, the structure G is a sequence of symbols with a fixed length l, e.g. a sequence of digits 4234. Given an embedded input φ(k) ∈ R l×d, the upstream MLP computes the output symbol based on the flattened representations f (φ(k)) = softmax(hE T) where h ∈ R d is the output of the last layer. However, to compute the unifying features φ U, definition 3, g uses a bi-directional GRU is the hidden state of the GRU at symbol k and W U ∈ R d×d is a learnable parameter. This model emphasises the flexibility around the boundary of f • g and that the unifying features can be computed in any differentiable manner. Unification CNN (UCNN) (f : CNN, g: CNN) Given a grid of embedded symbols φ(K) ∈ R w×h×d where w is the width and h the height, we use a convolutional neural network such that the final prediction is f (φ(K)) = softmax((W h + b)E T ) where h this time is the of global max pooling and W, b are learnable parameters. We also model g using a separate convolutional network with the same architecture as f and set φ U (k) = c 2 (relu(c 1 (k)) where c 1, c 2 are the convolutional layers. The grid is padded with 0s to obtain w × h × d after each convolution such that every symbol has a unifying feature. This model conveys how soft unification can be adapted to the specifics of the domain for example by using a convolution in a spatially structured input. Unification Memory Networks (UMN) (f : MemNN, g: RNN) Soft unification does not need to happen prior to f in a f • g fashion but can also be incorporated at any intermediate stage multiple times. To demonstrate this ability, we unify the symbols at different memory locations at each iteration of a Memory Network. Memory networks can handle a list of lists structure such as a tokenised story as shown in Figure 2. The memory network f uses the final hidden state of a bi-directional GRU (outer squares in Figure 2) as the sentence representations to compute a context attention. At each iteration, we unify the words between the attended sentences using the same approach in UMLP with another bi-directional GRU (inner diamonds in Figure 2) for unifying features φ U (bernhard) = W U Φ(bernhard). Following equation 2, the new unified representation of the memory slot is computed and f uses it to perform the next iteration. Concretely, g produces an unification tensor U ∈ R M ×m×N ×d where M and m is the number of sentences and words in the invariant respectively, and N is the number of sentences in the example such that after the context attentions are applied over M and N, we obtain φ(k) ∈ R m×d as the unified sentence at that iteration. Note that unlike in the UMLP case, the sentences can be of varying length. The prediction is then softmax(W h J I + b) where h J I is the hidden state of the invariant after J iterations. This setup, however, requires pre-training f such that the context attentions match the correct sentences. A task might contain different questions such as "Where is X?" and "Why did X go to Y?". To let the models differentiate between questions and potentially learn different invariants, we extend them with a repository of invariants I ∈ I and aggregate the predictions from each invariant. One simple approach is to sum the predictions of the invariants I∈I f • g(I, K) used in UMLP and UCNN. Another approach could be to use features from the invariants such as memory representations in the case of UMN. For UMN, we weigh the predictions using a bilinear attention η based on the hidden states at the first iteration h To initially form the repository of invariants, we use the bag-of-words representation of the questions and find the most dissimilar ones based on their cosine similarity as a heuristic to obtain varied examples. We use 4 datasets consisting of context, query and an answer (C, q, a): fixed length sequences of symbols, shapes of symbols in a grid, story based natural language reasoning with the bAbI dataset and logical reasoning represented as logic programs, examples shown in Table 1 with further samples in Appendix B. In each case we use an appropriate model: UMLP for fixed length sequences, UCNN for grid and UMN for iterative reasoning. We use synthetic datasets of which the data generating distributions are known to evaluate not only the quantitative performance but also the quality of the invariants learned by our approach. Fixed Length Sequences We generate sequences of length l = 4 with 8 unique symbols represented as digits to predict (i) a constant, (ii) the head of the sequence, (iii) the tail and (iv) the duplicate symbol. We randomly generate 1000 triples and then only take the unique ones to ensure the test split contains unseen examples. The training is then performed over a 5-fold cross-validation. Grid To spatially organise symbols, we generate a grid of size 3×3 with 8 unique symbols organised into 2 × 2 box of identical symbol, a vertical, diagonal or horizontal sequence of length 3, a cross or a plus shape and a triangle. In each task we predict (i) the identical symbol, (ii) the head of the sequence, (iii) the centre of the cross or plus and (iv) the corner of the triangle respectively. We follow the same procedure from sequences and randomly generate 1000 discarding duplicate triples. bAbI The bAbI dataset has become a standard benchmark for evaluating memory based networks. It consists of 20 synthetically generated natural language reasoning tasks (refer to for task details). We take the 1k English set and use 0.1 of the training set as validation. Each token is lower cased and considered a unique symbol. Following previous works , we take multiple word answers also to be a unique symbol in S. Logical Reasoning To demonstrate the flexibility of our approach and distinguish our notion of a variable from that used in logic based formalisms, we generate logical reasoning tasks in the form of logic programs using the procedure by. The tasks involve learning f (C, Q) = True ↔ C Q over 12 classes of logic programs exhibiting varying paradigms of logical reasoning including negation by failure . We generate 1k and 10k logic programs per task for training with 0.1 as validation and another 1k for testing. We set the arity of literals to 1 or 2 using one random character from the English alphabet for predicates and constants, e.g. p(p) and an upper case character for logical variables, e.g. p(X). We probe three aspects of soft unification: the impact of unification on performance over unseen data, the effect of multiple invariants and data efficiency. To that end, we train UMLP and UCNN with and without unification, UMN with pre-training using 1 or 3 invariants over either the entire training set or only 50 examples. Every model is trained 3 times via back-propagation using Adam on an Intel Core i7-6700 CPU using the following objective function: where L nll is the negative log-likelihood with sparsity regularisation over ψ at τ = 0.1 to discourage the models from utilising spurious number of variables. For UMLP and UCNN, we set λ K = 0, λ U = 1 for training just the unified output and the converse for the non-unifying versions. To pre-train the UMN, we start with λ K = 1, λ U = 0 for 40 epochs then set λ U = 1 to jointly train the unified output. For iterative tasks, the mean squared error between hidden states (h at each iteration j and, in the strongly supervised cases, the negative log-likelihood for the context attentions using provided supporting facts are also added to the objective function. Further details such as batch size and total number of epochs are available in Appendix C. Figure 3: Test accuracy over iterations for Unification MLP and Unification CNN models with 1 invariant versus no unification. We observe that with soft unification the models achieve higher accuracy with fewer iterations than their plain counterparts on both per task training sizes. Figure 3 portrays how soft unification generalises better to unseen examples in test sets -the same sequence or grid never appears in both the training and test sets as outlined in Section 4 -over plain models. Despite f • g having more trainable parameters than f alone, the models with unification not only maintain higher accuracy in each iteration and solve the tasks in as few as 250 iterations with ≤ 1000 training examples but also improve accuracy by ≈ 0.3 when trained with only ≤ 50 per task. We believe soft unification architecturally biases the models towards learning structural patterns which in return achieves better on recognising common patterns of symbols across examples. Results with multiple invariants are identical and the models seem to ignore the extra invariants due to the fact that the tasks can be solved with a single invariant and the regularisation applied on ψ zeroing out unnecessary invariants; further in Appendix D. The fluctuations in accuracy around iterations 750 to 1000 in UCNN are also caused by penalising ψ which forces the model to relearn the task with less variables half way through training. Following Tables 2 and 3, we observe a trend of better performance through strong supervision, more data per task and using only 1 invariant. We believe strong supervision aids with selecting the correct sentences to unify and in a weak setting the model attempts to unify arbitrary context sentences often failing to follow the iterative reasoning chain. The increase in performance with more data and strong supervision is consistent with previous work reflecting how f • g can be bounded by the efficacy of f modelled as a memory network. As a , only in the supervised case do we observe a minor improvement over MemNN by 0.7 in Table 2 and no improvement in the weak case over comparable memory based networks in Table 3 . This dependency on f also limits the ability of f • g to learn from 50 examples per task failing 17/20 and 12/12 of bAbI and logical reasoning tasks respectively. The increase in error rate with 3 invariants, we speculate, stems from having more parameters and more pathways in the model rendering training more difficult. 6 ANALYSIS After training, we can extract the learned invariants by applying a threshold on ∀s ∈ S : ψ(s) > t indicating whether a symbol is used as a variable or not. We set t = 0.0 for all datasets except for bAbI, we use t = 0.1. The magnitude of this threshold seems to depend on the amount of regularisation τ, equation 5, and the number of training steps along with batch size all controlling how much ψ is pushed towards 0. Sample invariants shown in Figure 4 describe the common patterns present in the tasks with parts that contribute towards the final answer becoming variables. Extra symbols such as is or travelled do not emerge as variables, as shown in Figure 4a; we attribute this behaviour to the fact that changing the token travelled to went does not influence the prediction but changing the action, the value of Z:left to'picked' does. However, based on random initialisation, our approach can convert an arbitrary symbol into a variable and let f compensate for the unifications it produces. For example, the invariant "X:8 5 2 2" could predict the tail of another example by unifying the head with the tail using φ U, equation 3, of those symbols. Further examples are shown in Appendix D. Pre-training f as done in UMN seems to produce more robust and consistent invariants compared to immediately training f • g since, we speculate, by equation 4 f might encourage g(I, K) ≈ K. Interpretability versus Ability A desired property of interpretable models is transparency . A novel outcome of the learned invariants in our approach is that they provide an approximation of the underlying general principle present in the data such as the structure of multi-hop reasoning shown in Figure 4e. However, certain aspects regarding the ability of the model such as how it performs temporal reasoning, are still hidden inside f. In Figure 4b, although we observe Z:morning as a variable, the overall learned invariant captures nothing about how changing the value of Z:morning alters the behaviour of f. The model might look before or after a certain time point (e) Logical reasoning task 5 with arity 1. The model captures how S:n could entail X:i in a chain. Figure 4: Invariants learned across the four datasets using the three architectures. For iterative reasoning datasets, bAbI and logical reasoning, they are taken from strongly supervised UMN. X:bill went somewhere depending what Z:morning binds to. Without the regularising term on ψ(s), we initially noticed the models using, one might call extra, symbols as variables and binding them to the same value occasionally producing unifications such as "bathroom bathroom to the bathroom" and still f predicting, perhaps unsurprisingly, the correct answer as bathroom. Hence, regularising ψ with the correct amount τ in equation 5 to reduce the capacity of unification seems critical in extracting not just any invariant but one that represents the common structure. Soft unification from equation 3 reveals three main patterns: one-to-one, one-to-many or many-toone bindings as shown in Figure 5; further examples are in Appendix D. Figure 5a captures what one might expect unification to look like where variables unify with their corresponding counterparts, e.g. X:bernhard with brian and Y:frog with lion. However, occasionally the model can optimise to use less variables and squeeze the required information into a single variable, for example by binding Y:bathroom to john and kitchen as shown in Figure 5b. We believe this occurs due to the sparsity constraint on ψ(s) encouraging the model to be as conservative as possible. Finally, if there are more variables than needed as in Figure 5c, we observe a many-to-one binding with Y:w and Z:e mapping to the same constant q. This behaviour begs the question how does the model differentiate between p(q) and p(q, q). We speculate the model uses the magnitude of ψ(w) = 0.037 and ψ(e) = 0.042 to encode the difference despite both variables unifying with the same constant. Learning an underlying general principle in the form of an invariant is often the means for arguing generalisation in neural networks. For example, Neural Turing Machines are tested on previously unseen sequences to support the view that the model might have captured the underlying pattern or algorithm. In fact, claim "MemNNs can discover simple linguistic patterns based on verbal forms such as (X, dropped, Y), (X, took, Y) or (X, journeyed to, Y) and can successfully generalise the meaning of their instantiations." However, this claim is based on the output of f and unfortunately it is unknown whether the model has truly learned such a representation or indeed is utilising it. Our approach sheds light to this ambiguity and presents these linguistic patterns explicitly as invariants ensuring their utility through g without solely analysing the output of f on previously unseen symbols. Although we associate these invariants with our existing understanding of the task to mistakenly anthropomorphise the machine, for example by thinking it has learned X:mary as someone, it is important to acknowledge that these are just symbolic patterns. In these cases, our interpretations do not necessarily correspond to any understanding of the machine, relating to the Chinese room argument made by. Learning invariants by lifting ground examples is related to least common generalisation by which inductive inference is performed on facts such as generalising went(mary,kitchen) and went(john,garden) to went(X,Y). Unlike in a predicate logic setting, our approach allows for soft alignment and therefore generalisation between varying length sequences. Existing neuro-symbolic systems focus on inducing rules that adhere to given logical semantics of what variables and rules are. For example, δILP constructs a network by rigidly following the given semantics of first-order logic. Similarly, Lifted Relational Neural Networks ground first-order logic rules into a neural network while Neural Theorem Provers build neural networks using backwardchaining on a given knowledge base with templates. However, the notion of a variable is pre-defined rather than learned with a focus on presenting a practical approach to solving certain problems, whereas our motivation stems from a cognitive perspective. At first it may seem the learned invariants, Section 6, make the model more interpretable; however, this transparency is not of the model f but of the data. The invariant captures patterns that potentially approximates the data generating distribution but we still do not know how the model f uses them upstream. Thus, from the perspective of explainable artificial intelligence (XAI) , learning invariants or interpreting them do not constitute an explanation of the reasoning model f even though "if someone goes somewhere then they are there" might look like one. Instead, it can be perceived as causal attribution in which someone being somewhere is attributed to them going there. This perspective also relates to gradient based model explanation methods such as Layer-Wise Relevance Propagation and Grad-CAM . Consequently, a possible view on ψ, equation 2, is a gradient based usefulness measure such that a symbol utilised upstream by f to determine the answer becomes a variable similar to how a group of pixels in an image contribute more to its classification. Finally, one can argue that our model maintains a form of counterfactual thinking in which soft unification g creates counterfactuals on the invariant example to alter the output of f towards the desired answer, equation 4. The question where Mary would have been if Mary had gone to the garden instead of the kitchen is the process by which an invariant is learned through multiple examples during training. This view relates to methods of causal inference in which counterfactuals are vital as demonstrated in structured models by. We presented a new approach for learning variables and lifting examples into invariants through the usage of soft unification. Evaluating on four datasets, we analysed how Unification Networks perform comparatively to existing similar architectures while having the benefit of lifting examples into invariants that capture underlying patterns present in the tasks. Since our approach is end-toend differentiable, we plan to apply this technique to multi-modal tasks in order to yield multi-modal invariants for example in visual question answering. A MODEL DETAILS Unification MLP (UMLP) To model f as a multi-layer perceptron, we take symbol embeddings of size d = 16 and flatten sequences of length l = 4 into an input vector of size φ(k) ∈ R 64. The MLP consists of 2 hidden layers with tanh non-linearity of sizes 2d and d respectively. To process the query, we concatenate the one-hot encoding of the task id to φ(k) yielding a final input of size 64 + 4 = 68. For unification features φ U, we use a bi-directional GRU with hidden size d and an initial state of 0. The hidden state at each symbol is taken with a linear transformation to give φ U (s) = W U Φ(s) where Φ(s) is the hidden state of the biGRU. The variable assignment is then computed as an attention over the according to equation 3. Unification CNN (UCNN) We take symbols embeddings of size d = 32 to obtain an input grid φ(K) ∈ R 3×3×32. Similar to UMLP, for each symbol we append the task id as a one-hot vector to get an input of shape 3 × 3 × (32 + 4). Then f consists of 2 convolutional layers with d filters each, kernel size of 3 and stride 1. We use relu non-linearity in between the layers. We pad the grid with 2 columns and 2 rows to a 5 × 5 such that the output of the convolutions yield again a hidden output H ∈ R 3×3×d of the same shape. As the final hidden output h, we take a global max pool to over H to obtain h ∈ R d. Unification function g is modelled identical to f without the max pooling such that φ U (K ij) = H ij where H is the hidden output of the convolutional layers. Unlike previous architectures, with UMN we interleave g into f. We use embedding sizes of d = 32 and model f with an iterative memory network. We take the final hidden state of a bi-directional GRU, with initial state 0, Φ M to represent the sentences of the context C and query q in a ddimensional vector M i = Φ M (C i) and the query m q = Φ M (q). The initial state of the memory network is h 0 = m q. At each iteration j: where Φ A is another d-dimensional bi-directional GRU and ρ(x, y) = [x; y; x y; (x − y) 2 ] with the element-wise multiplication and [;] the concatenation of vectors. Taking β j as the context attention, we obtain the next state of the memory network: and iterate J many times in advance. The final prediction becomes f (C, q) = softmax(W h J + b). All weight matrices W and bias vectors b are independent but are tied across iterations. B GENERATED DATASET SAMPLES Table 5: Training sizes for randomly generated fixed length sequences and grid tasks with 8 unique symbols. The reason for Grid task (i) to be smaller is because there are at most 32 combinations of 2 × 2 boxes in a 3 × 3 grid with 8 unique symbols. Task Sequences Grid i 704.7 ± 12.8 25.6 ± 1.8 ii 709.4 ± 13.8 623.7 ± 14.1 iii 709.7 ± 14.0 768.2 ± 12.5 iv 624.8 ± 12.4 795.2 ± 10.3 C TRAINING DETAILS C.1 UNIFICATION MLP & CNN Both unification models are trained on a 5-fold cross-validation over the generated datasets for 2000 iterations with a batch size of 64. We don't use any weight decay and save the training and test accuracies every 10 iterations, as presented in Figure 3. We again use a batch size of 64 and pre-train f for 40 epochs then f together with g for 260 epochs. We use epochs for UMN since the dataset sizes are fixed. To learn g alongside f, we combine error signals from the unification of the invariant and the example. Following equation 4, the objective function not only incorporates the negative log-likelihood L nll of the answer but also the mean squared error between intermediate states h j I and h j K at each iteration as an auxiliary loss: We pre-train by setting λ U = 0 for 40 epochs and then set λ U = 1. For strong supervision we also compute the negative log-likelihood L nll for the context attention β j, described in Appendix A, at each iteration using the supporting facts of the tasks. We apply a dropout of 0.1 for all recurrent neural networks used and only for the bAbI dataset weight decay with 0.001 as the coefficient. Figure 6: Results of Unification MLP and CNN on increasing number of invariants. There is no impact on performance when more invariants per task are given. Upon closer inspection, we noticed the models ignore the extra invariants and only use 1. We speculate the regularisation ψ encourages the models to use a single 1 invariant. Xsandra went back to the Y:bathroom is X:sandra in the Y:bathroom yes Figure 9: Invariants learned that do not match the data generating distribution from UMLP and UCNN using ≤ 1000 examples to train. In these instances the unification still bind to the the correct symbols in order to predict the desired answer; quantitatively we get the same . Variable default symbols are omitted for clarity. (a) bAbI task 2. When a variable is unused in the next iteration, e.g. Z:football, it unifies with random tokens often biased by position. (b) Logical reasoning task 1. A one-to-one alignment is created between predicates and constants. (c) Logical reasoning task 3. Arity 1 atom forced to bind with arity 2 creates a one-to-many mapping.

Represent the TLDR for finding the corresponding scientific article

End-to-end learning of invariant representations with variables across examples such as if someone went somewhere then they are there.

We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models


Question: Represent the scientific article to find a short TLDR

Question: We propose a new learning-based approach to solve ill-posed inverse problems in imaging. We address the case where ground truth training samples are rare and the problem is severely ill-posed---both because of the underlying physics and because we can only get few measurements. This setting is common in geophysical imaging and remote sensing. We show that in this case the common approach to directly learn the mapping from the measured data to the reconstruction becomes unstable. Instead, we propose to first learn an ensemble of simpler mappings from the data to projections of the unknown image into random piecewise-constant subspaces. We then combine the projections to form a final reconstruction by solving a deconvolution-like problem. We show experimentally that the proposed method is more robust to measurement noise and corruptions not seen during training than a directly learned inverse. A variety of imaging inverse problems can be discretized to a linear system y = Ax + η where y ∈ R M is the measured data, A ∈ R M ×N is the imaging or forward operator, x ∈ X ⊂ R N is the object being probed by applying A (often called the model), and η is the noise. Depending on the application, the set of plausible reconstructions X could model natural, seismic, or biomedical images. In many cases the ing inverse problem is ill-posed, either because of the poor conditioning of A (a consequence of the underlying physics) or because M N.A classical approach to solve ill-posed inverse problems is to minimize an objective functional regularized via a certain norm (e.g. 1, 2, total variation (TV) seminorm) of the model. These methods promote general properties such as sparsity or smoothness of reconstructions, sometimes in combination with learned synthesis or analysis operators, or dictionaries BID44 ).In this paper, we address situations with very sparse measurement data (M N) so that even a coarse reconstruction of the unknown model is hard to get with traditional regularization schemes. Unlike artifact-removal scenarios where applying a regularized pseudoinverse of the imaging operator already brings out considerable structure, we look at applications where standard techniques cannot produce a reasonable image (Figure 1). This highly unresolved regime is common in geophysics and requires alternative, more involved strategies BID12 ).An appealing alternative to classical regularizers is to use deep neural networks. For example, generative models (GANs) based on neural networks have recently achieved impressive in regularization of inverse problems BID7, BID29 ). However, a difficulty in geophysical applications is that there are very few examples of ground truth models available for training (sometimes none at all). Since GANs require many, they cannot be applied to such problems. This suggests to look for methods that are not very sensitive to the training dataset. Conversely, it means that the sought reconstructions are less detailed than what is expected in data-rich settings; for Figure 1: We reconstruct an image x from its tomographic measurements. In moderately ill-posed problems, conventional methods based on the pseudoinverse and regularized non-negative least squares (x ∈ N, N is image dimension) give correct structural information. In fact, total variation (TV) approaches give very good . A neural network BID23 ) can be trained to directly invert and remove the artifacts (NN). In a severely ill-posed problem on the other hand (explained in FIG2) with insufficient ground truth training data, neither the classical techniques nor a neural network recover salient geometric features.an example, see the reconstructions of the Tibetan plateau BID51 ).In this paper, we propose a two-stage method to solve ill-posed inverse problems using random low-dimensional projections and convolutional neural networks. We first decompose the inverse problem into a collection of simpler learning problems of estimating projections into random (but structured) low-dimensional subspaces of piecewise-constant images. Each projection is easier to learn in terms of generalization error BID10 ) thanks to its lower Lipschitz constant. In the second stage, we solve a new linear inverse problem that combines the estimates from the different subspaces. We show that this converts the original problem with possibly non-local (often tomographic) measurements into an inverse problem with localized measurements, and that in fact, in expectation over random subspaces the problem becomes a deconvolution. Intuitively, projecting into piecewise-constant subspaces is equivalent to estimating local averages-a simpler problem than estimating individual pixel values. Combining the local estimates lets us recover the underlying structure. We believe that this technique is of independent interest in addressing inverse problems. We test our method on linearized seismic traveltime tomography BID8 BID20 ) with sparse measurements and show that it outperforms learned direct inversion in quality of achieved reconstructions, robustness to measurement errors, and (in)sensitivity to the training data. The latter is essential in domains with insufficient ground truth images. Although neural networks have long been used to address inverse problems BID32 BID21; BID42 ), the past few years have seen the number of related deep learning papers grow exponentially. The majority address biomedical imaging BID16; BID22 ) with several special issues 1 and review papers BID28 BID30 ) dedicated to the topic. All these papers address reconstruction from subsampled or low-quality data, often motivated by reduced scanning time or lower radiation doses. Beyond biomedical imaging, machine learning techniques are emerging in geophysical imaging BID3; BID25; BID5 ), though at a slower pace, perhaps partly due to the lack of standard open datasets. Existing methods can be grouped into non-iterative methods that learn a feed-forward mapping from the measured data y (or some standard manipulation such as adjoint or a pseudoinverse) to the model Figure 2: Regularization by Λ random projections: 1) each orthogonal projection is approximated by a convolutional neural network which maps from a non-negative least squares reconstruction of an image to its projection onto a lower dimension subspace of Delaunay triangulations; 2) projections are combined to estimate the original image using regularized least squares.either the regularizer being a neural network BID26 ), or neural networks replacing various iteration components such as gradients, projectors, or proximal mappings BID24; BID1 a); BID9 ). These are further related to the notion of plug-and-play regularization BID47 ), as well as early uses of neural nets to unroll and adapt standard sparse reconstruction algorithms BID14; BID50 ). An advantage of the first group of methods is that they are fast; an advantage of the second group is that they are better at enforcing data consistency. Generative models A rather different take was proposed in the context of compressed sensing where the reconstruction is constrained to lie in the range of a pretrained generative network BID6 ). Their scheme achieves impressive on random sensing operators and comes with theoretical guarantees. However, training generative networks requires many examples of ground truth and the method is inherently subject to dataset bias. Here, we focus on a setting where ground-truth samples are very few or impossible to obtain. There are connections between our work and sketching BID15; BID35 ) where the learning problem is also simplified by random low-dimensional projections of some object-either the data or the unknown reconstruction itself BID54 ). This also exposes natural connections with learning via random features BID39 ). The two stages of our method are (i) decomposing a "hard" learning task of directly learning an unstable operator into an ensemble of "easy" tasks of estimating projections of the unknown model into low-dimensional subspaces; and (ii) combining these projection estimates to solve a reformulated inverse problem for x. The two stages are summarized in Figure 2. While our method is applicable to continuous and non-linear settings, we focus on linear finite-dimensional inverse problems. Statistical learning theory tells us that the number of samples required to learn an M -variate LLipschitz function to a given sup-norm accuracy is O(L M) BID10 ). While this is proved for scalar-valued multivariate maps, it is reasonable to expect the same scaling in L to hold for vector-valued maps. This motivates us to study Lipschitz properties of the projected inverse maps. We wish to reconstruct x, an N -pixel image from X ⊂ R N where N is large (we think of x as an √ N × √ N discrete image). We assume that the map from x ∈ X to y ∈ R M is injective so that it is invertible on its range, and that there exists an L-Lipschitz (generally non-linear) inverse G, DISPLAYFORM0 In order for the injectivity assumption to be reasonable, we assume that X is a low-dimensional manifold embedded in R N of dimension at most M, where M is the number of measurements. Since we are in finite dimension, injectivity implies the existence of L BID45 ). Due to ill-posedness, L is typically large. Consider now the map from the data y to a projection of the model x into some K-dimensional subspace S, where K N. Note that this map exists by construction (since A is injective on X), and that it must be non-linear. To see this, note that the only consistent 2 linear map acting on y is an oblique, rather than an orthogonal projection on S (cf. Section 2.4 in BID48). We explain this in more detail in Appendix A.Denote the projection by P S x and assume S ⊂ R N is chosen uniformly at random. 3 We want to evaluate the expected Lipschitz constant of the map from y to P S x, noting that it can be written as P S • G: DISPLAYFORM1 where the first inequality is Jensen's inequality, and the second one follows from DISPLAYFORM2 and the observation that E P S P S = K N I N. In other words, random projections reduce the Lipschitz constant by a factor of K/N on average. Since learning requires O(L K) samples, this allows us to work with exponentially fewer samples and makes the learning task easier. Conversely, given a fixed training dataset, it gives more accurate estimates. The above example uses unstructured random subspaces. In many inverse problems, such as inverse scattering BID4; Di Cristo and Rondi FORMULA8 ), a judicious choice of subspace family can give exponential improvements in Lipschitz stability. Particularly, it is favorable to use piecewiseconstant images: x = K k=1 x k χ k, with χ k being indicator functions of some domain subset. Motivated by this observation, we use piecewise-constant subspaces over random Delaunay triangle meshes. The Delaunay triangulations enjoy a number of desirable learning-theoretic properties. For function learning it was shown that given a set of vertices, piecewise linear functions on Delaunay triangulations achieve the smallest sup-norm error among all triangulations .We sample Λ sets of points in the image domain from a uniform-density Poisson process and construct Λ (discrete) Delaunay triangulations with those points as vertices. Let S = {S λ | 1 ≤ λ ≤ Λ} be the collection of Λ subspaces of piecewise-constant functions on these triangulations. Let further G λ be the map from y to the projection of the model into subspace S λ, G λ y = P S λ x. Instead of learning the "hard" inverse mapping G, we propose to learn an ensemble of simpler mappings {G λ} Λ λ=1. We approximate each G λ by a convolutional neural network, Γ θ(λ) (y): R N → R N, parameterized by a set of trained weights θ(λ). Similar to Jin et al. FORMULA3, we do not use the measured data y ∈ R M directly as this would require the network to first learn to map y back to the image domain; we rather warm-start the reconstruction by a non-negative least squares reconstruction, y ∈ R N, computed from y. The weights are chosen by minimizing empirical risk: DISPLAYFORM0 where DISPLAYFORM1 is a set of J training models and non-negative least squares measurements. By learning projections onto random subspaces, we transform our original problem into that of estimating DISPLAYFORM0. To see how this can be done, ascribe to the columns of B λ ∈ 2 Consistent meaning that if x already lives in S, then the map should return x.3 One way to construct the corresponding projection matrix is as P S = W W †, where W ∈ R N ×K is a matrix with standard iid Gaussian entries. N ×K a natural orthogonal basis for the subspace S λ, B λ = [χ λ,1, . . ., χ λ,K], with χ λ,k being the indicator function of the kth triangle in mesh λ. Denote by q λ def = q λ (y) the mapping from the data y to an estimate of the expansion coefficients of x in the basis for S λ: DISPLAYFORM0.., q Λ ∈ R KΛ; then we can estimate x using the following reformulated problem: DISPLAYFORM1 and the corresponding regularized reconstruction: DISPLAYFORM2 with ϕ(x) chosen as the TV-seminorm x TV. The regularization is not essential. As we show experimentally, if KΛ is sufficiently large, ϕ(x) is not required. Note that solving the original problem directly using x TV regularizer fails to recover the structure of the model (Figure 1). Since the true inverse map G has a large Lipschitz constant, it would seem reasonable that as the number of mesh subspaces Λ grows large (and their direct sum approaches the whole ambient space R N), the Lipschitz properties of G should deteriorate as well. Denote the unregularized inverse mapping in y → x by G. Then we have the following estimate: DISPLAYFORM0 with σ min (B) the smallest (non-zero) singular value of B and L K the Lipschitz constant of the stable projection mappings q λ. Indeed, we observe empirically that σ min (B) −1 grows large as the number of subspaces increases which reflects the fact that although individual projections are easier to learn, the full-resolution reconstruction remains ill-posed. Estimates of individual subspace projections give correct local information. They convert possibly non-local measurements (e.g. integrals along curves in tomography) into local ones. The key is that these local averages (subspace projection coefficients) can be estimated accurately (see Section 4).To further illustrate what we mean by correct local information, consider a simple numerical experiment with our reformulated problem, q = B T x, where x is an all-zero image with a few pixels "on". For the sake of clarity we assume the coefficients q are perfect. Recall that B is a block matrix comprising Λ subspace bases stacked side by side. It is a random matrix because the subspaces are generated at random, and therefore the reconstruction x = (B) † q is also random. We approximate E x by simulating a large number of Λ-tuples of meshes and averaging the obtained reconstructions. Results are shown in FIG1 for different numbers of triangles per subspace, K, and subspaces per reconstruction, Λ. As Λ or K increase, the expected reconstruction becomes increasingly localized around non-zero pixels. The following proposition (proved in Appendix B) tells us that this phenomenon can be modeled by convolution. 4 Proposition 1. Let x be the solution to q = B x given as (B) † q. Then there exists a kernel κ(u), with u a discrete index, such that E x = x * κ. Furthermore, κ(u) is isotropic. While FIG1 suggests that more triangles are better, we note that this increases the subspace dimension which makes getting correct projection estimates harder. Instead we choose to stack more meshes with a smaller number of triangles. Intuitively, since every triangle average depends on many measurements, estimating each average is more robust to measurement corruptions as evidenced in Section 4. Accurate estimates of local averages enable us to recover the geometric structure while being more robust to data errors. 4 NUMERICAL To demonstrate our method's benefits we consider linearized traveltime tomography BID20; BID8 ), but we note that the method applies to any inverse problem with scarce data. In traveltime tomography, we measure N 2 wave travel times between N sensors as in FIG2. Travel times depend on the medium property called slowness (inverse of speed) and the task is to reconstruct the spatial slowness map. Image intensities are a proxy for slowness maps-the lower the image intensity the higher the slowness. In the straight-ray approximation, the problem data is modeled as integral along line segments: DISPLAYFORM0 where x: R 2 → R + is the continuous slowness map and s i, s j are sensor locations. In our experiments, we use a 128 × 128 pixel grid with 25 sensors (300 measurements) placed uniformly in an inscribed circle, and corrupt the measurements with zero-mean iid Gaussian noise. We generate random Delaunay meshes each with 50 triangles. The corresponding projector matrices compute average intensity over triangles to yield a piecewise constant approximation P S λ x of x. We test two distinct architectures: (i) ProjNet, tasked with estimating the projection into a single subspace; and (ii) SubNet, tasked with estimating the projection over multiple subspaces. The ProjNet architecture is inspired by the FBPConvNet BID23 ) and the U-Net BID41 ) as shown in Figure 11a in the appendix. Crucially, we constrain the network output to live in S λ by fixing the last layer of the network to be a projector, P S λ (Figure 11a). A similar trick in a different context was proposed in BID43 ).We combine projection estimates from many ProjNets by regularized linear least-squares to get the reconstructed model (cf. Figure 2) with the regularization parameter λ determined on five held-out images. A drawback of this approach is that a separate ProjNet must be trained for each subspace. This motivates the SubNet (shown in Figure 11b). Each input to SubNet is the concatenation of a non-negative least squares reconstruction and 50 basis functions, one for each triangle forming a 51-channel input. This approach scales to any number of subspaces which allows us to get visually smoother reconstructions without any further regularization as in. On the other hand, the projections are less precise which can lead to slightly degraded performance. As a quantitative figure of merit we use the signal-to-noise ratio (SNR). The input SNR is defined as 10 log 10 (σ 2 signal /σ 2 noise) where σ 2 signal and σ 2 noise are the signal and noise variance; the output SNR is defined as sup a,b 20 log 10 (x 2 / x − ax − b 2) with x the ground truth andx the reconstruction.130 ProjNets are trained for 130 different meshes with measurements at various SNRs. Similarly, a single SubNet is trained with 350 different meshes and the same noise levels. We compare the ProjNet and SubNet reconstructions with a direct U-net baseline convolutional neural network that reconstructs images from their non-negative least squares reconstructions. The direct baseline has the same architecture as SubNet except the input is a single channel non-negative least squares reconstruction like in ProjNet and the output is the target reconstruction. Such an architecture was proposed by BID23 ) and is used as a baseline in recent learning-based inverse problem works BID29; Ye et al. FORMULA3 ) and is inspiring other architectures for inverse problems BID2 ). We pick the best performing baseline network from multiple networks which have a comparable number of trainable parameters to SubNet. We simulate the lack of training data by testing on a dataset that is different than that used for training. Robustness to corruption To demonstrate that our method is robust against arbitrary assumptions made at training time, we consider two experiments. First, we corrupt the data with zero-mean iid Gaussian noise and reconstruct with networks trained at different input noise levels. In FIG3 and Table 1, we summarize the with reconstructions of geo images taken from the BP2004 dataset 6 and x-ray images of metal castings BID31 ). The direct baseline and SubNet are trained on a set of 20,000 images from the arbitrarily chosen LSUN bridges dataset BID53 ) and tested with the geophysics and x-ray images. ProjNets are trained with 10,000 images from the LSUN dataset. Our method reports better SNRs compared to the baseline. We note that direct reconstruction is unstable when trained on clean and tested on noisy measurements as it often hallucinates details that are artifacts of the training data. For applications in geophysics it is important that our method correctly captures the shape of the cavities unlike the direct inversion which can produce sharp but wrong geometries (see outlines in FIG3). with Gaussian noise FIG3 ) the direct method completely fails to recover coarse geometry in all test cases. In our entire test dataset of 102 x-ray images there is not a single example where the direct network captures a geometric feature that our method misses. This demonstrates the strengths of our approach. For more examples of x-ray images please see Appendix E. FIG4 illustrates the influence of the training data on reconstructions. Training with LSUN, CelebA BID27 ) and a synthetic dataset of random overlapping shapes (see FIG3 in Appendix for examples) all give comparable reconstructions-a desirable property in applications where real ground truth is unavailable. We complement our with reconstructions of checkerboard phantoms (standard resolution tests) and x-rays of metal castings in Figure 7. We note that in addition to better SNR, our method produces more accurate geometry estimates, as per the annotations in the figure. We proposed a new approach to regularize ill-posed inverse problems in imaging, the key idea being to decompose an unstable inverse mapping into a collection of stable mappings which only estimate Figure 7: Reconstructions on checkerboards and x-rays with 10dB measurement SNR tested on 10dB trained networks. Red annotations highlight where the direct net fails to reconstruct correct geometry. low-dimensional projections of the model. By using piecewise-constant Delaunay subspaces, we showed that the projections can indeed be accurately estimated. Combining the projections leads to a deconvolution-like problem. Compared to directly learning the inverse map, our method is more robust against noise and corruptions. We also showed that regularizing via projections allows our method to generalize across training datasets. Our reconstructions are better both quantitatively in terms of SNR and qualitatively in the sense that they estimate correct geometric features even when measurements are corrupted in ways not seen at training time. Future work involves getting precise estimates of Lipschitz constants for various inverse problems, regularizing the reformulated problem using modern regularizers BID46 ), studying extensions to non-linear problems and developing concentration bounds for the equivalent convolution kernel. This work utilizes resources supported by the National Science Foundation's Major Research Instrumentation program, grant #1725729, as well as the University of Illinois at Urbana-Champaign. We gratefully acknowledge the support of NVIDIA Corporation with the donation of one of the GPUs used for this research. We explain the need for non-linear operators even in the absence of noise with reference to FIG5. Projecting x into a given known subspace is a simple linear operation, so it may not be a priori clear why we use non-linear neural networks to estimate the projections. Alas, we do not know x and only have access to y. Suppose that there exists a linear operator (a matrix) F ∈ R N ×M which acts on y and computes the projection of x on S λ. A natural requirement on F is consistency: if x already lives in S λ, then we would like to have F Ax = x. This implies that for any x, not necessarily in S λ, we require F AF Ax = F Ax which implies that F A = (F A)2 is an idempotent operator. Letting the columns of B λ be a basis for S λ, it is easy to see that the least squares minimizer for F is B λ (AB λ) †. However, because R(F) = S λ = R(A *) (A * is the adjoint of A, simply a transpose for real matrices), in general it will not hold that (F A) * = F A. Thus, F A is an oblique, rather than orthogonal projection into S. In FIG5 this corresponds to the point P oblique S λ x which can be arbitrarily far from the orthogonal projection P ortho S λ x. The nullspace of the oblique projection is precisely N (A) = R(A *) ⊥.Thus consistent linear operators can at best yield oblique projections which can be far from the orthogonal one. One could also see this geometrically from FIG5. As the angle between S λ and R(A *) increases to π/2 the oblique projection point travels to infinity (note that the oblique projection always happens along the nullspace of A, which is the line orthogonal to P R(A *). Since our subspaces are chosen at random, in general they are not aligned with R(A *). The only subspace on which we can linearly compute an orthogonal projection from y is R(A *); this is given by the Moore-Penrose pseudoinverse. Therefore, to get the orthogonal projection onto random subspaces, we must use non-linear operators. More generally, for any other ad hoc linear reconstruction operator W, W y = W Ax always lives in the column space of W A which is a subspace whose dimension is at most the number of rows of A. However, we do not have any linear subspace model for x. As shown in the right half of FIG5, as soon as A is injective on X, the existence of this non-linear map is guaranteed by construction: since y determines x, it also determines P S λ x. We show the of numerical experiments in Figures 9 and 10 which further illustrate the performance difference between linear oblique projectors and our non-linear learned operator when estimating the projection of an image into a random subspace. We refer the reader to the captions below each figure for more details. Figure 10: We try hard to get the best reconstruction from the linear approach. SNRs are indicated in the bottom-left of each reconstruction. In the linear approach, coefficients are obtained using the linear oblique projection method. Once coefficients are obtained, they are non-linearly reconstructed according to. Both linear approach reconstructions use the box-constraint (BC) mentioned in. For the 130 subspace reconstruction total-variation (TV) regularization is also used. Therefore, once the coefficients are obtained using the linear approach, the reconstruction of the final image is done in an identical manner as ProjNet for 130 subspaces and SubNet for 350 subspaces. To give the linear approach the best chance we also optimized hyperparameters such as the regularization parameter to give the highest SNR.Using the definition of the inner product and rearranging, we get DISPLAYFORM0. Now, the probability distribution of triangles around any point u is both shift-and rotation-invariant because a Poisson process in the plane is shift-and rotation-invariant. It follows that E κ(u, v) = κ(u − v) for some κ, meaning that DISPLAYFORM1 which is a convolution of the original model with a rotationally invariant (isotropic) kernel. Figure 11 explains the network architecture used for ProjNet and SubNet. The network consists of a sequence of downsampling layers followed by upsampling layers, with skip connections BID19 a) ) between the downsampling and upsampling layers. Each ProjNet output is constrained to a single subspace by applying a subspace projection operator, P S λ. We train 130 such networks and reconstruct from the projection estimate using. SubNet is a single network that is trained over multiple subspaces. To do this, we change its input to be [y B λ]. Moreover, we apply the same projection operator as ProjNet to the output of the SubNet. Each SubNet is trained to give projection estimates over 350 random subspaces. This approach allows us to scale to any number of subspaces without training new networks for each. Moreover, this allows us to build an over-constrained system q = Bx to solve. Even though SubNet has almost as many parameters as the direct net, reconstructing via the projection estimates allows SubNet to get higher SNR and more importantly, get better estimates of the coarse geometry than the direct inversion. All networks are trained with the Adam optimizer. projection into input subspace Figure 11: a) ProjNet architecture; b) SubNet architecture. In both cases, the input is a non-negative least squares reconstruction and the network is trained to reconstruct a projection into one subspace. In SubNet, the subspace basis is concatenated to the non-negative least squares reconstruction. We showcase more reconstructions on actual geophysics images taken from the BP2004 dataset in Figure 12. Note that all networks were trained on the LSUN bridges dataset. We show additional reconstructions for the largest corruption case, p = 1 8, for x-ray images (FIG1) and geo images FIG2. Our method consistently has better SNR. More importantly we note that there is not a single instance where the direct reconstruction gets a feature that our methods do not. In a majority of instances, the direct network misses a feature of the image. This is highly undesirable in settings such as geophysical imaging. The shapes dataset was generated using random ellipses, circle and rectangle patches. See FIG3 for examples. This dataset was used in FIG4. In Section 4 we train multiple ProjNets, each focusing on a different low-dimensional subspace. Here we train an ensemble of direct networks where each network is as described in Section 4.1.1 and evaluate the robustness of a method where the outputs of these networks are averaged to give a final reconstruction. Once again, we consider scenarios where the model is trained with data at a particular noise level and then tested with data at a different noise level and with erasures that were unseen during training time. We show that our proposed method is more robust to changes in the test scenario. In FIG4, we consider the erasure model with p = 1 8 (described in FIG3). 9 out of 10 randomly chosen direct network reconstructions fail to capture the key structure of the original image under this corruption mechanism. In TAB8, we summarize this with the SNRs of reconstructions from the erasure corruption mechanism. In that table we also report SNRs when reconstructing from measurements at different noise levels. The ensemble of direct networks performs well when the training and test data have the same measurement noise level. However, our method is more robust to changes in the test noise level. This further illustrates that direct networks are highly tuned to the training scenario and therefore not as stable as our proposed method (cf. Section 3). Original FIG4: Reconstructions of the original image from 10 individually trained direct inversion networks for 10dB noise under the p = 1 8 erasure corruptions model (described in FIG3). 9 out of the 10 reconstructions fail to capture the key structure of the original image. Single

represent the tldr for finding the corresponding scientific article.

We solve ill-posed inverse problems with scarce ground truth examples by estimating an ensemble of random projections of the model instead of the model itself.

represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.