Question: Represent the scientific article to find a short TLDR

Question: Graph neural networks have shown promising on representing and analyzing diverse graph-structured data such as social, citation, and protein interaction networks. Existing approaches commonly suffer from the oversmoothing issue, regardless of whether policies are edge-based or node-based for neighborhood aggregation. Most methods also focus on transductive scenarios for fixed graphs, leading to poor generalization performance for unseen graphs. To address these issues, we propose a new graph neural network model that considers both edge-based neighborhood relationships and node-based entity features, i.e. Graph Entities with Step Mixture via random walk (GESM). GESM employs a mixture of various steps through random walk to alleviate the oversmoothing problem and attention to use node information explicitly. These two mechanisms allow for a weighted neighborhood aggregation which considers the properties of entities and relations. With intensive experiments, we show that the proposed GESM achieves state-of-the-art or comparable performances on four benchmark graph datasets comprising transductive and inductive learning tasks. Furthermore, we empirically demonstrate the significance of considering global information. The source code will be publicly available in the near future. Graphs are universal data representations that exist in a wide variety of real-world problems, such as analyzing social networks , forecasting traffic flow , and recommending products based on personal preferences . Owing to breakthroughs in deep learning, recent graph neural networks (GNNs) have achieved considerable success on diverse graph problems by collectively aggregating information from graph structures; ). As a , much research in recent years has focused on how to aggregate the feature representations of neighbor nodes so that the dependence of graphs is effectively utilized. The majority of studies have predominantly depended on edges to aggregate the neighboring nodes' features. These edge-based methods are premised on the concept of relational inductive bias within graphs , which implies that two connected nodes have similar properties and are more likely to share the same label . While this approach leverages graphs' unique property of capturing relations, it appears less capable of generalizing to new or unseen graphs (b). To improve the neighborhood aggregation scheme, some studies have incorporated node information; They fully utilize node information and reduce the effects of relational (edge) information. A recent approach, graph attention networks (GAT), employs the attention mechanism so that weights used for neighborhood aggregation differ according to the feature of nodes (Veličković et al., 2018). This approach has yielded impressive performance and has shown promise in improving generalization for unseen graphs. Regardless of neighborhood aggregation schemes, most methods, however, suffer from a common problem where neighborhood information is considered to a limited degree . For example, graph convolutional networks (GCNs) only operate on data that are closely connected due to oversmoothing, which indicates the "washing out" of remote nodes' features via averaging. Consequently, information becomes localized and access to global information is restricted , leading to poor performance on datasets in which only a small portion is labeled. In order to address the aforementioned issues, we propose a novel method, Graph Entities with Step Mixture via random walk (GESM), which considers information from all nodes in the graph and can be generalized to new graphs by incorporating random walk and attention. Random walk enables our model to be applicable to previously unseen graph structures, and a mixture of random walks alleviates the oversmoothing problem, allowing global information to be included during training. Hence, our method can be effective, particularly for nodes in the periphery or a sparsely labeled dataset. The attention mechanism also advances our model by considering node information for aggregation. This enhances the generalizability of models to diverse graph structures. To validate our approach, we conducted experiments on four standard benchmark datasets: Cora, Citeseer, and Pubmed, which are citation networks for transductive learning, and protein-protein interaction (PPI) for inductive learning, in which test graphs remain unseen during training. In addition to these experiments, we verified whether our model uses information of remote nodes by reducing the percentage of labeled data. The experimental demonstrate the superior performance of GESM on inductive learning as well as transductive learning for datasets. Moreover, our model achieved enhanced accuracy for datasets with reduced label rates, indicating the contribution of global information. The key contributions of our approach are as follows: • We present graphs with step mixture via random walk, which can adaptively consider local and global information, and demonstrate its effectiveness through experiments on public benchmark datasets with few labels. • We propose Graph Entities with Step Mixture via random walk (GESM), an advanced model which incorporates attention, and experimentally show that it is applicable to both transductive and inductive learning tasks, for both nodes and edges are utilized for the neighborhood aggregation scheme. • We empirically demonstrate the importance of propagation steps by analyzing its effect on performance in terms of inference time and accuracy. Step-0 Step-1 Step- Figure 1: Random walk propagation procedure. From left to right are step-0, step-1, step-2, and stepinfinite. The values in each node indicate the distribution of a random walk. In the leftmost picture, only the starting node has a value of 100, and all other nodes are initialized to zero. As the number of steps increases, values spread throughout the graph and converge to some extent. Random walk, which is a widely used method in graph theory, mathematically models how node information propagates throughout the graph. As shown in Figure 1, random walk refers to randomly moving to neighbor nodes from the starting node in a graph. For a given graph, the transition matrix P, which describes the probabilities of transition, can be formulated as follows: where A denotes the adjacency matrix of the graph, and D the diagonal matrix with a degree of nodes. The probability of moving from one node to any of its neighbors is equal, and the sum of the probabilities of moving to a neighboring node adds up to one. Let u t be the distribution of the random walk at step t (u 0 represents the starting distribution). The t step random walk distribution is equal to multiplying P, the transition matrix, t times. In other words, The entries of the transition matrix are all positive numbers, and each column sums up to one, indicating that P is a matrix form of the Markov chain with steady-state. One of the eigenvalues is equal to 1, and its eigenvector is a steady-state . Therefore, even if the transition matrix is infinitely multiplied, convergence is guaranteed. The attention mechanism was introduced in sequence-to-sequence modeling to solve long-term dependency problems that occur in machine translation . The key idea of attention is allowing the model to learn and focus on what is important by examining features of the hidden layer. In the case of GNNs , GATs (Veličković et al., 2018) achieved stateof-the-art performance by using the attention mechanism. Because the attention mechanism considers the importance of each neighboring node, node features are given more emphasis than structural information (edges) during the propagation process. Consequently, using attention is advantageous for training and testing graphs with different node features but the same structures (edges). Given the many benefits of attention, we incorporate the attention mechanism to our model to fully utilize node information. The attention mechanism enables different importance values to be assigned to nodes of the same neighborhood, so combining attention with mixture-step random walk allows our model to adaptively highlight features with salient information in a global scope. Let G = (V, E) be a graph, where V and E denote the sets of nodes and edges, respectively. Nodes are represented as a feature matrix X ∈ R n×f, where n and f respectively denote the number of nodes and the input dimension per node. A label matrix is Y ∈ R n×c with the number of classes c, and a learnable weight matrix is denoted by W. The adjacency matrix of graph G is represented as A ∈ R n×n. The addition of self-loops to the adjacency matrix is A = A + I n, and the column normalized matrix of A isÂ = AD −1 withÂ 0 = I n. Most graph neural networks suffer from the oversmoothing issue along with localized aggregation. Although JK-Net tried to handle oversmoothing by utilizing GCN blocks with mulitple propagation, it could not completely resolve the issue as shown in Figure 4b. We therefore propose Graph Step Mixture (GSM), which not only separates the node embedding and propagation process but also tackles oversmoothing and localized aggregation issues through a mixture of random walk steps. GSM has a simple structure that is composed of three stages, as shown in Figure 2. Input X passes through a fully connected layer with a nonlinear activation. The output is then multiplied by a normalized adjacency matrixÂ for each random walk step that is to be considered. The for each step are concatenated and fed into another fully connected layer, giving the final output. The entire propagation process of GSM can be formulated as: where is the concatenation operation, s is the maximum number of steps considered for aggregation, andÂ k is the normalized adjacency matrixÂ multiplied k times. As can be seen from Equation 3, weights are shared across nodes. In our method, the adjacency matrixÂ is an asymmetric matrix, which is generated by random walks and flexible to arbitrary graphs. On the other hand, prior methods such as JK-Net and MixHop , use a symmetric Laplacian adjacency matrix, which limits graph structures to given fixed graphs. (a) Traditional global aggregation scheme Step-1 Step-2 Step-3 (b) Our step-mixture scheme For the concatenation operation, localized sub-graphs are concatenated with global graphs, which allows the neural network to adaptively select global and local information through learning (see Figure 3). While traditional graph convolution methods consider aggregated information within three steps by A(A(AXW )W )W, our method can take all previous aggregations into account To develop our base model which depends on edge information, we additionally adopt the attention mechanism so that node information is emphasized for aggregation, i.e., Graph Entity Step Mixture (GESM). We simply modify the nonlinear transformation of the first fully connected layer in GSM by replacing it with the attention mechanism denoted by H multi (see Equations 3 and 4). As described in Equation 4, we employ multi-head attention, where H multi is the concatenation of m attention layers and α is the coefficient of attention computed using concatenated features of nodes and its neighboring nodes. By incorporating attention to our base model, we can avoid or ignore noisy parts of the graph, providing a guide for random walk . Utilizing attention can also improve combinatorial generalization for inductive learning, where training and testing graphs are completely different. In particular, datasets with the same structure but different node information can benefit from our method because these datasets can only be distinguished by node information. Focusing on node features for aggregation can thus provide more reliable in inductive learning. The time complexity of our base model is O(s × l × h), where s is the maximum number of steps considered for aggregation, l is the number of non-zero entries in the adjacency matrix, and h is the hidden feature dimension. As suggested by , we can assume h << l under realistic assumptions. Our model complexity is, therefore, highly efficient with time complexity O(s × l), which is on par with vanilla GCN . Transductive learning. We utilize three benchmark datasets for node classification: Cora, Citeseer, and Pubmed . These three datasets are citation networks, in which the nodes represent documents and the edges correspond to citation links. The edge configuration is undirected, and the feature of each node consists of word representations of a document. Detailed statistics of the datasets are described in Table 1. For experiments on datasets with the public label rate, we follow the transductive experimental setup of. Although all of the nodes' feature vectors are accessible, only 20 node labels per class are used for training. Accordingly, 5.1% for Cora, 3.6% for Citeseer, and 0.3% for Pubmed can be learned. In addition to experiments with public label rate settings, we conducted experiments using datasets where labels were randomly split into a smaller set for training. To check whether our model can propagate node information to the entire graph, we reduced the label rate of Cora to 3% and 1%, Citeseer to 1% and 0.5%, Pubmed to 0.1%, and followed the experimental settings of for these datasets with low label rates. For all experiments, we report the using 1,000 test nodes and use 500 validation nodes. Inductive learning. We use the protein-protein interaction PPI dataset ,which is preprocessed by Veličković et al.. As detailed in Table 1, the PPI dataset consists of 24 different graphs, where 20 graphs are used for training, 2 for validation, and 2 for testing. The test set remains completely unobserved during training. Each node is multi-labeled with 121 labels and 50 features regarding gene sets and immunological signatures. For transductive learning, we compare our model with numbers of state-of-the-art models according to the reported in the corresponding papers. Our model is compared with baseline models specified in (Veličković et al., 2018) such as label propagation (LP) , graph embeddings via random walk (DeepWalk) , and Planetoid . We also compare our model with models that use self-supervised learning (Union), learnable graph convolution (LGCN) , GCN based multi-hop neighborhood mixing (JK-GCN and MixHop) , multi-scale graph convolutional networks (AdaLNet) and maximal entropy transition (PAN) . We further include models that utilize teleport term during propagation APPNP , conduct convolution via spectral filters such as ChebyNet , GCN , SGC (a), and GWNN and models that adopt attention between nodes, such as GAT (Veličković et al., 2018) and AGNN . For inductive learning tasks, we compare our model against four baseline models. This includes graphs that use sampling and aggregation (GraphSAGE-LSTM) , and jumping-knowledge (JK-LSTM) , along with GAT and LGCN which are used in the transductive setting. Regarding the hyperparameters of our transductive learning models, we used different settings for datasets with the public split and random split. We set the dropout probability such that 0.3 of the data were kept for the public split and 0.6 were kept for the random split. We set the number of multi-head m = 8 for GESM. The size of the hidden layer h ∈ {64, 512} and the maximum number of steps used for aggregation s ∈ {10, 30} were adjusted for each dataset. We trained for a maximum of 300 epochs with L2 regularization λ = 0.003 and learning rate lr = 0.001. We report the average classification accuracy of 20 runs. For inductive learning, the size of all hidden layers was the same with h = 256 for both GSM, which consisted of two fully connected layers at the beginning and GESM. We set the number of steps s = 10 for GSM, and s = 5, m = 15 for GESM. L2 regularization and dropout were not used for inductive learning (Veličković et al., 2018). We trained our models for a maximum of 2,000 epochs with learning rate lr = 0.008. The evaluation metric was the micro-F1 score, and we report the averaged of 10 runs. For all the models, the nonlinearity function of the first fully connected layer was an exponential linear unit (ELU) . Our models were initialized using Glorot initialization and were trained to minimize the cross-entropy loss using the Adam . We employed an early stopping strategy based on the loss and accuracy of the validation sets, with a patience of 100 epochs. Results on benchmark datasets. Table 2 summarizes the comparative evaluation experiments for transductive and inductive learning tasks. In general, not only are there a small number of methods that can perform on both transductive and inductive learning tasks, but the performance of such methods is not consistently high. Our methods, however, are ranked in the top-3 for every task, indicating that our method can be applied to any task with large predictive power. For transductive learning tasks, the experimental of our methods are higher than or equivalent to those of other methods. As can be identified from the table, our base model GSM, which is computationally efficient and simple, outperforms many existing baseline models. These indicate the significance of considering both global and local information and using random walks. It can also be observed that GESM yielded more impressive than GSM, suggesting the importance of considering node information in the aggregation process. For the inductive learning task, our base model GSM, which employs an edge-based aggregation method, does not invariably obtain the highest accuracy. However, our model with attention, GESM, significantly improves performance of GSM by learning the importance of neighborhood nodes, and surpasses the of GAT, despite the fact that GAT consists of more attention layers. These for unseen graphs are in good agreement with shown by Veličković et al., in which reducing the influence of structural information improved generalization. Results on datasets with low label rates. To demonstrate that our methods can consider global information, we experimented on sparse datasets with low label rates of transductive learning datasets. As indicated in Table 3, our models show remarkable performance even on the dataset with low label rates. In particular, we can further observe the superiority of our methods by inspecting Table 2 and 3, in which our methods trained on only 3% of the Cora dataset outperformed some other methods trained on 5.1% of the data. Because both GSM and GESM showed enhanced accuracy, it could be speculated that using a mixture of random walks played a key role in the experiments; the improved can be explained by our methods adaptively selecting node information from local and global neighborhoods, and allowing peripheral nodes to receive information. Oversmoothing and Accuracy. As shown in Figure 4a, GCN , SGC (a), and GAT (Veličković et al., 2018) suffer from oversmoothing. GCN and GAT show severe degradation in accuracy after the 8th step; The accuracy of SGC does not drop as much as GCN and GAT but nevertheless gradually decreases as the step size increases. The proposed GSM, unlike the others, maintains its performance without any degradation, because no rank loss occurs and oversmoothing is overcome by step mixture. Interestingly, JK-Net also keeps the training accuracy regardless of the step size by using GGN blocks with multiple steps according to Figure 4a. We further compared the test accuracy of GSM with JK-Net, a similar approach to our model, in regards to the step size. To investigate the adaptability to larger steps of GSM and JK-Net, we concatenated features after the 10th step. As shown in Figure 4b, GSM outperforms JK-Net, even though both methods use concatenation to alleviate the oversmoothing issue. These are in line with the fact that JK-Net obtains global information similar to GCN or GAT. Consequently, the larger the step, the more difficult it is for JK-Net to maintain performance. GSM, on the other hand, maintains a steady performance, which confirms that GSM does not collapse even for large step sizes. Test Accuracy GSM public (5.1%) 3% 1% GESM public (5.1%) 3% 1% We also observe the effect on accuracy as the number of steps increases under three labeling conditions for GSM and GESM. As represented in Figure 5, it is evident that considering remote nodes can contribute to the increase in accuracy. By taking into account more data within a larger neighborhood, our model can make reliable decisions, ing in improved performance. Inspection of the figure also indicates that the accuracy converges faster for datasets with higher label rates, presumably because a small number of walk steps can be used to explore the entire graph. Moreover, the addition of attention benefits performance in terms of higher accuracy and faster convergence. Inference time. As shown in Figure 6, the computational complexity of all models increases linearly as the step size increases. We can observe that the inference time of GSM is faster than GCN especially when the number of steps is large. The inference time of GESM is much faster than GAT (Veličković et al., 2018) while providing higher accuracies and stable (see Appendix A). Our methods are both fast and accurate due to the sophisticated design with a mixture of random walk steps. Embedding Visualization. Figure 7 visualizes the hidden features of Cora from our models by using Figure 7: t-SNE plot of the last hidden layer trained on the Cora dataset. the t-SNE algorithm . The figure illustrates the difference between edgebased and node-based aggregation. While the nodes are closely clustered in the from GSM, they are scattered in that of GESM. According to the in Table 2, more closely clustered GSM does not generally produce better than loosely clustered GESM, which supports findings that the attention mechanism aids models to ignore or avoid noisy information in graphs . Table 5: Average test set accuracy and standard deviation over 100 random train/validation/test splits with 20 runs. Top-3 for each column are highlighted in bold, and top-1 values are underlined. Coauthor CS Coauthor Physics Amazon Computers Amazon Photo MLP 88.3 ± 0.7 88.9 ± 1.1 44.9 ± 5.8 69.6 ± 3.8 LogReg 86.4 ± 0.9 86.7 ± 1.5 64.1 ± 5.7 73.0 ± 6.5 LP 73.6 ± 3.9 86.6 ± 2.0 70.8 ± 8.1 72.6 ± 11.1 GCN 91.1 ± 0.5 92.8 ± 1.0 82.6 ± 2.4 91.2 ± 1.2 GraphSAGE 91.3 ± 2.8 93.0 ± 0.8 82.4 ± 1.8 91.4 ± 1.3 GAT (Veličković et al., 2018) 90.5 ± 0.6 92.5 ± 0.9 78.0 ± 19.0 85.7 ± 20.3 GSM (our base model) 91.8 ± 0.4 93.3 ± 0.6 79.2 ± 2.0 89.3 ± 1.9 GESM (GSM+attention) 92.0 ± 0.5 93.7 ± 0.6 79.3 ± 1.7 90.0 ± 2.0 For an in-depth verification of overfitting, we extended our experiments to four types of new node classification datasets. Coauthor CS and Coauthor Physics are co-authorship graphs from the KDD Cup 2016 challenge 1, in which nodes are authors, features represent the article keyword for each author's paper, and class labels indicate each author's most active research areas. Amazon Computers and Amazon Photo are co-purchase graphs of Amazon, where nodes represent the items, and edges indicate that items have been purchased together. The node features are bag-of-words of product reviews, and class labels represent product categories. Detailed statistics of the datasets are described in Table 4 and we followed the experimental setup of. We used the same values for each hyperparameter (unified size: 64, step size: 15, multi-head for GAT and GESM: 8) without tuning. The in Table 5 prove that our proposed methods do not overfit to a particular dataset. Moreover, in comparison to GAT, the performance of GESM is more accurate, and more stable. We visualized the distribution of attention vectors. Figure 8a plots the distribution of neighbors with equal importance and Figure 8b displays the distribution of attention weighted neighbors that we trained with GESM. Although both figures look similar to some degree, we can conjecture that GESM slightly adjusts the weight values, contributing to improved performance.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

Simple and effective graph neural network with mixture of random walk steps and attention

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a new sparse and structured attention mechanism, TVmax, which promotes sparsity and encourages the weight of related adjacent locations to be the same.


Question: Represent the scientific article to find a short TLDR

Question: Basis pursuit is a compressed sensing optimization in which the l1-norm is minimized subject to model error constraints. Here we use a deep neural network prior instead of l1-regularization. Using known noise statistics, we jointly learn the prior and reconstruct images without access to ground-truth data. During training, we use alternating minimization across an unrolled iterative network and jointly solve for the neural network weights and training set image reconstructions. At inference, we fix the weights and pass the measurements through the network. We compare reconstruction performance between unsupervised and supervised (i.e. with ground-truth) methods. We hypothesize this technique could be used to learn reconstruction when ground-truth data are unavailable, such as in high-resolution dynamic MRI. Deep learning in tandem with model-based iterative optimization -, i.e. model-based deep learning, has shown great promise at solving imaging-based inverse problems beyond the capabilities of compressed sensing. These networks typically require hundreds to thousands of examples for training, consisting of pairs of corrupted measurements and the desired ground-truth image. The reconstruction is then trained in an end-to-end fashion, in which data are reconstructed with the network and compared to the ground-truth . in many cases, collecting a large set of fully sampled data for training is expensive, impractical, or impossible. In this work, we present an approach to model-based deep learning without access to ground-truth data -. We take advantage of (known) noise statistics for each training example and formulate the problem as an extension of basis pursuit denoising with a deep convolutional neural network (CNN) prior in place of image sparsity. During training, we jointly solve for the CNN weights and the reconstructed training set images. At inference time, we fix the weights and pass the measured data through the network. As proof of principle, we apply the technique to undersampled, multi-channel magnetic resonance imaging (MRI). We compare our Deep Basis Pursuit (DBP) formulation with and without supervised learning, as well as to MoDL, a recently proposed unrolled modelbased network that uses ground-truth data for training. We show that in the unsupervised setting, we are able to approach the image reconstruction quality of supervised learning, thus opening the door to applications where collecting fully sampled data is not possible. We focus on the discretized linear signal model under additive white Gaussian noise: where x ∈ C N is the vectorized unknown image, A ∈ C M ×N is the discretized forward model describing the imaging system, y ∈ C M is a vector of the acquired measurements, and v ∼ N c 0, σ 2 I is a complex-valued Gaussian noise vector. We are interested in the ill-posed regime, where M < N. To make the inverse problem well-posed, x is commonly solved through a regularized least-squares: where Q(x) is a suitable regularization term, and λ > 0 is the strength of the regularization. An alternative, equivalent formulation that directly accounts for the model error due to noise is the constrained problem: where = σ √ M is the square-root of the expected noise power in the measurements. When an 1 -norm is used for regularization, this is known as basis pursuit denoising, and provides an intuitive formulation as it finds the best (sparsist) representation given a noise error constraint. CNNs have recently been used to solve imaging inverse problems, relying on the network architecture and training data to learn the inverse mapping. When a large corpus of training data is available, it is possible to learn the inverse mapping directly from under-sampled measurements, typically by first transforming the measurements to the image domain either through the adjoint operation A * y or through a conventional reconstruction. Except for the initial transformation, these models do not take advantage of knowledge of the imaging system in the network architecture. Thus, they require substantial training data and are prone to overfitting and CNN artifacts. More recently, network architectures that combine both CNN blocks and data consistency blocks incorporating knowledge of the forward model have grown in popularity, as they allow for robustness against CNN artifacts and training with limited data,. These architectures are inspired by conventional first-order iterative algorithms intended to solve the unconstrained problem, and typically alternate between data consistency and manifold projection. To facilitate training with backpropagation, the iterative algorithms are unrolled for a finite number of steps and optimized in an end-to-end manner. As the network is differentiable, gradient updates can be computed through the application of the forward operator with auto-differentiation. For a particular network architecture, we can view the image reconstruction as a feed-forward network where F w is a deep network parameterized by weights w that operates on the measurements and optionally incorporates knowledge of the forward model. Given a training set of inputs {y and corresponding ground-truth images {x, the network weights can be trained in a traditional end-to-end fashion by minimizing the average training loss as measured by the loss function L: For inference, the weights are fixed and new measurements are reconstructed through a forward pass of. Inspired by other model-based deep learning architectures -, we propose a new unrolled network based on basis pursuit denoising, which we call Deep Basis Pursuit (DBP). We assume the noise statistics of the measurements are known and we use them to selfregularize the solution. In turn, we propose to train in an unsupervised fashion in the measurement domain, taking advantage of explicit control of the error between the measurements and the output of the network. We first describe the DBP model, and then discuss training the in an unsupervised fashion without ground-truth data. We combine the data consistency constraint of basis pursuit denoising with the 2 -norm incorporating a CNN auto-encoder. The DBP optimization is given by arg min where N w (x) ≡ x − R w (x) is a CNN parameterized by weights w that aims to estimate noise and aliasing,. In other words, R w (x) represents a denoised version of x. In this way, we seek to find the "cleanest" representation of x while allowing for the expected data inconsistency due to noise. To approximately solve, we consider an alternating minimization, repeated N 1 times: x k = arg min Subproblem is a forward pass through the CNN. Subproblem is convex and can solved with ADMM. We introduce the slack variable z = Ax and the dual variable u, and apply the following update steps, repeated N 2 times: where ρ > 0 is the ADMM penalty parameter and L2Proj(z,) is the projection of z onto the 2 -ball of radius. The update steps are amenable to matrix-free optimization, as the forward and adjoint calculations can be represented as computationally efficient operators. In particular, subproblem can be approximately solved with N 3 iterations of the Conjugate Gradient Method. Altogether, we can view DBP as an unrolled optimization alternating between CNN layers and data consistency layers, as shown in Fig. 1. At each layer, the same CNN is used, though it is possible in general to relax this requirement. For a fixed CNN R w, the DBP model is a special case of:xw ≡ Fw(y; A,), wherẽ w = (w, ρ) are the network parameters, and the network uses measurements together with knowledge of the system and noise power to return an estimate of the image. and ground-truth training data are available, the network weights can be trained in a traditional end-to-end fashion according to. When ground-truth data are not available, we consider a loss functionL imposed in the measurement domain: The measurement loss can be a useful surrogate for the true loss, as the measurements contain (noisy) information about the ground-truth image,. Thus, we may hope to learn about the image statistics given a large-enough training set that includes a diversity of measurements. We consider the application to under-sampled, multichannel MRI. The MRI reconstruction task is well-suited to DBP, as the noise statistics are Gaussian and can be measured during a short pre-scan. We first describe the multi-channel MRI forward operator and general sampling strategy. Then we discuss the experimental setup, including the dataset and implementation details. In multi-channel MRI, the signal is measured by an array of receive coils distributed around an object, each with a spatially-varying sensitivity profile. In the measurement model, the image is linearly mixed with each coil sensitivity profile, Fourier transformed, and sampled. We can describe the measurement model as A = (P F S 1) · · · (P F S C) ∈ C M ×N, where C is the number of receive coils, S c ∈ C N ×N is a diagonal operator containing the spatial sensitivity profile of the c th coil along the diagonal, F is the Fourier transform operator, and P ∈ {0, 1} M C ×N is a diagonal operator that selects the sampled frequencies. Data: We used the "Stanford Fully Sampled 3D FSE Knees" dataset from mridata.org, containing 3D Cartesian proton-density knee scans of 20 healthy volunteers. Each 3D volume consisted of 320 slices with matrix size 320×256 and was scanned with an 8-channel receive coil array. Although each slice is fully sampled, in practice the "ground-truth" data itself has noise. To aid in experimental comparison, "noise-free" ground-truth data were created by averaging the data from seven adjacent slices. For each slice, the spatial sensitivity profiles of each coil were estimated using ESPIRiT, a self-calibrated parallel imaging method. Ground-truth images were reconstructed by solving using the fully sampled data without regularization. Each slice was then passed through the forward model and retrospectively under-sampled using a different variable-density Poisson-disc sampling pattern, with a 16×16 calibration region and acceleration factor R ≈ 12. Slices from the first 16 volunteers were used for training, discarding the first and last 20 edge slices of each volume (4,384 slices). Similarly, slices from the next two volunteers were used for validation (548 slices), and slices from the last two volunteers were used for testing (548 slices). We added complex-valued Gaussian noise with standard deviation σ = 0.01 to the noise-free, averaged data. Implementation: For all experiments we used a Euclidean norm loss function for training. When training with ground-truth (supervised), the loss was applied in the image domain. For unsupervised training, the loss was applied in the measurement (Fourier) domain. We used a U-Net architecture for the CNN autoencoder, with separate input channels for the real and imaginary components. The U-Net consisted of three encoding layers with ReLU activation functions and 64, 128, and 256 channels, respectively, followed by three similar decoding layers. A final convolutional layer with no activation function mapped the decoder back to two channels. All convolutions used a 3 × 3 kernel size. For comparison, MoDL was also implemented using the same unrolled parameters and CNN architecture. All networks were implemented in PyTorch. Evaluation: DBP was separately trained with and without ground-truth data. We also trained MoDL with ground-truth data. In addition, we also evaluated parallel imaging and compressed sensing (PICS) using BART, with 1 -Wavelet regularization parameter optimized over the validation set. Normalized root mean-squared error (NRMSE) was used to compare reconstructions. V. Fig. 2 shows the mean NRMSE on the training set for each epoch. In addition to a performance gap between supervised and unsupervised learning, unsupervised DBP has noisier updates, likely because the loss function in the measurement domain is a noisy surrogate to the NRMSE. Fig. 3 shows the NRMSE across the validation set for different numbers of unrolls during inference. Even though the networks were trained with 5 unrolls, best performance is seen for different number of unrolls (6, 10 and 12 for MoDL, unsupervised DBP, and supervised DBP, respectively). Compared to MoDL, the DBP formulation behaves more stably as the number of unrolls increases, which may be due to the hard data consistency constraint. At the optimal number of unrolls, unsupervised DBP outperforms PICS. Fig. 4. Box plot of test set NRMSE for supervised and unsupervised DBP at two different unrolls -the first matching unrolls at training, and the second chosen to minimize validation set mean NRMSE. Also shown is PICS NRMSE for optimized regularization on validation set. Fig. 5 shows some of the intermediate output stages for the supervised and unsupervised DBP networks, indicating that similar structure is learned in both CNNS; however, the supervised DBP appears to better amplify and denoise features in the image. The magnitude reconstructions and error maps of a representative slice from the test set are shown in Fig. 6. Supervised DBP achieves the lowest error, followed by unsupervised DBP, PICS, and MoDL. Small details in edge sharpness are retained with DBP, but reduced with MoDL and PICS. There are strong connections to iterative optimization and unrolled deep networks,,. Jointly optimizing over the images and weights could be viewed a non-linear extension to dictionary learning. Nonetheless, there is a cost in reconstruction error when moving to unsupervised learning, highlighting the importance of a large training data set to offset the missing ground-truth information. The choice of measurement loss function and data SNR may also greatly impact the quality. Fortunately, in many practical settings there is an abundance of undersampled or corrupted measurement data available for training. In , the combination of basis pursuit denoising and deep learning can take advantage of undersampled data and provide a means to learn model-based deep learning reconstructions without access to groundtruth images.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We present an unsupervised deep learning reconstruction for imaging inverse problems that combines neural networks with model-based constraints.

Represent the short summary for finding the corresponding scientific article (long):

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent text

Question: Deep learning approaches usually require a large amount of labeled data to generalize. However, humans can learn a new concept only by a few samples. One of the high cogntition human capablities is to learn several concepts at the same time. In this paper, we address the task of classifying multiple objects by seeing only a few samples from each category. To the best of authors' knowledge, there is no dataset specially designed for few-shot multiclass classification. We design a task of mutli-object few class classification and an environment for easy creating controllable datasets for this task. We demonstrate that the proposed dataset is sound using a method which is an extension of prototypical networks. Deep learning approaches are usually capable of solving a classification problem when a large labeled dataset is available during the training BID12; BID4. However, when a very few samples of a new category is shown to a trained classifier, it either fails to generalize or overfit on the new samples. Humans, however, can easily generalize their prior knowledge to learn a new concept even with one sample. Few-shot learning approaches are proposed to address this gap between human capablity of learning a new concept with only a very few labled samples and machine capablity in generalizing to a new concept. mini-ImageNet and tiered-Imagenet BID19 are two main datasets that are developed to help the research community addressing the problem of few-shot classification. Although that human are capable of learning a new concept with only a very few samples. Learning a few new concepts at the same time, and with only a very few samples of each is considered as a high cognition task BID1 and very challenging even for humans. It is yet an active area of study to know how human are capable of doing this. There could be many factors involved in this high cognition process, and there are many hypothesis around this. One popular hypothesis is that the brain is able to learn a good representation that has high capacity and can generalize well BID7. Studying the reasons behind human high cognitive capablity of learning a few new concepts in paralell and with only a very few samples, is out of the scope of this paper. However, in this paper, we propose to extend the few shot learning problem to multi-class few shot classification problem and moving a step towards filling the gap between human cognitive capablity of learning multiple new concepts in paralel and with only a few samples, and machine learning approaches. To do so, our first step is to define a dataset and a setup to address this problem, and an evaluation metric to measure our progression towards solving this problem. We argue that the existing datasets are not desirable for this task. Omniglot BID13, mini-ImageNet, tiered-ImagaNet, are designed for single object classification. Such datasets as, MS COCO BID15 and Pascal VOC BID5 have multiple object classes but they are not well suited for few-shot learning. The issue is the high imbalance of class cooccurrence (for example, 'human' label occures with all other classes). Therefore it is hard to prevent the learner from "sneak peeking" new concepts. To sum it up, this work's contribution is two-fold: 1. We propose the new task of mutli-object few-shot classification to test model's ability to disentagle and represent several object on an image (see Section 3) and propose an extension to prototypical-style models to efficiently solve the task (Section 3.1);2. We construct a new dataset which provides clean and controlled environment of multiobject images (see Section 4) and provide the framework, benchmarks and the code for the community to explore controlled scenarios and alternative few-shot classification tasks. The problem of learning new concepts with small number of labeled data is usually referred to as few-shot learning BID14. Two of the most famous datasets to address the problem of few shot classification are mini-Imagenet and tiered-Imagenet BID19. Both of these datasets address the problem of few shot classification of single objects. BID10 addresses the problem of few shot object detection in natural images. There are two main groups of approaches addressing this problem (i) optimization-based frameworks and (ii) metric-based frameworks. Optimization-based framework BID0 BID6 BID20, is a class of algorithms that learn how to quickly learn new concepts. Other notable performant approaches that do not fall into these two categories is SNAIL BID16.Meta-learning approaches work by learning a parameterized function that maps labeled training sets into classifiers. Metric-based framework learn a representation that minimize intra-class distances while maximize the distance between variant classes. These approaches usually rely on an episodic training framework: the model is trained with sub-tasks (episodes) in which there are only a few training samples for each category. Matching networks trains a similarity function between images. In each episode, it uses an attention mechanism (over the encoded support) as a similarity measure for one-shot classification. In prototypical networks , a metric space is learned where embeddings of queries of one category are close to the centroid (or prototype) of support of the same category, and far away from centroids of other classes in the episode. Due to simplicity and performance of this approach, many methods extended this work. For instance, BID19 propose a semi-supervised fewshot learning approach and show that leveraging unlabeled samples outperform purely supervised prototypical networks. BID10 propose to augment the support set by generating hallucinated examples. Task-dependent adaptive metric (TADAM) BID17 ) relies on conditional batch normalization BID18 to provide task adaptation (based on task representations encoded by visual features) to learn a task-dependent metric space. In order to test the ability to disentangle unseen objects on a given image, we propose a task of multi-object few-shot classification. Few-shot classification First, we briefly summarize the single object few-shot classification BID14. While a dataset for supervised learning is comprised of a number of input-label pairs, a dataset D for meta-learning is comprised of a number of tasks. In a common K-shot N -way classification every task has a support set S = {(x i, y i)} i of the size KN and a query set DISPLAYFORM0 The learner is given a number of support-query pairs for training. For testing, the learner is given am unseen support set S and asked to predict labels for a certain query Q = x, which label is one of the labels in the support. Usually, prototypes are computed in the following way. First, the images from the support set e i = CNN(x i) are embedded, the prototype for each class is computed as an average embeding p n = mean i:yi=n e i for all n ∈ 1... N. In this work, we propose to extend this task for a multi category case. We define a K-shot N -way D-dense task to have the support set S = {(x i, y) is a tuple of D labels corresponding the objects. This way, the learner is exposed to every object class exactly K times, therefore our methods can be easier compared among varying D. Similarly, the query Q = {(x i, Y i)} i is a set of images each containing D objects with ground truth labels y i. A naïve aproach requires exponential in D pseudo-label to represent all possible combinations. The learner can only be exposed to a limited number of possible combinations of objects. The exponential size of the label quickly overpasses few shots (commonly from 5 to 20) available for training. We propose multi-prototype networks to tackle aforementioned exponential explosion. For simplicity, in this work we assume that the labels y i are given in the order that objects appear on the image x i from left to right. This is a major simplification and we will lift it in the near future work. To extend the proto-net setup, we train a model to produce D embeddings for every image e. The rest of the procedure is identical to the proto-net -every query is compared to the prototypes and the distance to the correct prototype is pushed down, all the rest are pushed up. We aim to have a controlled environment for reliable experiments. To achieve this, we develop a dataset based on Shapenet 3D models renderred with Blender in the setup similar to CLEVR. This provides us flexibility to construct single or multiple object tasks and change the task parameters -number of shots and ways. The dataset along with the generation tools will be made publically available. In the next sections we describe in the detail the dataset generation procedure. Then, to compare the complexity of the dataset to existing ones, we run a popular model TADAM BID17 for a traditional single-object-per-image setup. Then, we increase the number of objects per image and report with our proposed model. We generated a dataset using methods similarly to visual reasoning dataset CLEVR BID9. To increase the variability of the object classes, we used 3D meshes from Shapenet BID3 dataset. Beforehand, we splitted 55 Shapenet classes into three subsets randomly for training, validation and testing (see Appendix A). The sizes of the subsets are 35, 10, 10, respectively. We render images task-by-task. We uniformly sample classes used in the task. Then, we sample a mesh of each class K times. We distribute object over canvases. Finally, we render images by randomly placing meshes on each canvas. In order to produce the representation of an object independent of color, we strip the texture and apply random color random texture of either rubber or metal. FIG3 demonstrates some samples from the test split. The model should be able to tackle We train a prototypical network and TADAM with 12-block residual feature extractor BID8. We summarize our in the TAB0 and compare to performance on mini-ImageNet. As it can be seen, our dataset is simpler than mini-ImageNet, but it cannot be trivially solved by close to state-of-the-art methods even in the simpest single-object case. Therefore, we conclude that the dataset is not trivial nor too hard. Having shown that the proposed dataset is sound for single-object task, we increase the number of objects per image and apply the proposed MultiProtoNet (see Section 3.1). In all experiments below, we used 12-block residual network that produces a corresponding number of embeddings per image. All networks are optimized with Adam with the Euclidian distance metric. The experiments are summarized in the TAB1. We notice that while the accuracy accuracy drops significantly when transitioning from single to multiple objects, it drop as much from 2 to 3 objects. In this work we introduced a task of few-shot multi-object classification and an environment for generating datasets for this task. We compared the proposed dataset to existing ones in singleobject case. Then, we used a simple extension of prototypical networks to conduct experiments multi-object case. We believe that this task will help diagnosing metric-learning models that need to disentangle several objects on an image. One of the future directions we are taking is to lift the limitation of known object order (Section 3.1). Then we plan to use stronger feature extractors BID17 and extend the work to more natural data.

Represent the short summary for finding the corresponding scientific article (long)

We introduce a diagnostic task which is a variation of few-shot learning and introduce a dataset for it.

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We present a new approach to defining a sequence loss function to train a summarizer by using a secondary encoder-decoder as a loss function, alleviating a shortcoming of word level training for sequence outputs. The technique is based on the intuition that if a summary is a good one, it should contain the most essential information from the original article, and therefore should itself be a good input sequence, in lieu of the original, from which a summary can be generated. We present experimental where we apply this additional loss function to a general abstractive summarizer on a news summarization dataset. The is an improvement in the ROUGE metric and an especially large improvement in human evaluations, suggesting enhanced performance that is competitive with specialized state-of-the-art models. Neural networks are a popular solution to the problem of text summarization, the task of taking as input a piece of natural language text, such as a paragraph or a news article, and generating a more succinct text that captures the most essential information from the original. One popular type of neural network that has achieved state of the art is an attentional encoderdecoder neural network;;. In an encoder-decoder network, the encoder scans over the input sequence by ingesting one word token at a time to create an internal representation. The decoder is trained to compute a probability distribution over next words conditioned on a sequence prefix. A beam search decoder is typically used to find a high likelihood output sequence based on these conditional word probability distributions. Since the next word depends heavily on previous words, the decoder has little hope of a correct distribution over next words unless it has the correct previous words. Thus the decoder is typically trained using teacher forcing , where the reference sequence prefix is always given to the decoder at each decoding step. In other words, regardless of what distributions are output by the decoder in training for timesteps (1, ..., t−1), at timestep t, it is given the reference sequence prefix (y Training at the sequence level can alleviate this discrepancy, but requires a differentiable loss function. In the Related Work section we review previous efforts. We present a novel approach to address the problem by defining a loss function at the sequence level using an encoder-decoder network as a loss function. In training, the summarizer's beam search decoded output sequence is fed as input into another network called the recoder. The recoder is an independent attentional encoder-decoder trained to produce the reference summary. Our experiments show that adding the recoder as a loss function improves a general abstractive summarizer on the popular CNN/DailyMail dataset ; , achieving significant improvements in the ROUGE metric and an especially large improvement in human evaluations. We first give a high level overview of an attentional encoder-decoder model for sequence to sequence learning. We present the specific abstractive model of , which serves as the baseline model in our experiments for comparison. We chose this as the baseline model because it is a general and popular model whose concepts have often appeared in other abstractive summarization works, and its have often been used as a comparison baseline ; ; . The ideas presented in this paper are largely independent of the specific summarizer and may be applicable to the training of other sequence to sequence models, many of which are also trained at the word loss level using teacher forcing. Since our focus is to demonstrate the effectiveness of our new loss function, the generality of the model, which does not account for specifics of the summarization problem such as article or sentence structure , suits our purpose well. The source article and target summary are each treated as a sequence of tokens. A token represents either a word or a punctuation mark. Let (x i) be the sequence of tokens in an article or input sequence, where i ranges from 1 to the length of the input sequence. Let (y t) be the sequence of word tokens in the summary or output sequence, where 1 ≤ t. The goal is to train a neural network to compute, at each timestep t, the probability distribution P (y t |y 1, ..., y t−1) over next tokens y t given previous tokens (y 1, ..., y t−1). In the teacher forcing training method, the network is trained to minimize the loss J ml, also called the maximum likelihood loss, defined as the average over timesteps t of (J ml) t = − log P (y * t |y * 1, ..., y * t−1) where the sequence prefix y * 1,..., y * t−1 is from the reference summary. At test time, the network computes P (y t |y 1, ..., y t−1) where y 1,..., y t−1 are chosen by a beam search decoder based on previously output distributions. The sequence ends when the special token STOP, appended to the end of all reference summaries in training, is output. Each x i is embedded as a dense vector w i using a v × d weight matrix W emb, where v is the size of the vocabulary and d is the number of hidden dimensions, a hyperparameter. The weights W emb are learned as part of training. Each w i is given in order as input into an LSTM to get hidden states (h f i), and also in reverse order into another LSTM to get backward states (h b i). The two state sequences are then stacked to form hidden states (h i) for reference by the decoding phase's attention mechanism. The decoding phase also uses an LSTM. At each output timestep t, a token y is embedded as a dense vector using the same W emb and fed into the LSTM, and the LSTM outputs a probability distribution over next tokens. For t = 0, y is the special placeholder symbol START. For t > 0, when training using teacher forcing, y = y * t−1 is the corresponding token from the reference sequence. When beam search decoding at test time, y = y t−1 is chosen based on the decoder's output probabilities at step t − 1. This difference in information about previous output tokens causes discrepancy between training and test conditions. The decoder makes use of an additional piece of information called attention , which allows it to "read" the encoder's output states. At each timestep with decoder LSTM state s t, the attention vector is computed as Finally, another linear transformation using a weight matrix V and weight vector b is used to map these outputs in the hidden space, through a softmax layer, into output probabilities over the vocabulary. That is, where V, V, b, b are trainable parameters. The pointer generator mechanism allows the decoder to copy a token directly from the input. It is particularly helpful for out-of-vocabulary tokens. The decoder "copies" source tokens with a distribution based on the attention vector a t and mixes it in with weight 1 − p gen, where p gen is a trainable function of the context vectors, states, and next word input. The final distribution P (y t |y 1, ..., y t−1) is the combined The summarizer loss function J S = J ml is computed based on this distribution. In one model variant, a coverage loss J cov that encourages attention vectors a t to look at different encoder states i is also added to the final loss function. That is, Full details on the pointer-generator and coverage mechanisms are found in. We have omitted them here for brevity. The recoder does not depend on these particulars of the summarization model. We focus on the quantities P vocab, P, and the total summarizer loss J S. Specific details of how these are computed are not needed in the following sections. We are now ready to present our main contribution. The recoder is a neural network model that takes the decoded sequence from the summarizer as input and is trained to output the reference summary. The purpose of the recoder is to serve as a sophisticated loss function to help train the summarizer, backpropagating errors to the summarizer's decoded output sequence during training. The intuition is that a good output sequence from the summarizer should allow the recoder to produce a good summary. In principle, the recoder can be any sequence to sequence model that can backpropagate to its inputs. One obvious choice is to use the same model structure as the summarizer itself. For our experiments we found it was sufficient to use the same network structure with lower dimensions, with the same attentional pointer-generator encoder-decoder network as the summarizer, but with half the number of hidden dimensions (from 256 to 128) and a instead of an LSTM in the encoder. This helped reduce the amount of memory required in training. Our first task is to represent the summarizer's decoded outputs as inputs to the recoder in a way that is differentiable. A beam search decoder will find a sequence (y 1, y 2, ...) of high average log probability − log P (y t |y 1, ..., y t−1) over timesteps t. We cannot use this discrete token sequence directly, but we can look at the underlying signals that determine the choice of each token. The output token y t is chosen based on the computed probability distributions P t (y t |y 1, ..., y t−1). Let us denote this distribution by P t for short. For a beam search of width k, the chosen token y t will have a probability in P t that is among the k highest. The exact choice is determined by the beam search mechanism, so we do not have a continuous function that relates P t directly to y t. However, P t does determine the range of choices, and feeding it as an input to the recoder can ensure that it contains the right information. Since each P t is computed by the summarizer based on the decoded prefix sequence (y 1, ..., y t−1), propagating errors to P t improves the summarizer via exposure to what it would see at test time, even if the summarizer is not "aware" of the search mechanism and cannot optimize for it. Since P t has dimensions equal to the size of the vocabulary (v = 50000 in the experiments), we multiply P t by a weight matrix to produce a dense representation in a lower dimensional space (128 in the experiments), just as we did for the one-hot token inputs of the summarizer. While we can use a new weight matrix here, reusing the same embedding matrix W emb as the summarizer helps avoid increasing the number of parameters. For the same reason we also reuse the summarizer's mapping V back to vocabulary space. Together they account for 90% of the summarizer's parameters. The input to the recoder is the sequence (w R t) where w R t = P t W emb with P t treated as a vector of length v. By training the recoder to output the reference summary, errors are propagated to its inputs w R t and then to P t. Note that although the summarizer's input w i is in effect W emb multiplied with one-hot vectors corresponding to x i, P t is the output of a softmax function and will be more evenly distributed than a one-hot vector. Embedded representations w R t will thus be a weighted sum of embedding vectors instead of individual ones like w i. The recoder can accommodate this difference since it is an independent network with its own parameters. The recoder can be trained jointly with the summarizer from scratch. We have found that it also suffices to add the recoder to the pretrained summarizer and continue training jointly using their combined losses. The recoder is trained using teacher forcing to produce the reference summary, analogously to J ml. To be specific, we minimize the loss function J R ml equal to the average of − log P R (y * t |y * 1, ..., y * t−1) across timesteps t, where P R denotes the recoder's output distributions. This maximum likelihood loss is sufficient for the recoder because its output is never used for decoding at test time. Meaningful recoder output depends on relevant information from the original article being encoded in its input w R t. However, J R ml only places requirements on the presence of information. We have not placed any requirements on brevity, so longer sequences w R t would likely yield better under this loss metric, barring any confounding effects from too much information. The effect is only constrained because recoder training is performed jointly with the summarizer, and summaries that are longer than reference summaries would tend to do worse with respect to J s. Intuitively, J R ml encourages recall of information. We can add a loss function on length to counterbalance for precision. The end of an output sequence is determined when the special STOP token is output. Actual words are likely to convey more useful information to the recoder, so training using J R ml lowers the STOP token's output probability. We can control length by applying a loss to every other token for timesteps beyond a desired length. We define the length loss as the average over t of The function Penalty(t) ≥ 1 defines a penalty curve for timesteps beyond a desired length, while hyperparameter λ defines the desired tradeoff between precision and recall. Other ways to control output length are possible, such as by explicitly adding the desired length into decoder state; , although these methods require changes to the model. Article some teenagers get driving lessons from their parents. other teens are taught by licensed instructors. but malia obama is n't your average 16-year-old: her driving lessons were provided by the u.s. secret service. asked who taught malia how to drive, first lady michelle obama told celebrity chef and daytime talk-show host rachael ray in an interview that it was the armed agents who provide around-the-clock security for the family. [...] pgen cov first lady michelle obama told celebrity chef and daytime talk-show host rachael ray in an interview that it was the armed agents who provide around-the-clock security for the family. mrs. obama has n't driven herself in seven or eight years. pgen cov+recoder malia obama, seen with her mother michelle obama in april 2009, reportedly was taught how to drive by secret service agents. but malia obama is n't your average 16-year-old: her driving lessons were provided by the u.s. secret service. asked who taught malia how to drive, first lady michelle obama said in an interview that it was the armed agents who provide around-the-clock security for the family. reference michelle obama told talk-show host rachael ray that secret service agents taught her daughter malia how to drive. mrs. obama has n't driven herself in seven or eight years, she said. she added that driving gives malia' a sense of normalcy,' helping her feel like the rest of her friends who are also driving. The final recoder loss J R is comprised of the teacher forced recoder loss and the length loss Abstracting away individual components of the summarizer's and recoder's loss functions, the end to end training is performed using a combination of their losses J: In Figure 1 comparing output summaries from the baseline pgen cov model trained with J S (described in Experiments section below) and the pgen cov+recoder model additionally trained with loss J S + J R, the latter's output contains mention of "malia", a relevant name in the article that also appears in the reference summary. If we trained only with the recoder loss J R, we would encourage the summarizer to output the right information, but the output may not conform to a language model, except to the extent it helps make the output more intelligible for the recoder. For the purpose of illustration, we continued training the pgen cov model using only J R. It produced the following summary for one of the articles in the CNN/DailyMail dataset: boston's miserable winter is now also its snowiest season it had a special 2.9 inches pushed the city into 108.6 inches in one season. This example output contains some relevant information but has grammatical errors. Summarization is a well studied problem, including both extractive and abstractive approaches;;. Many of the existing abstractive models are based on encoder-decoders and trained using teacher forcing. In this work we focus on improving the training of models such as these, and we have picked one particular model as the baseline for improvement. Many aspects of the By using an unspecialized model with commonly occurring elements as baseline, we hope the same concepts can apply to more advanced and specialized models. There are a number of related techniques to address exposure bias from teacher forcing in a sequence learning problem. One class of such techniques is motivated by the ideas of reinforcement learning (RL) for training actors to make discrete decisions The encoder-decoder is analogous to an actor being trained to learn a policy for choosing an output word at each time step. These decisions are evaluated in training not based on its output probabilities, but based on a reward function defined on the full output sequence, selected via beam search or other methods. Broadly speaking, there are a couple ways to turn rewards into gradients for training probability distributions at each timestep. One technique is the REINFORCE algorithm, Cast in these terms, the recoder serves the role of both components. However, instead of training toward a heuristic such as ROUGE, the recoder uses an encoder-decoder network, allowing it to account for complexities in evaluation of quality that are as sophisticated as the model being trained. An algorithm can only be as good as the metric toward which it is trained, and the recoder helps ease this upper bound on quality. In short, the key difference from reinforcement learning approaches is that the recoder replaces a simple reward heuristic with an encoder-decoder and its loss function. Another approach to account for the beam search decoded output sequence is. In this work, beam search is performed as part of training, and a loss is defined that penalizes the reference sequence prefix falling out of the beam. on two nonsummarization word sequence tasks, a differentiable approximation to beam search is used, and loss is defined on the sequence using Hamming distance. While these approaches can account for the decoder test time sequence output, they do not have the flexibility to credit alternative summaries that differ slightly in phrasing. In comparison, our approach can account for a range of correct summaries that differ from one another, regardless of subsequence overlap. Since the recoder is itself a type of summarization network, it too can keep pace should further improvements in summarizers be developed that output more varied summaries of high quality. However, using a differentiable approximation to beam search enhances backpropagation, and may be a direction for future improvement. The idea of re-encoding an output summary also appears in , where StraightThrough Gumbel-Softmax reparameterization is used to derive a loss function that allows backpropagation. Their reconstruction cycle loss variant is closest in concept to our work, except that since there is no reference summary in their problem, they train their analogue of the recoder to produce the original source. We did not take this approach because in general the summary is a lossy representation of the original, so training the recoder to produce it would subject it a large source of error that it cannot possibly reduce. In the extension to scheduled sampling for machine translation, a decoded sequence prefix is used in training to predict the next reference word after weighting over possible alignments, which allows for flexibility in word ordering. For the summarization problem, alternative summaries that differ by more than word alignment can still be of high quality. The idea of using a second model to help train the primary model has appeared in other contexts.; for machine translation, the decoded output of a translation model is fed as input into a second backtranslation model. The two separate models translate back and forth between source and target languages. We can think of the backtranslation model from target to source language as serving an analogous role to the recoder, although the aim of these works has been to generate synthetic training data to augment limited human training data, rather than to compute gradients directly in an end-to-end model for the purpose of improving the original network. In contrast to the translation problem, we also cannot expect the original article to be recreated from the summary, which in general will be a lossy representation of the article, so the recoder is asked to recreate the reference summary and not the original article. An analogous idea also appears in the problem of sentence compression; Févry & , where decoding from a shortened version of a sentence back to the original sentence is analogous to backtranslating from target to source language. In the related work on dual learning, the two models for the forward and backward problems share parameters to mutual benefit. In contrast, the recoder shares no parameters with the summarizer except vocabulary mappings, relying instead on direct backpropagation. , the decoded output sequence of a translation model is used to train a student model to mimic the original model's output. The aim there is to better distill knowledge into a smaller student model, not to improve the original model. We ran experiments to test the effectiveness of adding the loss function to the attentional encoderdecoder summarization model of. The Tensorflow code, which is based on their published code, will also be made public along with trained models. We started with the provided trained summarizer models and added on recoder losses and continued training. The additional training consumes about double the memory and computation time, as it requires a beam search decoding as part of training, using a beam width of 4. The additional time was about 24 hours for each model, using an Nvidia Tesla P100 GPU on a cloud service provider. Test times are not affected since the recoder is only used during training. We ran experiments using the two versions of the model from , with and without the coverage mechanism, as baselines. The model with coverage is pgen cov, while pgen omits the coverage mechanism. In terms of loss functions, pgen cov is trained using a loss of J S = J ml +J cov as presented above, while pgen is trained using J S = J ml. In either case, the comparison models with recoder is additionally trained using J S + J R. We used the summarizer's (P vocab) t directly in place of P t in computing recoder inputs, bypassing backpropagation to the pointer-generator mechanism, which is most helpful for out-of-vocabulary words that will be treated as UNK for w R t anyway. This yielded similar while converging faster and requiring less computational resources. We fixed the length penalty Penalty(t) to be a graduated curve 1.04 where L is the length of the reference summary for the example. We experimented with different settings of λ to see its effects on output lengths and quality. The final setting of λ = 0.1 used for the ROUGE comparison in Table 1, discussed below, was selected based on its having the highest ROUGE-1 score on the validation set, where scores appeared in the same relative order as on the test set shown in Table 2. In line with the baseline and for comparison with previous work, we first assess quality using the ROUGE metric. The recoder can in principle capture more complex notions of quality than ROUGE, so this heuristic cannot fully capture all aspects of improvement that the human evaluations described below can. Nonetheless, since we do not train toward this metric, it should serve as an independent indicator of overall quality improvements. , e.g. 39.47 vs 39.53 for ROUGE-1. The pretrained models provided with their published code ed in lower scores (differences may be due to changes in the underlying software libraries over time), but we were able to achieve these comparable scores by further training with a lower learning rate. The are shown in Table 1. For pgen, we see that adding the recoder loss in an improvement in scores from (36.08, 15.60, 32.95) to (37.07, 16.02, 33.83). For the higher scoring pgen cov baseline, adding the recoder loss improves scores from (39.47, 17.37, 36.26) to (40.44, 18.15, 36.90). In both cases we see close to a 1 point improvement in ROUGE-1 scores, and a smaller improvement in ROUGE-2 and ROUGE-L scores versus the baseline. We see that adding the recoder to two variations of a strong abstractive summarizer has improved their performance by a significant amount, even though we have not changed the models, only the loss function used to train them. These scores are also an improvement over using the first 3 sentences from the article as the summary (lead-3), which had outperformed the baseline models. To get a sense of the magnitude of improvement, we look at one recently published model that uses the same non-anonymized version of the dataset without further processing, allowing for a relatively direct comparison. The "hierarchical" model of is an abstractive model that accounts for sentence level structure. Overall are close to pgen cov+recoder, which has higher ROUGE-1 and ROUGE-2 scores but lower ROUGE-L. Applying the recoder to a generic baseline has ed in ROUGE gains that match that of a model with more advanced mechanisms. There are other sophisticated abstractive models with higher reported ROUGE scores, especially those that are extractive hybrids or employ RL techniques;; , with a high of 41.69 ROUGE-1 score in the latter work after eliminating duplicate trigrams in post-processing. If we also eliminated duplicate trigrams in post-processing, the pgen cov+recoder scores would be (40.71, 18.20, 37.12), but a full treatment should also eliminate duplicate trigrams during training, requiring a far more complex implementation. These models are generally more tuned to the specific problem and metric, such as accounting for sentence boundaries or including the ROUGE metric as reward, and it is reasonable that they achieve higher scores on this metric. Since the recoder has the potential to capture notions of similarity beyond n-gram overlap as used in ROUGE, we performed further evaluation using human readers. We ran human evaluation experiments comparing both the pgen cov+recoder λ = 0.1 model and the λ = 0.2 model against the pgen cov baseline. The λ = 0.2 model's average summary length of 68.7 is close to the baseline's 66.1, which minimizes differences due to length. From the CNN/DailyMail test set where the model gave beam search decoded outputs that differed from the baseline (95.2% for λ = 0.1 model and 95.5% for λ = 0.2 model, out of 11490 examples), we randomly sampled 300 examples from each model. Workers from Amazon Mechanical Turk were shown up to the first 400 tokens from the article (same as model inputs), the reference summary, and the two generated summaries, randomly assigned to side A and side B. They were asked to select an overall preference between A and B, and then preference in terms of readability and relevance. Since we do not need a confident score for any one example for the purpose of evaluating the model, we limited each example to one worker and each worker to 5 examples to increase diversity across example-worker pairs. Workers were asked the following questions with no other guidance on interpretation: • Which summary is better overall? • Which summary has better readability? • Which summary contains information that is more relevant? The preference are shown in Table 3. We see that pgen cov+recoder was preferred overall 60.3% (λ = 0.1) and 55.0% (λ = 0.2) of the time over the pgen cov baseline. If we account for the remaining 4.8% and 4.5% of cases where the models gave outputs identical to the baseline by assigning equal preference to them, the preference ratios would be adjusted slightly to 59.8%, 54.8% overall and 62.1%, 55.1% relevance respectively. The overall improvement may be largely explained by the improvement in relevance, as the relevance preference was different from overall preference for only 19 and 25 examples respectively. The most direct comparison from previous work may be the human evaluations of that had also compared their model (rnn-ext+abs+RL+rerank), with a ROUGE-1 score of 40.88, against the pgen cov baseline. They allowed a choice of "Equally good/bad" which our survey did not, but if we assign those ratings equally to the two sides, their suggest a preference for relevance 52.8% of the time. If we similarly split "equal" ratings in the head-to-head human evaluations in , their overall preference would be 59.3%. However, their comparison pitted a model (m7) with a ROUGE-1 score of 41.69 against a baseline model (m3) that had only achieved a ROUGE-1 score of 38.01, which is lower than our pgen cov comparison baseline. Differences such as post-processing and survey format make these comparisons imprecise, but they give us a sense of the magnitude of improvement reflected by a 54.8%-59.8% overall preference over the pgen cov baseline. We have presented the use of an encoder-decoder as a sophisticated loss function for sequence outputs in the problem of summarization. The recoder allows us to define a differentiable loss function on the decoded output sequence during training. Experimental using both ROUGE and human evaluations show that adding the recoder in training a general abstractive summarizer significantly boosts its performance, without requiring any changes to the model itself. In future work we may explore whether the general concept of using a model as loss function has wider applicability to other problems.

We present the use of a secondary encoder-decoder as a loss function to help train a summarizer.

Represent the TLDR for finding the corresponding scientific article!

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.