Question:
represent the scientific article to find a short tldr.
E.g.
"This paper presents a system for immersive visualization of Non-Euclidean spaces using real-time ray tracing. It exploits the capabilities of the new generation of GPU’s based on the NVIDIA’s Turing architecture in order to develop new methods for intuitive exploration of landscapes featuring non-trivial geometry and topology in virtual reality." == "Immersive Visualization of the Classical Non-Euclidean Spaces using Real-Time Ray Tracing." != "Interactively generating image from incrementally growing scene graphs in multiple steps using GANs while preserving the contents of image generated in previous steps"

Question: We present a deep generative model, named Monge-Amp\`ere flow, which builds on continuous-time gradient flow arising from the Monge-Amp\`ere equation in optimal transport theory. The generative map from the latent space to the data space follows a dynamical system, where a learnable potential function guides a compressible fluid to flow towards the target density distribution. Training of the model amounts to solving an optimal control problem. The Monge-Amp\`ere flow has tractable likelihoods and supports efficient sampling and inference. One can easily impose symmetry constraints in the generative model by designing suitable scalar potential functions. We apply the approach to unsupervised density estimation of the MNIST dataset and variational calculation of the two-dimensional Ising model at the critical point. This approach brings insights and techniques from Monge-Amp\`ere equation, optimal transport, and fluid dynamics into reversible flow-based generative models. Generative modeling is a central topic in modern deep learning research BID20 which finds broad applications in image processing, speech synthesis, reinforcement learning, as well as in solving inverse problems and statistical physics problems. The goal of generative modeling is to capture the full joint probability distribution of high dimensional data and generate new samples according to the learned distribution. There have been significant advances in generative modeling in recent years. Of particular interests are the variational autoencoders (VAEs) BID33 BID47, generative adversarial networks (GANs) BID19, and autoregressive models BID18 BID34 BID55 a; BID44. Besides, there is another class of generative models which has so far gained less attention compared to the aforementioned models. These models invoke a sequence of diffeomorphism to connect between latent variables with a simple base distribution and data which follow a complex distribution. Examples of these flow-based generative models include the NICE and the RealNVP networks BID12, and the more recently proposed Glow model BID32. These models enjoy favorable properties such as tractable likelihoods and efficient exact inference due to invertibility of the network. A key concern in the design of flow-based generative models has been the tradeoff between the expressive power of the generative map and the efficiency of training and sampling. One typically needs to impose additional constraints in the network architecture BID12 BID46 BID34 BID44, which unfortunately weakens the model compared to other generative models. In addition, another challenge to generative modeling is how to impose symmetry conditions such that the model generates symmetry related configurations with equal probability. To further unleash the power of the flow-based generative model, we draw inspirations from the optimal transport theory BID56 BID57 BID45 and dynamical systems BID30. Optimal transport theory concerns the problem of connecting two probability distributions p(z) and q(x) via transportation z → x at a minimal cost. In this context, the Brenier theorem BID4 states that under the quadratic distance measure, the optimal generative map is the gradient of a convex function. This motivates us to parametrize the vector-valued generative map as the gradient of a scalar potential function, thereby formulating the generation process as a continuous-time gradient flow BID0. In this regard, a generative map is naturally viewed as a deterministic dynamical system which evolves over time. Numerical integration of the dynamical system gives rise to the neural network representation of the generative model. To this end, E proposes a dynamical system perspective to machine learning, wherein the training procedure is viewed as a control problem, and the algorithm like back-propagation is naturally derived from the optimal control principle BID36. Moreover, implemented the generative map as an ODE integration and employed efficient adjoint analysis for its training. In this paper, we devise the Monge-Ampère flow as a new generative model and apply it to two problems: density estimation of the MNIST dataset and variational calculation of the Ising model. In our approach, the probability density is modeled by a compressible fluid, which evolves under the gradient flow of a learnable potential function. The flow has tractable likelihoods and exhibits the same computational complexity for sampling and inference. Moreover, a nice feature of the MongeAmpère flow is that one can construct symmetric generative models more easily by incorporating the symmetries into the scalar potential. The simplicity and generality of this framework allow the principled design of the generative map and gives rise to lightweight yet powerful generative models. Consider latent variables z and physical variables x both living in R N. Given a diffeomorphism between them, x = x(z), the probability densities in the latent and physical spaces are related by p(z) = q(x) det ∂x ∂z. The Brenier theorem BID4 implies that instead of dealing with a multi-variable generative map, one can consider a scalar valued generating function x = ∇u(z), where the convex Brenier potential u satisfies the Monge-Ampère equation BID5 p DISPLAYFORM0 Given the densities p and q, solving the Monge-Ampère equation for u turns out to be challenging due to the nonlinearity in the determinant. Moreover, for typical machine learning and statistical physics problems, an additional challenge is that one does not even have direct access to both probability densities p and q. Instead, one only has independent and identically distributed samples from one of them, or one only knows one of the distributions up to a normalization constant. Therefore, solving the Brenier potential in these contexts is a control problem instead of a boundary value problem. An additional computational challenge is that even for a given Brenier potential, the righthand side of equation 1 involves the determinant of the Hessian matrix, which scales unfavorably as O(N 3) with the dimensionality of the problem. To address these problems, we consider the Monge-Ampère equation in its linearized form, where the transformation is infinitesimal BID56. We write the convex Brenier potential as u(z) = ||z|| 2 /2 + ϕ(z), thus x − z = ∇ϕ(z), and correspondingly ln q(x) − ln p(z) = − Tr ln I + DISPLAYFORM1 In the last equation we expand the logarithmic function and write the trace of a Hessian matrix as the Laplacian operator. Finally, taking the continuous-time limit → 0, we obtain DISPLAYFORM2 DISPLAYFORM3 such that x = z, p(x, 0) = p(z), and p(x, T) = q(x), where t denotes continuous-time and T is a fixed time horizon. For simplicity here, we still keep the notion of x, which now depends on time. The evolution of x from time t = 0 to T then defines our generative map. We notice that used a more general form of these two equations as a continuous-time normalizing flow BID46.The two ordinary differential equations (ODEs) compose a dynamical system, which describes the flow of continuous random variables and the associated probability densities under iterative changeof-variable transformation. To match p(x, T) and the target density q(x), one can optimize a functional I[p(x, T), q(x)] that measures the difference between p(x, T) and q(x). Thus, the training process amounts to solving an optimal control problem for the potential function: DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 which is the Liouville equation, namely, the continuity equation of a compressible fluid with density p(x, t) and velocity field v = ∇ϕ(x). Obeying the continuity equation ensures that the flow conserves the total probability mass. Moreover, the velocity field is curl free ∇ × v ≡ 0 and the fluid follows a form of gradient flow BID0. The irrotational flow matches one's heuristic expectation that the flow-based generative model transports probability masses. It should be stressed that although we use the optimal transport theory to motivate model architecture design, i.e., the gradient flow for generative modeling, we do not have to employ the optimal transport objective functions. The difference is that in generative modeling one typically fixes only one end of the probability density and aims at learning a suitable transformation to reach the other end. While for optimal transport one has both ends fixed and aims at minimizing the transportation cost. BID1; BID3 BID17 adapted the Wasserstein distances in the optimal transport theory as an objective function for generative modeling. We parametrize the potential function ϕ(x) using a feedforward neural network and integrate the ODEs and to transform the data and their log-likelihoods. Then, by applying the backpropagation algorithm through the ODE integration, we tune the parametrized potential function so that the probability density flows to the desired distribution 1. DISPLAYFORM0 Figure 2: Schematic illustration of two applications (a) unsupervised density estimation and (b) variational free energy calculation for a statistical mechanics problem. In both cases, we integrate equations 2 and 3 under a parametrized potential function ϕ(x), and optimize ϕ(x) such that the density at the other end matches to the desired one. In general, one can represent the potential function in various ways as long as the gradient and Laplacian are well defined. Here we adopt a densely connected neural network with only one hidden layer in our minimalist implementation. We use the softplus activation function in the hidden layers so that the potential function is differentiable to higher orders. The gradient and the Laplacian of the potential appearing in the right-hand side of equations 2 and 3 can be computed via automatic differentiation. We integrate equations 2 and 3 using a numerical ODE integrator. At each integration step one advances the ODE system by a small amount, as shown in FIG1. Therefore, the numerical integration of ODE is equivalent to the forward evaluation of a deep residual neural network BID27 BID14 BID39 BID6 BID41 BID48. Alternatively, the integration can also be viewed as a recurrent neural network in which one feeds the output back into the block iteratively. In both pictures, the network heavily shares parameters in the depth direction since the gradient and Laplacian information of the same potential function is used in each integration step. As a , the network is very parameter efficient. We employ the fourth order Runge-Kutta scheme with a fixed time step, which is set such that the numerical integrator is accurate enough. Thus, the ODE integrator block shown in figure 1(b) contains four layers of neural networks. With a hundred integration steps, the whole ODE integration procedure corresponds to four hundreds layers of a neural network. At the end of the integration, we obtain samples x and their likelihoods ln p(x, T), which depends on the parametrized potential function ϕ(x). Differentiable optimization of such deep neural network is feasible since the integrator only applies small changes to the input signal of each layer. The deep generative model ing from the discretization of an ODE system shows some nice features compared to the conventional neural networks. At training time, there is then a tradeoff between the total number of integration steps and the expressibility of the potential function. Longer integration time corresponds to a deeper network, and hence a simpler transformation at each step. While at testing time, one can construct a variable depth neural network for the potential function. For example, one can use a larger time step and a smaller number of integration steps for efficient inference. Moreover, by employing a reversible ODE integrator, one can also integrate the equations 2 and 3 backward in time with the same computational complexity, and return to the starting point deterministically. We apply the Monge-Ampère flow to unsupervised density estimation of an empirical dataset and variational calculation of a statistical mechanics problem. Figure 2 illustrates the schemes of the two tasks. To draw samples from the model and to evaluate their likelihoods, we simulate the fluid dynamics by integrating the ODEs and. We use the KL divergence as the measure of dissimilarity between the model probability density and the desired ones. Moreover, we choose the base distribution at time t = 0 to be a simple Gaussian p(x, 0) = N (x). See Appendix C for a summary of the hyperparameters used in the experiments. First we perform the maximum likelihood estimation, which reduces the dissimilarity between the empirical density distribution π(x) of a given dataset and the model density p(x, T) measured by the KL-divergence D KL (π(x) p(x, T)). It is equivalent to minimize the negative log-likelihood (NLL): DISPLAYFORM0 To compute the model likelihood we start from MNIST samples and integrate backwards from time T to 0. By accumulating the change in the log-likelihood 0 T d ln p(x(t), t) we obtain an estimate of the objective function in equation 6. To model the MNIST data we need to transform the images into continuous variables. Following BID44, we first apply the jittering and dequantizing procedure to map the original grayscale MNIST data to a continuous space. Next, we apply the logit transformation to map the normalized input to the logit space x → logit(λ + (1 − 2λ)x), with λ = 10 −6. FIG2 shows the training and test NLL compared with the best performing MADE BID18, Real-NVP BID13, and MAF BID44 ) models, all reported in BID44. Note that we have selected the with standard Gaussian base distribution for a fair comparison. The test NLL of the present approach is lower than previously reported values, see TAB0 for a summary. The Monge-Ampère flow is quite parameter-efficient as it only uses about one-tenth of learnable parameters of the MAF model BID44.The way we carry out density estimation in figure 2(a) is equivalent to transforming the data distribution π(x) to the base Gaussian distribution BID44. BID43 BID38. Inset shows representative Ising configurations generated at epochs 0, 500, 1000, 1500, respectively. Each inset contains a 5 × 5 tile of a 16 2 Ising model. In the lower panel, the generated samples exhibit two domains because we still impose the Z 2 inversion symmetry in the network. Without imposing the inversion symmetry the model will generate almost all black/white images due to the ferromagnetic correlations.ization of the Gaussianization technique BID8. Conversely, by integrating the equations forward in time one can use the flow to map Gaussian noises to meaningful images. For variational calculation we minimize the reverse KL divergence between the model and a given Boltzmann distribution of a physical problem D KL p(x, T) DISPLAYFORM0 an unknown partition function due to its intractability. In practical this amounts to minimizing the following expectation DISPLAYFORM1 To draw samples from p(x, T) we start from the base distribution x ∼ p(x, 0) = N (x) and evolve the samples according to equation 2. We keep track of the likelihoods of these samples by integrating equation 3 together. The loss function equation 7 provides a variational upper bound to the physical free energy − ln Z, since DISPLAYFORM2 the objective function employs the reparametrization pathwise derivative for the gradient estimator, which exhibits low variance and scales to large systems BID33.We apply this variational approach to the Ising model, a prototypical problem in statistical physics. The Ising model exhibits a phase transition from ferromagnetic to paramagnetic state at the critical temperature BID43. Variational study of the Ising model at the critical point poses a stringent test on the probabilistic model. To predict the thermodynamics accurately, the flow has to capture long-range correlations and rich physics due to the critical fluctuation. In the continuous representation BID15 BID61 BID38, the energy function of the Ising model reads (Appendix B) DISPLAYFORM3 We set the coupling K ij = (1 + √ 2)/2 to be at the critical temperature of square lattice Ising model BID43, where i, j are nearest neighbors on the square periodic lattice. The expectation inside the bracket of equation 7 is the energy difference between the model and the target problem. Since one has the force information for both target problem and the network as well, one can introduce an additional term in the loss function for the force difference BID11 BID59 ).The Ising model on a periodic square lattice has a number of symmetries, e.g., the spin inversion symmetry, the spatial translational symmetry, and the D 4 rotational symmetry. Mathematically, the symmetries manifest themselves as an invariance of the energy function E(x) = E(gx), where g is the symmetry operator that inverts or permutes the physical variables. For physics applications, it is essential to have a generative model that respects these physical symmetries, such that they will generate symmetry-related configurations with equal probability. Although simple symmetries such as the Z 2 inversion can be implemented by careful design of the network or simply averaging over the elements in the symmetry group BID38, respecting even more general symmetries in a generative model turns out to be a challenging problem. The Monge-Ampère flow naturally solves this problem by imposing the symmetry condition on the scalar potential function since the initial Gaussian distribution respects the maximum symmetry. There have been a number of studies about how to ensure suitable symmetries in the feedforward networks in the context of discriminative modeling (; BID10 . Here we employ a simpler approach to encode symmetry in the generating process. We express the potential as an average over all elements of the symmetry group G under consideration ϕ(x) = 1 |G| g∈Gφ (gx), where theφ in each term shares the same network parameters. At each step of the numerical integration, we sample only one term in the symmetric average to evaluate the gradient and the Laplacian of the potential. Thus, on average one restores the required symmetry condition in the generated samples. This is feasible both for training and data generation since equations 2 and 3 are both linear with respect to the potential function. The upper panel of Figure 4 shows that the variational loss equation 7 decreases towards the exact solution of the free energy (Appendix B). The achieved 1% relative accuracy in the variational free-energy is comparable to the previously reported value (0.5%) using a network which exploits the two-dimensional nature of the problem BID38. The inset shows the directly generated Ising configurations from the network at different training stage. At late stages, the network learned to produce domains of various shapes. One also notices that the generated samples exhibit a large variety of configurations, which respects the physical symmetries. Moreover, the network can estimate log-likelihoods for any given sample which is valuable for the quantitative study of the physical problem. The invertible flow can also perform inverse mapping of the energy function from the physical space to the latent space. Therefore, it can be used to accelerate Monte Carlo simulations, by either recommending the updates as a generative model BID28 BID40 or performing hybrid Monte Carlo in the latent space BID38. Figure 4 also compares the performances of the generative map with and without translational and rotational symmetries imposed. It is noticeable that using the same parameters in the network and the same training strategy, the variational free energy of the one without symmetry constraint is significantly higher than the one with symmetry constraint. Moreover, by inspecting the generated samples, one sees that the model distribution collapses to the ones with two horizontal domains. These symmetry breaking configuration correspond to a metastable local minimum of the free energy landscape. This comparison shows that imposing the necessary symmetry conditions is crucial for variational studies of physical problems using generative models Normalizing flows: The present approach is closely related to the normalizing flows, where one composes bijective and differentiable transformations for generative modeling . To scale to large dataset it is crucial to make the Jacobian determinant of the transformation efficiently computable. BID12; BID38; BID32 strive the balance between expressibility and efficiency of the bijective networks by imposing block structures in the Jacobian matrix. Our approach reduces the cubical scaling Hessian determinant to the linear scaling Laplacian calculation by taking the continuous-time limit. The continuoustime gradient flow is more flexible in the network design since bijectivity and the efficiently tractable Jacobian determinant is naturally ensured.(Inverse) autoregressive flows: The autoregressive property could be interpreted as imposing a triangular structure in the Jacobian matrix of the flow to reduce its computational cost. There is a trade-off between the forward and inverse transformation. Therefore, typically the autoregressive flows are used for density estimation BID44, while the inverse autoregressive flows are suitable for posterior in variational inference BID34. Moreover, these flows are only implicitly reversible, in the sense that the inverse calculation requires solving a nonlinear equation. However, the continuous-time gradient flow has the same time complexity for both the forward and inverse transformation. derived the continuous-time limit of the normalizing flows BID46, which is a general form of equations 2 and 3 without the irrational condition on the velocity field. Besides conceptual connection to the Brenier theorem in optimal transport theory, the motivation of using the gradient flow instead of more general transformations in is that it is convenient to encode symmetries in the scalar potential function. Furthermore, BID21 made use of the Hutchinson's trace estimator BID29 to simplify the computation of the trace of Jacobian, which we could also employ for the Laplacian in equation FORMULA3. BID50 FORMULA0 consider generative models based on diffusion processes. Our work is different since we consider deterministic and reversible advection flow of the fluid. There is, in general, no stochastic force in our simulation (except the random sampling of symmetric functions done in section 4.2). And, we always integrate the flow equation for a finite time interval instead of trying to reach a steady probability density in the asymptotic long time limit. Dynamical system and control based methods: In a more general perspective, our approach amounts to defining a dynamical system with a target terminal condition, which is naturally viewed as a control problem (E, 2017). Along with this direction, BID24 BID28 offer some insights into interpreting and solving this control problem. Moreover, the Neural ODE implements an efficient back-propagation scheme through blackbox ODE integrators based on adjoint computation, which we did not utilize at the moment. Gradient flow of compressible fluids in a learnable potential provides a natural way to set up deep generative models. The Monge-Ampère flow combines ideas and techniques in optimal transport, fluid dynamics, and differential dynamical systems for generative modeling. We have adopted a minimalist implementation of the Monge-Ampère flow with a scalar potential parameterized by a single hidden layer densely connected neural network. There are a number of immediate improvements to further boost its performance. First, one could extend the neural network architecture of the potential function in accordance with the target problem. For example, a convolutional neural network for data with spatial or temporal correlations. Second, one can explore better integration schemes which exactly respect the time-reversal symmetry to ensure reversible sampling and inference. Lastly, by employing the backpropagation scheme of through the ODE integrator one can reduce the memory consumption and achieve guaranteed convergence in the integration. Furthermore, one can employ the Wasserstein distances BID1 instead of the KLdivergence to train the Monge-Ampère flow. With an alternative objective function, one may obtain an even more practically useful generative model with tractable likelihood. One may also consider using batch normalization layers during the integration of the flow BID13 BID44. However, since the batch normalization breaks the physical interpretation of the continuous gradient flow of a fluid, one still needs to investigate whether it plays either a theoretical or a practical role in the continuous-time flow. Moreover, one can use a time-dependent potential ϕ(x, t) to induce an even richer gradient flow of the probability densities. BID2 has shown that the optimal transport flow (in the sense of minimizing the spatial-time integrated kinetic energy, which upper bounds the squared Wasserstein-2 distance) follows a pressureless flow in a time-dependent potential. The fluid moves with a constant velocity that linearly interpolates between the initial and the final density distributions. Practically, a time-dependent potential corresponds to a deep generative model without sharing parameters in the depth direction as shown in FIG1. Since handling a large number of independent layers for each integration step may be computationally inefficient, one may simply accept one additional time variable in the potential function, or parametrize ϕ(x, t) as the solution of another differential equation, or partially tie the network parameters using a hyper-network BID22.Besides applications presented here, the Monge-Ampère flow has wider applications in machine learning and physics problems since it inherits all the advantages of the other flow-based generative models BID12 BID46 BID34 BID13 BID44. A particular advantage of generative modeling using the Monge-Ampère flow is that it is relatively easy to impose symmetry into the scalar potential. It is thus worth exploiting even larger symmetry groups, such as the permutation for modeling exchangeable probabilities BID35. Larger scale practical application in statistical and quantum physics is also feasible with the Monge-Ampère flow. For example, one can study the physical properties of realistic molecular systems using Monge-Ampère flow for variational free energy calculation. Lastly, since the mutual information between variables is greatly reduced in the latent space, one can also use the Monge-Ampère flow in conjunction with the latent space hybrid Monte Carlo for efficient sampling BID38. To gain intuition about the probability density flow under the Monge-Ampère equation, we work out a close form solution of equations 2 and 3 in a one dimensional toy problem. Since a quadratic potential merely induces scale transformation to the data, a Gaussian distribution will remain to be a Gaussian with a different variance. Thus, we consider ϕ(x) = λx 2 /2, and p(x, 0) = N (x), and parametrize the time dependent density as p(x, t) = DISPLAYFORM0, and α = 1. Substitute this ansatz of the time-dependent probability density into equation 3, we have DISPLAYFORM1 where we used equation 2, i.e. = −λ, and α(t) = exp(−λt). Therefore, under a quadratic potential the fluid parcel moves with acceleration. The ones that are far away from the origin move faster. And the width of the Gaussian distribution changes exponentially with time. The Ising model is a fundamental model in statistical mechanics which allows exact solution in two dimension BID43. The Ising model partition function reads DISPLAYFORM0 To make it amenable to the flow approach we reformulate the Ising problem to an equivalent representation in terms of continuous variables following BID15 BID61 BID38. First, we offset the coupling to K + αI such that all of its eigenvalues are positive, e.g. the minimal eigenvalue of K + αI is 0.1. This step does not affect the physical property of the Ising model except inducing a constant offset in its energies. Next, using the Gaussian integration trick (Hubbard-Stratonovich transformation) we can decouple the Ising interaction with continuous auxiliary variables Z Ising ∝ s dx exp − We list the hyperparameters of the network and training in TAB1. in the integration step in the fourth order Runge-Kutta integration. d is the total number of integration steps. Thus T = d is the total integration time. h is the number of hidden neurons in the potential function. B is the mini-batch size for training.

Represent the TLDR for finding the corresponding scientific article.

A gradient flow based dynamical system for invertible generative modeling

represent the tldr for finding the corresponding scientific article

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Graph Neural Networks (GNNs) are a class of deep models that operates on data with arbitrary topology and order-invariant structure represented as graphs. We introduce an efficient memory layer for GNNs that can learn to jointly perform graph representation learning and graph pooling. We also introduce two new networks based on our memory layer: Memory-Based Graph Neural Network (MemGNN) and Graph Memory Network (GMN) that can learn hierarchical graph representations by coarsening the graph throughout the layers of memory. The experimental demonstrate that the proposed models achieve state-of-the-art in six out of seven graph classification and regression benchmarks. We also show that the learned representations could correspond to chemical features in the molecule data. Graph Neural Networks (GNNs) (; ; are a class of deep architectures that operate on data with arbitrary topology represented as graphs such as social networks , knowledge graphs , molecules , point clouds , and robots. Unlike regular-structured inputs with spatial locality such as grids (e.g., images and volumetric data) and sequences (e.g., speech and text), GNN inputs are variable-size graphs consisting of permutationinvariant nodes and interactions among them. GNNs such as Gated GNN (GGNN) , Message Passing Neural Network (MPNN) , Graph Convolutional Network (GCN) , and Graph Attention Network (GAT) learn node embeddings through an iterative process of transferring, transforming, and aggregating the node embeddings from topological neighbors. Each iteration expands the receptive field by one hop and after k iterations the nodes within k hops influence the node embeddings of one another. GNNs are shown to learn better representations compared to random walks , matrix factorization , kernel methods , and probabilistic graphical models . These models, however, cannot learn hierarchical representation as they do not exploit the graph compositionality. Recent work such as Differentiable Pooling (DiffPool) , TopKPool , and Self-Attention Graph Pooling (SAGPool) define parametric graph pooling layers that let models learn hierarchical graph representation by stacking interleaved layers of GNN and pooling layers. These layers cluster nodes in the latent space such that the clusters are meaningful with respect to the task. These clusters might be communities in a social network or potent functional groups within a chemical dataset. Nevertheless, these models are not efficient as they require an iterative process of message passing after each pooling layer. In this paper, we introduce a memory layer for joint graph representation learning and graph coarsening that consists of a multi-head array of memory keys and a convolution operator to aggregate the soft cluster assignments from different heads. The queries to a memory layer are node embeddings from the previous layer and the outputs are the node embeddings of the coarsened graph. The memory layer does not explicitly require connectivity information and unlike GNNs relies on the global information rather than local topology. These properties make them more efficient and improve their performance. We also introduce two networks based on the proposed layer: Memory-based Graph Neural Network (MemGNN) and Graph Memory Network (GMN). MemGNN consists of a GNN that learns the initial node embeddings, and a stack of memory layers that learns hierarchical graph representation up to the global graph embedding. GMN, on the other hand, learns the hierarchical representation purely based on memory layers and hence does not require message passing. Memory Augmented Neural Networks (MANNs) utilize external memory with differentiable read-write operators allowing them to explicitly access the past experiences and are shown to enhance reinforcement learning , meta learning , few-shot learning , and multi-hop reasoning. Unlike RNNs, in which the memory is represented within their hidden states, the decoupled memory in MANNs lets them to store and retrieve longer term memories with less parameters. The memory can be implemented as a key-value memory such as neural episodic control and product-key memory layers or as a array-structured memory such as Neural Turing Machine (NTM) , prototypical networks , memory networks, and Sparse Access Memory (SAM) . Our memory layer consists of a multi-head array of memory keys. Graph Neural Networks (GNNs) use message passing to learn node embeddings over graphs. GraphSAGE (b) learns embedding by sampling and aggregating neighbor nodes whereas GAT uses attention to aggregate embeddings from all neighbors. GCN models extend the convolution to arbitrary topology. Spectral GCNs (; ;) use spectral filters over graph Laplacian to define the convolution in the Fourier domain. These models are less efficient compared to spatial GCNs which directly define the convolution on graph patches centered on nodes. Our memory layer uses a feed-forward network to learn the node embeddings. Graph pooling can be done globally or hierarchically. In former, node embeddings are aggregated into a graph embedding using arithmetic operators such as sum or max (a;) or set neural networks such as Set2Set and SortPool . In latter, graphs are coarsened in each layer to capture the hierarchical structure. Non-parametric methods such as clique pooling , kNN pooling , and Graclus rely on topological information and are efficient, but are outperformed by parametric models such as edge contraction pooling . DiffPool trains two parallel GNNs to compute node embeddings and cluster assignments using a combination of classification loss, link prediction loss, and entropy loss, whereas Mincut pool trains a sequence of a GNN and an MLP using classification loss and the minimum cut objective. TopKPool computes a node score by learning a projection vector and then drops all the nodes except the top scoring nodes. SAGPool extends the TopKPool by using graph convolutions to take neighbor node features into account. We use a clustering-friendly distribution to compute the attention scores between nodes and clusters. We define a memory layer M (l): R n l ×d l −→ R n l+1 ×d l+1 in layer l as a parametric function that takes in n l query vectors of size d l and generates n l+1 query vectors of size d l+1 such that n l+1 < n l. The input and output queries represent the node features of the input graph and the coarsened graph, respectively. The memory layer learns to jointly coarsen the input nodes (i.e., pooling) and transform their features (i.e., representation learning). As shown in Figure 1, it consists of arrays of memory keys (i.e., multi-head memory) and a convolutional layer. Assuming |h| memory heads, a shared input query is compared against all the keys in each head ing in |h| attention matrices which are then aggregated into a single attention matrix using the convolution layer. In a content addressable memory (; ;, the task of attending to memory (i.e., addressing scheme) is formulated as computing the similarly between memory keys to a given query q. Specifically, the attention weight of key k j for query q is defined as w j = sof tmax(d(q, k j)) where d is a similarity measure, typically Euclidean distance or cosine similarity . The soft read operation on memory is defined as a weighted average over the memory keys: r = j w j k j. Figure 1: The proposed Architecture for hierarchical graph representation learning using the introduced memory layer. The query network projects the initial node attributes to a query embedding space and each memory layer jointly coarsens the input queries and transforms them into a new query space. In this work, we treat the input queries Q (l) ∈ R n l ×d l as the node embeddings of an input graph and treat the keys K (l) ∈ R n l+1 ×d l as the cluster centroids of the queries. To satisfy this assumption, we impose a clustering-friendly distribution as the distance metric between keys and a query. Following , we use the Student's t-distribution as a kernel to measure the normalized similarity between query q i and key k j as follows: where C ij is the normalized score between query q i and key k j (i.e., probability of assigning node i to cluster j or attention score between query q i and memory key k j) and τ is the degree of freedom of the Student's t-distribution (i.e., temperature). To increase the model capacity, we model the memory keys as a multi-head array. Applying a shared input query against the memory keys produces a tensor of cluster assignments [C |h|] ∈ R |h|×n l+1 ×n l where |h| denotes the number of heads. To aggregate the heads into a single assignment matrix, we treat the heads and the matrix rows and columns as depth, height, and width in standard convolution analogy and apply a convolution operator over them. Because there is no spatial structure, we use [1 × 1] convolution to aggregate the information across heads and therefore the convolution behaves as a weighted pooling that reduces the heads to a single matrix. The aggregated assignment matrix is computed as follows: where Γ φ is a [1 × 1] convolutional operator parametrized by φ, || is the concatenation operator, and C (l) is the aggregated soft assignment matrix. A memory read generates a value matrix V (l) ∈ R n l+1 ×d l that represents the coarsened node embeddings in the same space as the input queries and is defined as the product of the soft assignment scores and the original queries as follows: The value matrix is fed to a single layer neural network consisting of a weight matrix and a LeakyReLU activation function to project the coarsened embeddings from R n l+1 ×d l into R n l+1 ×d l+1 representing the output queries Q (l+1): Thanks to these parametrized transformations, a memory layer can jointly learn the node embeddings and coarsens the graph end-to-end. The computed queries Q (l+1) are the input queries to the subsequent memory layer M (l+1). For graph classification, one can simply stack layers of memory up to the level where the input graph is coarsened into a single node representing the global graph embedding and then feed it to a fully-connected layer to predict the graph class as follows: where Q 0 = f q (g) is the initial query embedding 1 generated by the query network f q over graph g. We introduce two architectures based on the memory layer: GMN and MemGNN. These two architectures are different in the way that the query network is implemented. More specifically, GMN uses a feed-forward network for initializing the query: f q (g) = FFN θ (g), whereas MemGNN implements the query network as a message passing GNN: f q (g) = GNN θ (g). A GMN is a stack of memory layers on top of a query network f q (g) that generates the initial query embeddings without any message passing. Similar to set neural networks and transformers , graph nodes in a GMN are treated as a permutation-invariant set of embeddings. The query network projects the initial node attributes into an embedding space that represents the initial query space. Assume a training set D = [g 1, g 2, ..., g N] of N graphs where each graph is represented as g = (A, X, Y) and A ∈ {0, 1} n×n denotes the adjacency matrix, X ∈ R n×din is the initial node attribute, and Y ∈ R n is the graph label. Considering that the GMN model treats a graph as a set of permutation-invariant nodes and does not use message passing, and also considering that the memory layers do not rely on connectivity information, the topological information of each node should be somehow encoded into its initial embedding. Inspired by transformers , we encode this information along with the initial attribute into the initial query embeddings using a query network f q implemented as a two-layer feed-forward neural network: where W 0 ∈ R n×din and W 1 ∈ R 2din×d0 are the parameters of the query networks, and || is the concatenation operator. Unlike the GMN architecture, the query network in MemGNN relies on the iterative process of passing messages and aggregating them to compute the initial query Q 0: where query network G θ is an arbitrary parameterized message passing GNN (; ; ;). In our implementation of MemGNN, we use a modified variant of GAT . Specifically, we introduce an extension to the original GAT model called edge-based GAT (e-GAT) and use it as the query network. Unlike GAT, e-GAT learns attention weights not only from the neighbor nodes but also from the input edge attributes. This is especially important for data containing edge information (e.g., various bonds among atoms represented as edges in molecule datasets). In an e-GAT layer, attention score between two neighbor nodes is computed as follows. where i→j denote the embedding of node i and the embedding of the edge connecting node i to its one-hop neighbor node j in layer l, respectively. W n and W e are trainable node and edge weights and W is the parameter of a single-layer feed-forward network that computes the attention score. We jointly train the model using two loss functions: a supervised classification loss and an unsupervised clustering loss. The supervised loss denoted as L ent is defined as the cross-entropy loss between the predicted and true graph class labels. The unsupervised clustering loss is inspired by deep clustering methods (; ;). It encourages the model to learn clustering-friendly embeddings in the latent space by urging it to learn from high confidence assignments with the help of an auxiliary target distribution. The unsupervised loss is defined as the Kullback-Leibler (KL) divergence loss between the soft assignments C (l) and the auxiliary distribution P (l) as follows: For the target distributions P (l), we use the distribution proposed in which normalizes the loss contributions and improves the cluster purity while emphasizing on the samples with higher confidence. This distribution is defined as follows: We define the total loss as follows where L is the number of memory layers and λ is a scalar weight. We initialize the model parameters, the keys, and the queries randomly and optimize them jointly with respect to L using mini-batch stochastic gradient descent. To stabilize the training, the gradients of L ent are back-propagated batch-wise while the gradients of L KL. This technique has also been applied in to avoid trivial solutions in deep clustering problem. We use nine graph benchmarks including seven classification and two regression datasets to evaluate the proposed method. These datasets are commonly used in both graph kernel (; ; ; ; ;) and GNN (; ; ;) literature. The summary of these datasets is as follows (i.e., first two benchmarks are regression tasks and the rest are classification tasks): ESOL contains water solubility data for compounds. Lipophilicity contains experimental of octanol/water distribution of compounds. Bace provides quantitative binding for a set of inhibitors of human β-secretase 1 (BACE-1). DD is used to distinguish enzyme structures from non-enzymes. Enzymes is for predicting functional classes of enzymes. Proteins is used to predict the protein function from structure. COLLAB is for predicting the field of a researcher given her For more information about the datasets and implementation details refer to Appendix A.2 and A.1, repectively. To evaluate the performance of our models on DD, Enzymes, Proteins, and COLLAB datasets, we follow the experimental protocol in and perform 10-fold cross-validation and report the mean accuracy over all folds. We also report the performance of four kernel-based methods including Graphlet , shortest path , Weisfeiler-Lehman (WL) , and WL Optimal Assignment , and ten deep models. The shown in Table 1 suggest that: (i) our models significantly improve the performance on DD, Enzymes, and Proteins datasets by absolute margins of 14.49%, 4.75%, and 0.49% accuracy, respectively, (ii) both proposed models achieve better performance on these three datasets compared to the baselines, (iii) MemGNN outperforms GMN on COLLAB whereas GMN achieves better on the Enzymes, Proteins, and DD datasets. On COLLAB, our models are outperformed by a variant of DiffPpool (i.e., diffpool-det) and WL Optimal Assignment . The former is a GNN augmented with deterministic clustering algorithm 2, whereas the latter is a graph kernel method. We speculate that because of the high edge-to-node ratio of COLLAB, these augmentations help in extracting near-optimal cliques. For the ESOL and Lipophilicity datasets, we follow the evaluation protocol in and report the Root-Mean-Square Error (RMSE) for these regression benchmarks. Considering that these datasets contain initial edge attributes (refer to Appendix A.2 for further details), we train the MemGNN model and compare the to the baseline models reported in including graph-based methods such as GCN, MPNN, Directed Acyclic Graph (DAG) based models, Weave as well as other conventional methods such as Kernel Ridge Regression (KRR) and Influence Relevance Voting (IRV). Tables 2 and 5 show that our MemGNN model achieves state-ofthe-art by absolute margin of 0.04 and 0.1 RMSE on ESOL and Lipophilicity benchmarks, respectively. For further details on these datasets and the baselines see . We also achieve state-of-the-art on the Bace, Reddit-Binary, and Tox21 datasets. For more details see Appendix A.3. To investigate the effect of the proposed e-GAT model, we train our MemGNN model using both GAT and e-GAT models as the query network. Considering that the ESOL, Lipophilicity, and BACE datasets contain edge attributes, we use them as the benchmarks. Since nodes have richer features compared to edges, we set the node and edge feature dimensions to 16 and 4, respectively. The comparative performance evaluation of the two models on the ESOL dataset is shown in Appendix A.4 demonstrating that e-GAT achieves better on the validation set in each epoch compared to the standard GAT model. We observed the same effect on Lipophilicity and BACE datasets too. To investigate the effect of the topological embeddings on the GMN model, we evaluated three initial topological features including adjacency matrix, normalized adjacency matrix, and Random Walk with Restart (RWR). For further details on RWR, see section A.5. The suggested that using the adjacency matrix as the initial feature achieves the best performance. For instance, 10-fold cross validation accuracy of a GMN model trained on ENZYMES with adjacency matrix, normalized adjacency matrix, and RWR is 78.66%, 77.16%, and 77.33%, respectively. We investigated two methods to down-sample the neighbors in dense datasets such as COLLAB (i.e., average of 66 neighbors per node) to enhance the memory and computation. The first method randomly selects 10% of the edges whereas the second method ranks the neighbors based on their RWR scores with respect to the center node and then keeps the top 10% of the edges. We trained the MemGNN model on COLLAB using both sampling methods which ed in 73.9% and 73.1% 10-fold cross validation accuracy for random and RWR-based sampling methods respectively, suggesting that random sampling performs slightly better than a random walk sampling. We stipulate that although keys represent the clusters, the number of keys is not necessarily proportional to the number of the nodes in the input graphs. In fact, datasets with smaller graphs might have more meaningful clusters to capture. For example, molecules are comprised of numerous functional groups and yet the average number of nodes in the ESOL dataset is 13.3. Moreover, our experiments show that for ENZYMES with average number of 32.69 nodes, the best performance is achieved with 10 keys whereas for the ESOL dataset 64 keys in the best performance. In ESOL 8, 64, and 160 keys in RMSE of 0.56, 0.52, and 0.54, respectively. We also observed that keeping the number of parameters fixed, increasing the number of memory heads improves the performance. For instance, when the model is trained on ESOL with 160 keys and 1 head, it achieves RMSE of 0.54, whereas when trained with 32 keys of 5 heads, the same model achieves RMSE of 0.53. Intuitively, the memory keys represent the cluster centroids and enhance the model performance by capturing meaningful structures and coarsening the graph. To investigate this intuition, we used the learned keys to interpret the knowledge learned by the models through visualizations. Figure 2 visualizes the learned clusters over atoms (i.e., atoms with same color are within the same cluster) indicating that the clusters mainly consist of meaningful chemical substructures such as a carbon chain and a Hydroxyl group (OH) (i.e., Figure 2a), as well as a Carboxyl group (COOH) and a benzene ring (i.e., Figure 2b). From a chemical perspective, Hydroxyl and Carboxyl groups, and carbon chains have a significant impact on the solubility of the molecule in water or lipid. This confirms that the network has learned chemical features that are essential for determining the molecule solubility. It is noteworthy that we tried initializing the memory keys using K-Means algorithm over the initial node embeddings to warm-start them but we did not observe any significant improvement over the randomly selected keys. We proposed an efficient memory layer and two deep models for hierarchical graph representation learning. We evaluated the proposed models on nine graph classification and regression tasks and achieved state-of-the-art on eight of them. We also experimentally showed that the learned representations can capture the well-known chemical features of the molecules. Our study indicated that node attributes concatenated with corresponding topological embeddings in combination with one or more memory layers achieves notable without using message passing. We also showed that for the topological embeddings, the binary adjacency matrix is sufficient and thus no further preprocessing step is required for extracting them. Finally, we showed that although connectivity information is not directly imposed on the model, the memory layer can process node embeddings and properly cluster and aggregate the learned embeddings. Limitations: In section 4.2, we discussed that on the COLLAB dataset, kernel methods or deep models augmented with deterministic clustering algorithm achieve better performance compared to our models. Analyzing samples in this dataset suggests that in graphs with dense communities, such as cliques, our model lacks the ability to properly detect these dense sub-graphs. Moreover, the of the DD dataset reveals that our MemGNN model outperforms the GMN model which implies that we need message passing to perform better on this dataset. We speculate that this is because the DD dataset relies more on local information. The most important features to train an SVM on this dataset are surface features which have local behavior. This suggest that for data with strong local interactions, message passing is required to improve the performance. Future Directions: We are planning to introduce a model based on the MemGNN and GMN architectures that can perform node classification by attending to the node embeddings and centroids of the clusters from different layers of hierarchy that the node belongs to. We are also planning to investigate the representation learning capabilities of the proposed models in self-supervised setting. A APPENDIX We implemented the model with PyTorch and optimized it using Adam optimizer. We trained the model for a maximum number of 2000 epochs and decayed the learning rate every 500 epochs by 0.5. The model uses batch-normalization , skip-connections, LeakyRelu activation functions, and dropout for regularization. We decided the hidden dimension and number of model parameters using random hyper-parameter search strategy. The best performing hyper-parameters for the datasets are shown in Table 3. A.2 DATASET STATISTICS Table 4 indicates the statistics of the datasets we used for graph classification and regression tasks. In addition to discussed in section 4.2, we report the evaluation on three other graph classification benchmarks including BACE , Tox21 (, and Reddit-Binary datasets. The BACE dataset provides quantitative binding for a set of inhibitors of human β-secretase 1 (BACE-1), whereas the is Tox21 dataset is for predicting toxicity on 12 different targets. We follow the evaluation protocol in and report the Area Under the Curve Receiver Operating Characteristics (AUC-ROC) measure for this task. Moreover, considering that the BACE and Tox21 datasets contain initial edge attributes, we train the MemGNN model and compare its performance to the baseline models reported in . The shown in Table 5 suggest that our model achieves stateof-the-art by absolute margin of 4.0% AUC-ROC on BACE benchmark. The also suggest that our model is competitive with the state-of-the-art GCN model on the Tox21 dataset. Reddit-Binary dataset is for predicting the type of community (i.e., question-answer-based or discussion-based communities) given a graph of online discussion threads. In this dataset, nodes represent the users and edges denote interaction between users. To evaluate the performance of our models on this dataset, we follow the experimental protocol in and perform 10-fold cross-validation to evaluate the model performance and report the mean accuracy over all folds. The reported in Table 6 show that the introduced GMN model achieves state-of-the-art accuracy by absolute margin of 0.44%. In section 4.3.1, we introduced e-GAT. Figure 3a illustrates the RMSE on the validation set of the ESOL dataset for a MemGNN model using GAT and e-GAT implementation as its query network, respectively. Since ESOL is a regression benchmark, we also plotted the R 2 score in Figure 3b. As shown in these figures, e-GAT performs better compared to GAT on both metrics. Consider a weighted or unweighted graph G. A random agent starts traversing the graph from node i and iteratively walks towards its neighbors with a probability proportional to the edge weight that 0.859 ± 0.000 0.907 ± 0.000 Table 6: Mean validation accuracy over 10-folds. Method REDDIT-BINARY Graphlet 78.04 WL 68.2 DiffPool 85.95 TopKPool 74.70 SAGPool 73.90 GMN (ours) 86.39 MemGNN (ours) 85.55 connects them. The agent also may restart the traverse from the starting node i with probability p. Eventually, the agent will stop at node j with a probability called relevance score of node j with respect to node i . The relevance score of node i with every other node of the graph is summarized in t i in equation 12 where t i is the RWR corresponding to node i, p is the restart probability, A is the normalized adjacency matrix and e i is the one-hot vector representing node i. t i = pÃ t i + (1 − p) e i Solving this linear system in r i defined in equation 13 Now we can nominate the nodes with higher relevance score w.r.t. node i for receiving messages in an MeMGNN or use them as topological embeddings in a GMN. Note that the restart probability in equation 13 defines how far the agent can walk from the source node and therefore the t i can represent whether local or global structures around the node i. We used p = 0.5 in our studies. Calculating the inverse of the adjacency matrix of a big graph is costly. Although we could exactly compute it for all of our datasets but there are existing methods to make the estimation more efficient . Figure 5: Clusters learned by a MeMGNN for ESOL and LIPO dataset. Chemical groups like OH (hydroxyl group), CCl3, COOH (carboxyl group), CO (ketone group) as well as benzene rings have been recognized during the learning procedure. These chemical groups are highly active and have a great impact on the solubility of molecules.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce an efficient memory layer that can learn representation and coarsen input graphs simultaneously without relying on message passing.

Represent the short summary for finding the corresponding scientific article (long)

We propose a new sparse and structured attention mechanism, TVmax, which promotes sparsity and encourages the weight of related adjacent locations to be the same.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Deep neural networks require extensive computing resources, and can not be efficiently applied to embedded devices such as mobile phones, which seriously limits their applicability. To address this problem, we propose a novel encoding scheme by using {-1,+1} to decompose quantized neural networks (QNNs) into multi-branch binary networks, which can be efficiently implemented by bitwise operations (xnor and bitcount) to achieve model compression, computational acceleration and resource saving. Our method can achieve at most ~59 speedup and ~32 memory saving over its full-precision counterparts. Therefore, users can easily achieve different encoding precisions arbitrarily according to their requirements and hardware resources. Our mechanism is very suitable for the use of FPGA and ASIC in terms of data storage and computation, which provides a feasible idea for smart chips. We validate the effectiveness of our method on both large-scale image classification (e.g., ImageNet) and object detection tasks. tational acceleration, many solutions have been proposed, such as network sparse and pruning 23, low-rank approximation, architecture design BID2 10, 8, BID1 16], 24 model quantization BID0 13, 3, 9, 18, 14], and so on. constrain their weights to {−1, +1} 25 or {−1, 0, 1} and achieve limited acceleration by using simple accumulation instead of complicated 26 multiplication-accumulations. In particular, [2, 18, 4, BID10 24] quantize activation values and weights BID10 to bits and use bitwise logic operations to achieve extreme acceleration ratio in inference process but 28 they are suffering from significant performance degradation. However, most models are proposed 29 for fixed precision, and can not extend to other precision models. They easily fall into local optimal 30 solutions and face slow convergence speed in training process. In order to bridge the gap between 31 low-bit and full-precision and be applied to many cases, we propose a novel encoding scheme of 32 using {−1, +1} to easily decompose trained QNNs into multi-branch binary networks. Therefore, 33 the inference process can be efficiently implemented by bitwise operations to achieve model com-34 pression, computational acceleration and resource saving. As the basic computation in most neural network layers, matrix multiplication costs lots of resources 37 and also is the most time consuming operation. Modern computers store and process data in bina-38 ry format, thus non-negative integers can be directly encoded by {0, 1}. We propose a novel de- DISPLAYFORM0 All of the above operations consist of N multiplications and (N −1) additions. Based on the above 43 encoding scheme, the vector x can be encoded to binary form using M bits, i.e., DISPLAYFORM1 Then we convert the right-hand side of into the following form: DISPLAYFORM2 where DISPLAYFORM3. In such an encoding scheme, the number of represented states is not greater than 2 M. In addition, we encode another vector w with K-bit numbers in the same way. Therefore, the dot product of the 48 two vectors can be computed as follows: DISPLAYFORM0 From the above formulas, the dot product is decomposed into M ×K sub-operations, in which each 50 element is 0 or 1. Because of the restriction of encoding and without using the sign bit, the above 51 representation can only be used to encode non-negative integers. However, it's impossible to limit 52 the weights and the values of the activation functions to non-negative integers. In order to encode 53 both positive and negative integers, we propose a novel encoding scheme, which uses {-1, +1} as 54 the basic elements rather than {0, 1}. Then we can use multiple bitwise operations (i.e., xnor and 55 bitcount) to effectively achieve the above vector multiplications. Our operation mechanism can be 56 suitable for all vector/matrix multiplications. Besides fully connected layers, our mechanism is also 57 suitable for convolution and deconvolution layers in deep neural networks. ing of input data. Therefore, the dot product can be computed by the formula. Without other 63 judgment and mapping calculation, we use trigonometric functions as the basic encoding functions. In the end, we use the sign function to hard divide to -1 or +1. The mathematical expression can be 65 formulated as follows: In this section, we use the same network architecture described in for CIFAR-10 and choose DISPLAYFORM0 ResNet-18 as the basic network for ImageNet. It is very hard to train on large-scale training sets (e.g., ImageNet), and thus parameter initialization is particularly important. In particular, the well-trained 72 full-precision model parameters activated by ReLU can be directly used as initialization parameters 73 for our 8-bit quantized network. After fine-tuning dozens of epochs, 8-bit quantized networks can be 74 well-trained. Similarly, we use the 8-bit model parameters as the initialization parameters to train 7-75 bit quantized networks, and so on. We use the loss computed by quantized parameters to update full 76 precision parameters described as the straight-through estimator. TAB2

Represent the TLDR for finding the corresponding scientific article

A novel encoding scheme of using {-1, +1} to decompose QNNs into multi-branch binary networks, in which we used bitwise operations (xnor and bitcount) to achieve model compression, computational acceleration and resource saving.

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose a novel method for incorporating conditional information into a generative adversarial network (GAN) for structured prediction tasks. This method is based on fusing features from the generated and conditional information in feature space and allows the discriminator to better capture higher-order statistics from the data. This method also increases the strength of the signals passed through the network where the real or generated data and the conditional data agree. The proposed method is conceptually simpler than the joint convolutional neural network - conditional Markov random field (CNN-CRF) models and enforces higher-order consistency without being limited to a very specific class of high-order potentials. Experimental demonstrate that this method leads to improvement on a variety of different structured prediction tasks including image synthesis, semantic segmentation, and depth estimation. Convolutional neural networks (CNNs) have demonstrated groundbreaking on a variety of different learning tasks. However, on tasks where high dimensional structure in the data needs to be preserved, per-pixel regression losses typically in unstructured outputs since they do not take into consideration non-local dependencies in the data. Structured prediction frameworks such as graphical models and joint CNN-graphical model-based architectures e.g. CNN-CRFs have been used for imposing spatial contiguity using non-local information BID13 BID2 BID24. The motivation to use CNN-CRF models stems from their ability to capture some structured information from second order statistics using the pairwise part. However, statistical interactions beyond the second-order are tedious to incorporate and render the models complicated BID0 BID12 ). Other approaches have used task-specific perceptual losses to solve this problem BID10.Generative models provide another way to represent the structure and spacial contiguity in large high-dimensional datasets with complex dependencies. Implicit generative models specify a stochastic procedure to produce outputs from a probability distribution. Such models are appealing because they do not demand parametrization of the probability distribution they are trying to model. Recently, there has been great interest in CNN-based implicit generative models using autoregressive BID4 and adversarial training frameworks BID16.Generative adversarial networks (GANs) BID6 can be seen as a two player minimax game where the first player, the generator, is tasked with transforming a random input to a specific distribution such that the second player, the discriminator, can not distinguish between the true and synthesized distributions. The most distinct feature of adversarial networks is the discriminator that assesses the discrepancy between the current and target distributions. The discriminator acts as a progressively precise critic of an increasingly accurate generator. Despite their structured prediction capabilities, such a training paradigm is often unstable and can suffer from mode collapse. However, recent work on spectral normalization (SN) and gradient penalty has significantly increased training stability BID7. Conditional GANs (cGANs) BID19 incorporate conditional image information in the discriminator and have been widely used for class conditioned image generation. To that effect, unlike in standard GANs, a discriminator for cGANs discriminates between DISPLAYFORM0 Adversarial loss (a) Concatenated Image Conditioning x y Adversarial loss DISPLAYFORM1 Discriminator models for image conditioning. We propose fusing the features of the input and the ground truth or generated image rather than concatenating.the generated distribution and the target distribution on pairs of samples y and conditional information x. For class conditioning, several unique strategies have been presented to incorporate class information in the discriminator BID23 BID22. However, a cGAN can also be conditioned by structured data such as an image. Such conditioning is much more useful for structured prediction problems. Since the discriminator in image conditioned-GANs has access to large portions of the image the adversarial loss can be interpreted as a learned loss that incorporates higher order statistics, essentially eliminating the need to manually design higher order loss functions. Consequently, this variation of cGANs has extensively been used for image-to-image translation tasks. However, the best way of incorporating image conditional information into a GAN is not always clear and methods of feeding generated and conditional images to the discriminator tend to use a naive concatenation approach. In this work we address this gap by proposing a discriminator architecture specifically designed for image conditioning. Such a discriminator can contribute to the promise of generalization GANs bring to structured prediction problems whereby a singular and simplistic setup can be used for capturing higher order non-local structural information from higher dimensional data without complicated modeling of energy functions. Contributions. We propose an approach to incorporating conditional information into a cGAN using a fusion architecture (Fig. 1b). In particular, we make the following key contributions:1. We propose a novel discriminator architecture optimized for incorporating conditional information in cGANs for structured prediction tasks. The method is designed to incorporate conditional information in feature space and thereby allows the discriminator to enforce higher-order consistency in the model. At the same time, this method is conceptually simpler than alternative structured prediction methods such as CNN-CRFs where higher-order potentials have to be manually incorporated in the loss function.2. We demonstrate the effectiveness of this method on a variety of structured prediction tasks including semantic segmentation, depth estimation, and generating real images from semantic masks. Our empirical study demonstrates that using a fusion discriminator is more effective in preserving high-order statistics and structural information in the data.2 RELATED WORK 2.1 CNN-CRF MODELS Models for structured prediction have been extensively studied in computer vision. In the past these models often entailed the construction of hand-engineered features. In 2015, BID15 demonstrated that a fully convolutional approach to semantic segmentation could yield state-ofthe-art at that time with no need for hand-engineering features. BID1 showed that post-processing the of a CNN with a conditional Markov random field led to significant improvements. Subsequent work by many authors have refined this approach by incorporating the CRF as a layer within a deep network and thereby enabling the parameters of both models to be learnt simultaneously BID11. Many researchers have used this approach for other structured prediction problems, including image-to-image translation and depth estimation BID14.In most cases CNN-CRF models only incorporate unary and pairwise potentials. Recent work by BID0 has investigated incorporating higher-order potentials into CNN-based models for semantic segmentation, and has found that while it is possible to learn the parameters of these potentials, they can be tedious to incorporate and render the model quite complex. There is a need for developing methods that can incorporate higher-order statistical information with out manual modeling of higher order potentials. Adversarial Training. Generative adversarial networks were introduced in BID6. A GAN consists of a pair of models (G, D), where G attempts to model the distribution of the source domain and D attempts to evaluate the divergence between the generative distribution q and the true distribution p. GANs are trained by training the discriminator and the generator in turn, iteratively refining both the quality of the generated data and the discriminator's ability to distinguish between p and q. The is that D and G compete to reach a Nash equilibrium that can be expressed by the training procedure. While GAN training is often unstable and prone to issues such as mode collapse, recent developments such as spectral normalization and gradient penalty have increased GAN training stability BID7. Furthermore, GANs have the advantage of being able to access the joint configuration of many variables, thus enabling a GAN to enforce higher-order consistency that is difficult to enforce via other methods BID16.Conditional GANs. A conditional GAN (cGAN) is a GAN designed to incorporate conditional information BID19. cGANs have shown promise for several tasks such as class conditional image synthesis and image-to-image translation BID19. There are several advantages to using the cGAN model for structured prediction, including the simplicity of the framework. Image conditioned cGANs can be seen as a structured prediction problem tasked with learning a new representation given an input image while making use of non-local dependencies. However, the method by which the conditional information should be incorporated into the model is often unmotivated. Usually, the conditional data is concatenated to some layers in the discriminator (often the input layers). A notable exception to this methodology is the projection cGAN, where for data is either known or assumed to follow certain simple distributions and a hard mathematical rule for incorporating conditional data can be derived from the underlying probabilistic graphical model. As mentioned in, the method is more likely to produce good if the data follows one of the prescribed distributions. For structured prediction tasks where the GAN framework has to be conditioned by an image, this is often not the case. In the following section we introduce the fusion discriminator and explain the motivation behind it. As mentioned, the most significant part of cGANs for structured prediction is the discriminator. The discriminator has continuous access to pairs of the generated data or real data y and the conditional information (i.e. the image) x. The cGAN discriminator can then be defined as, DISPLAYFORM0, where A is the activation function, and f is a function of x and y and θ represents the parameters of f. Let p and q designate the true and the generated distributions. The adversarial loss for the discriminator can then be defined as DISPLAYFORM1 Here, A represents the sigmoid function, D represents the conditional discriminator, and G represents the generator. By design, this frameworks allows the discriminator to significantly effect the generator BID6. The most common approach currently in use to incorporate conditional image information into a GAN is to concatenate the conditional image information to the input of the discriminator at some layer, often the first. Other approaches for conditional information fusion are limited to class conditional fusion where conditional information is often a one-hot vector rather than higher dimensional structured data. Since the discriminator classifies pairs of input and output images, concatenating high-dimensional data may not exploit some inherent dependencies in the structure of the data. Fusing the input and output information in an intuitive way such as to preserve the dependencies is instrumental in designing an adversarial framework with high structural capacity. We propose the use of a fusion discriminator architecture with two branches. The branches of this discriminator are convolutional neural networks, say φ(x) and ψ(y), that extract features from both the conditional data and the generated or real data respectively. The features extracted from the conditional data are then fused with the features from the real or generated data at various stages FIG0. The proposed discriminator architecture is similar to the encoder-part of the FuseNet architecture, which has been used to incorporate depth information from RGB-D images for semantic segmentation BID8. In FIG0, we illustrate a four layer and a VGG16-style fusion discriminator, in which both branches are similar in depth and structure to the VGG16 model BID26. The key ingredient of the fusion discriminator architecture is the fusion block, which combines the feature maps of x and y. The fusion layer (red, FIG0) is implemented as element-wise summation and is always inserted after a convolution → spectral normalization → ReLU instance. By making use of this fusion layer the discontinuities in the features maps of x and y are added into the y branch in order to enhance the overall feature maps. This preserves representation from both x and y. For structured prediction tasks x and y often have features that complement each other; for instance, in tasks like depth estimation, semantic segmentation, and image synthesis x and y all have complimentary features. Theoretical Motivation. When the data is passed through two networks with identical architectures and the activations at corresponding layers are added, the effect in general is to pass forward through the combined network (the upper branch in FIG0) a stronger signal than would be passed forward by applying an activation to concatenated data. To see this, suppose the k th feature map in the l th layer is denoted by hk. Let the weights and biases for this feature and layer be denoted W DISPLAYFORM0 T and b DISPLAYFORM1 Further, let h = x T y T T, where x and y represent the learned features from the conditional and real or generated data respectively. Assuming a ReLU activation function, DISPLAYFORM2 Based on the inequality in Eq. 4 we demonstrate that the fusion of the activations in ψ(x) and φ(y) produces a stronger signal than the activation on concatenated inputs. Indeed, strengthening some of the activations is not by any means a guarantee of improved performance in general. However, using the fusion discriminator not only increases the neuron-wise activation values but also preserves activations at different neuron locations. In the context of conditional GANs the fusing operation in the strongest signals being passed through the discriminator specifically at those places where the model finds useful information simultaneously in both the conditional data and the real or generated data. In addition, the model preserves a signal, albeit a weaker one, when either x or y contain useful information both for the learning of higher-level features and for the discriminator's eventual classification of the data. Empirical Motivation. We use gradient-weighted Class Activation Mapping (Grad-CAM) BID25 which uses the class-specific gradient information going into the final convolutional layer of a trained CNN to produce a coarse localization map of the important regions in the image. We visualized the outputs of a fusion and concatenated discriminator for several different tasks to observe the structure and strength of the signal being passed forward. We observed that the fusion discriminator architecture always had a visually strong signal at important features for the given task. Representative images from classifying x and y pairs as'real' for two different structured prediction tasks are shown in FIG1. This provides visual evidence that a fusion discriminator preserves more structural information from the input and output image pairs and classifies overlapping patches based on that information. Indeed, this is not evidence that a stronger signal will lead to a more accurate classification, but it is a heuristic justification that more representative features from x and y will be used to make the determination. In order to evaluate the effectiveness of the proposed fusion discriminator we conducted three sets of experiments on structured prediction problems: 1) generating real images from semantic masks (Cityscapes); 2) semantic segmentation (Cityscapes); 3) depth estimation (NYU v2). For all three tasks we used a U-Net based generator. We applied spectral normalization to all weights of the In order to demonstrate the structure preserving abilities of our discriminator we use the proposed setup in the image-to-image translation setting. We focus on the application of generating realistic images from semantic labels. This application has recently been studied for generating realistic synthetic data for self driving cars BID28 BID3. Unlike recent approaches where the objective is to generate increasingly realistic high definition (HD) images, the purpose of this experiment is to explore if a generic fusion discriminator can outperform a concatenated discriminator when using a simple generator. We used 2,975 training images from the Cityscapes dataset BID5 and re-scaled them to 256 × 256 for computational efficiency. The provided Cityscapes test set with 500 images was used for testing. Our ablation study focused on changing the discriminator between a standard 4-layer concatenation discriminator used in seminal image-to-image translation work, a combination of this 4-layer discriminator with spectral normalization (SN), a VGG-16 concatenation discriminator and the proposed 4-layer and VGG-16 fusion discriminators. Since standard GAN evaluation metrics such as inception score and FID can not directly be applied to image-to-image translation tasks we use an evaluation technique previously used for such image synthesis. To quantitatively evaluate and comparatively analyze the effectiveness of our proposed discriminator architecture we perform semantic segmentation on synthesized images and compare the similarity between the predicted segments and the input. The intuition behind this kind of experimentation is that if the generated images corresponds to the input label map an existing semantic segmentation model such as a PSPNet BID29 should be able to predict the input segmentation mask. Similar experimentation has been suggested in and. Table 1 reports segmentation both pixel-wise accuracy and overall intersection-over-union (IoU), the proposed fusion discriminator outperforms the concatenated discriminator by a large margin. Our is closer to the theoretical upper bound achieved by real images. This confirms that the fusion discriminator contributes to preserving more Figure 5: A comparative analysis of concatenation, projection and fusion discriminators on three different structured prediction tasks, i.e., image synthesis, semantic segmentation, and depth estimation. Table 1: PSPNet-based semantic segmentation IoU and accuracy scores using generated images from different discriminators. Our outperform concatenation-based methods by a large margin and is close to the accuracy and IoU on actual images (GT/Oracle). Mean IoU Pixel Accuracy 4-Layer Concat. structure in the output image. The fusion discriminator could be used with high definition images, however, such analysis is beyond the scope of the current study. Representative images for this task are shown in FIG2. Fig. 5 shows a comparative analysis of the concatenation, projection and fusion discriminators in an ablation study upto 550k iterations. The projection discriminator was modified image conditioning according to the explanation given in for the super-resolution task. Semantic segmentation is vital for visual scene understanding and is often formulated as a dense labeling problem where the objective is to predict the category label for each individual pixel. Semantic segmentation is a classical structured prediction problem and CNNs with pixel-wise loss often fail to make accurate predictions BID16. Much better have been achieved by incorporating higher order statistics in the image using CRFs as a post-processing step or jointly training them with CNNs BID2. It has been shown that incorporating higher order potentials continues to improve semantic segmentation improvement, making this an ideal task for evaluating the structured prediction capabilities of GANs and their enhancement using our proposed discriminator. Here, we empirically validate that the adversarial framework with the fusion discriminator can preserve more spacial context in comparison to CNN-CRF setups. We demonstrate that our proposed fusion discriminator is equipped with the ability to preserve higher order details. For comparative analysis we compare with relatively shallow and deep architectures for both concatenation and fusion discriminators. We also conduct an ablation study to analyze the effect of spectral normalization. The generator for all semantic segmentation experiments was a U-Net. For the experiment without spectral normalization, we trained each model for 950k iterations, which was sufficient for Depth estimation is another structured prediction task that has been extensively studied because of its wide spread applications in computer vision. As with semantic segmentation, both per-pixel losses and non-local losses such as CNN-CRFs have been widely used for depth estimation. Stateof-the art with depth estimation has been achieved using a hierarchical chain of non-local losses. We argue that it is possible to incorporate higher order information using a simple adversarial loss with a fusion discriminator. In order to validate our claims we conducted a series of experiments with different discriminators, similar to the series of experiments conducted for semantic segmentation. We used the Eigen testtrain split for the NYU v2? dataset containing 1449 images for training and 464 images for testing. We observed that as with image synthesis and semantic segmentation the fusion discriminator outperforms concatenation-based methods and pairwise CNN-CRF methods every time. Structured prediction problems can be posed as image conditioned GAN problems. The discriminator plays a crucial role in incorporating non-local information in adversarial training setups for structured prediction problems. Image conditioned GANs usually feed concatenated input and output pairs to the discriminator. In this research, we proposed a model for the discriminator of cGANs that involves fusing features from both the input and the output image in feature space. This method provides the discriminator a hierarchy of features at different scales from the conditional data, and thereby allows the discriminator to capture higher-order statistics from the data. We qualitatively demonstrate and empirically validate that this simple modification can significantly improve the general adversarial framework for structured prediction tasks. The presented in this paper strongly suggest that the mechanism of feeding paired information into the discriminator in image conditioned GAN problems is of paramount importance. The generator G tries to minimize the loss expressed by equation 5 while the discriminator D tries to maximize it. In addition, we impose an L1 reconstruction loss: DISPLAYFORM0 leading to the objective, DISPLAYFORM1 6.2 GENERATOR ARCHITECTURE We adapt our network architectures from those explained in. Let CSRk denote a Convolution-Spectral Norm -ReLU layer with k filters. Let CSRDk donate a similar layer with dropout with a rate of 0.5. All convolutions chosen are 4 × 4 spatial filters applied with a stride 2, and in decoders they are up-sampled by 2. All networks were trained from scratch and weights were initialized from a Gaussian distribution of mean 0 and standard deviation of 0.02. All images were cropped and rescaled to 256 × 256, were up sampled to 268 × 286 and then randomly cropped back to 256 × 256 to incorporate random jitter in the model. Decoder: CSRD512→CSRD1024→CSRD1024→CSR1024→CSR1024→CSR512→CSR256→CSR128The last layer in the decoder is followed by a convolution to map the number of output channels (3 in the case of image synthesis and semantic labels and 1 in the case of depth estimation). This is followed by a Tanh function. Leaky ReLUs were used throughout the encoder with a slope of 0.2, regular ReLUs were used in the decoder. Skip connections are placed between each layer l in the encoder and layer ln in the decoder assuming l is the maximum number of layers. The skip connections concatenate activations from the l th layer to layer (l − n) th later.

We propose a novel way to incorporate conditional image information into the discriminator of GANs using feature fusion that can be used for structured prediction tasks.

This paper proposes a new approach to incorporating desired invariance to representations learning, based on the observations that the current state-of-the-art AFL has practical issues.