Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Recently various neural networks have been proposed for irregularly structured data such as graphs and manifolds. To our knowledge, all existing graph networks have discrete depth. Inspired by neural ordinary differential equation (NODE) for data in the Euclidean domain, we extend the idea of continuous-depth models to graph data, and propose graph ordinary differential equation (GODE). The derivative of hidden node states are parameterized with a graph neural network, and the output states are the solution to this ordinary differential equation. We demonstrate two end-to-end methods for efficient training of GODE: indirect back-propagation with the adjoint method; direct back-propagation through the ODE solver, which accurately computes the gradient. We demonstrate that direct backprop outperforms the adjoint method in experiments. We then introduce a family of bijective blocks, which enables $\mathcal{O}$ memory consumption. We demonstrate that GODE can be easily adapted to different existing graph neural networks and improve accuracy. We validate the performance of GODE in both semi-supervised node classification tasks and graph classification tasks. Our GODE model achieves a continuous model in time, memory efficiency, accurate gradient estimation, and generalizability with different graph networks. Convolutional neural networks (CNN) have achieved great success in various tasks, such as image classification and segmentation , video processing and machine translation . However, CNNs are limited to data that can be represented by a grid in the Euclidean domain, such as images (2D grid) and text (1D grid), which hinders their application in irregularly structured datasets. A graph data structure represents objects as nodes and relations between objects as edges. Graphs are widely used to model irregularly structured data, such as social networks , protein interaction networks , citation and knowledge graphs . Early works use traditional methods such as random walk (Lovász et al., 1993), independent component analysis (ICA) (Hyvärinen &) and graph embedding to model graphs, however their performance is inferior due to the low expressive capacity. Recently a new class of models called graph neural networks (GNN) were proposed. Inspired by the success of CNNs, researchers generalize convolution operations to graphs to capture the local information. There are mainly two types of methods to perform convolution on a graph: spectral methods and non-spectral methods. Spectral methods typically first compute the graph Laplacian, then perform filtering in the spectral domain . Other methods aim to approximate the filters without computing the graph Laplacian for faster speed . For non-spectral methods, the convolution operation is directly performed in the graph domain, aggregating information only from the neighbors of a node . The recently proposed GraphSAGE learns a convolution kernel in an inductive manner. To our knowledge, all existing GNN models mentioned above have a structure of discrete layers. The discrete structure makes it hard for the GNN to model continuous diffusion processes in graphs. The recently proposed neural ordinary differential equation (NODE) ) views a neural network as an ordinary differential equation (ODE), whose derivative is parameterized by the network, and the output is the solution to this ODE. We extend NODE from the Euclidean domain to graphs and propose graph ordinary differential equations (GODE), where the message propagation on a graph is modeled as an ODE. NODEs are typically trained with adjoint method. NODEs have the advantages of adaptive evaluation, accuracy-speed control by changing error tolerance, and are free-form continuous invertible models ). However, to our knowledge, in benchmark image classification tasks, NODEs are significantly inferior to state-of-the-art discrete-layer models (error rate: 19% for NODE vs 7% for ResNet18 on CIFAR10) . In this work, we show this is caused by error in gradient estimation during training of NODE, and propose a memory-efficient framework for accurate gradient estimation. We demonstrate our framework for free-form ODEs generalizes to various model structures, and achieves high accuracy for both NODE and GODE in benchmark tasks. Our contribution can be summarized as follows: 1. We propose a framework for free-form NODEs to accurately estimate the gradient, which is fundamental to deep-learning models. Our method significantly improves the performance on benchmark classification (reduces test error from 19% to 5% on CIFAR10). 2. Our framework is memory-efficient for free-form ODEs. When applied to restricted-form invertible blocks, the model achieves constant memory usage. 3. We generalize ODE to graph data and propose GODE models. 4. We demonstrate improved performance on different graph models and various datasets. There have been efforts to view neural networks as differential equations. viewed a residual network as a discretization of a differential equation and proposed several new architectures based on numerical methods in ODE solver. proposed a stable architecture based on analysis of the ODE. proposed neural ordinary differential equation (NODE), which treats the neural network as a continuous ODE. NODE was later used in a continuous normalizing flow for generative models . There have been many studies on the training of NODE. The adjoint method has long been widely used in optimal control and geophysical problems , and recently applied to ODE. proposed augmented neural ODEs to improve the expressive capacity of NODEs. However, to our knowledge, none of the methods above discusses the inaccurate gradient estimation issue; empirical performances of NODE in benchmark classification tasks are significantly inferior to state-of-the-art discrete-layer models. GNNs can be divided into two categories: spectral methods and non-spectral methods. Spectral GNNs perform filtering in the Fourier domain of a graph, thus need information of the whole graph to determine the graph Laplacian. In contrast, non-spectral GNNs only consider message aggregation around neighbor nodes, therefore are localized and generally require less computation . We first briefly introduce several spectral methods. first introduced graph convolution in the Fourier domain based on the graph Laplacian, however the computation burden is heavy because of non-localized filters. incorporated a graph estimation procedure in spectral networks and parameterized spectral filters into a localized version with smooth coefficients. used Chebyshev expansion to approximate the filters without the need to compute the graph Laplacian and its eigenvectors, therefore significantly accelerated the running speed. proposed to use a localized first-order approximation of graph convolution on graph data and achieved superior performance in semi-supervised tasks for node classification. proposed fast localized spectral filtering on graphs. Non-spectral methods typically define convolution operations on a graph, only considering neighbors of a certain node. MoNet uses a mixture of CNNs to generalize convolution to graphs. GraphSAGE samples a fixed size of neighbors for each node for fast localized inference. Graph attention networks (Veličković et al., 2017) learn different weights for different neighbors of a node. The graph isomorphism network (GIN) has a structure as expressive as the Weisfeiler-Lehman graph isomorphism test. Invertible blocks are a family of neural network blocks whose forward function is a bijective mapping. Therefore, the input to a bijective block can be accurately reconstructed from its outputs. Invertible blocks have been used in normalizing flow (; ; ; ;), where the model is required to be invertible in order to calculate the log-density of data distribution. Later on, used bijective blocks to build invertible networks. proposed to use invertible blocks to perform back propagation without storing activation, which achieves a memory-efficient network structure. They were able to discard activation of middle layers, because each layer's activation can be reconstructed from the next layer with invertible blocks. We first consider discrete-layer models with residual connection , which can be represented as: where x k is the states in the kth layer; f k (·) is any differentiable function whose output has the same shape as its input. When we add more layers with shared weights, and let the stepsize in Eq. 1 go to infinitesimal, the difference equation turns into a neural ordinary differential equation (NODE): We use z(t) in the continuous case and x k in the discrete case to represent hidden states. f (·) is the derivative parameterized by a network. Note that a key difference between Eq. 1 and 2 is the form of f: in the discrete case, different layers (different k values) have their own function f k; while in the continuous case, f is shared across all time t. The forward pass of model with discrete layers can be written as: where K is the total number of layers. Then an output layer (e.g. fully-connected layer for classification) is applied on x K. The forward pass of a NODE is: where z = input and T is the integration time, corresponding to number of layers K in the discrete case. The transformation of states z is modeled as the solution to the NODE. Then an output layer is applied on z(T). Integration in the forward pass can be performed with any ODE solver, such as the Euler Method, Runge-Kutta Method, VODE solver and Dopris Solver (; ;). The adjoint method is widely used in optimal process control and functional analysis . We follow the method by. Denote model parameters as θ, which is independent of time. Define the adjoint as: Figure 1: Comparison of two methods for back-propagation on NODE. As in figure (a), the ODE solver is discretized at points {t0, t1, ..., tN} during forward pass. Black dashed curve shows hidden state solved in forward-time, denoted as z(t). Figure (b) shows the adjoint method, red solid line shows the hidden state solved in reverse-time, denoted as h(t). Ideally z(t) = h(t) and dashed curve overlaps with solid curve; however, the reverse-time solution could be numerically unstable, and causes z(t) = h(t), thus causes error in gradient. Figure (c) shows the direct back-propagation through ODE solver. In direct back-propagation, we save evaluation time points {t0, t1, ...tN} during forward pass; during backward pass, we re-build the computation graph by directly evaluating at the same time points. In this way, z(ti) = h(ti). Since the hidden state can be accurately reconstructed, the gradient can be accurately evaluated. where L is the loss function. Then we have with detailed proof from optimization perspective in appendix F. Then we can perform gradient descent to optimize θ to minimize L. Eq. 6 is a reverse-time integration, which can be solved with any ODE solver. To evaluate, we need to determine z(t) by solving Eq. 2 reverse-time (Directly storing z(t) during forward pass requires a large memory consumption, because the continuous model is equivalent to an infinite-layer model). To summarize, in the forward pass we solve Eq. 2 forward in time; in the backward pass, we solve Eq. 2 and 6 reverse in time, with initial condition determined from Eq. 5 at time T. We give an intuition why the reverse-time ODE solver causes inaccurate gradient in adjoint methods. The backward pass (Eq. 6) requires determining f (z(t), t, θ) and, which requires determining z(t) by solving Eq. 2 reverse-time. As shown in Fig. 1 (a,b), the hidden state solved forward-time (z(t i)) and the hidden state solved reverse-time (h(t i)) may not be equal; this could be caused by the instability of reverse-time ODE, and is represented by the mismatch between z(t) (dashed curve) and h(t) (solid curve). Error h(t) − z(t) will cause error in gradient dL dθ. Proposition 1 For an ODE in the form, denote the Jacobian of f as J f. If this ODE is stable both in forward-time and reverse-time, then Re(λ i (J f)) = 0 ∀i, where λ i (J f) is the ith eigenvalue of J f, and Re(λ) is the real part of λ. Detailed proof is in appendix C. Proposition 1 indicates that if the Jacobian of the original system Eq. 2 has eigenvalues whose real-part are not 0, then either the reverse-time or forward-time ODE is unstable. When |Re(λ)| is large, either forward-time or reverse-time ODE is sensitive to numerical errors. This phenomenon is also addressed in. This instability affects the accuracy of solution to Eq. 2 and 6, thus affects the accuracy of the computed gradient. The adjoint method might be sensitive to numerical errors when solving the ODE in reverse-time. To resolve this, we propose to directly back-propagate through the ODE solver. As in Fig. 1(a), the ODE solver uses discretization for numerical integration, evaluated at time points {t 0, t 1, ...t N}. Fig. 1(c) demonstrates the direct back-propagation with accurate hidden states h(t i), which can be achieved with two methods: the activation z(t i) can be saved in cache for back-prop, but requires huge memory; or we can accurately reconstruct z(t i) by re-building the computation graph directly at evaluated time points {t i}. Since the model is evaluated at the same time points t i in forward-time, it's guaranteed that z(t i) = h(t i). Therefore direct back-prop is accurate, regardless of the stability of Eq. 2. Similar to the continuous case, we can define the adjoint with discrete time. Then we have: where a i is the adjoint for the ith step in discrete forward-time ODE solution. Eq. 7 can be viewed as a numerical discretization of Eq. 6. We show Eq. 6 can be derived from an optimization perspective. Detailed derivations of Eq. 6-7 are in appendix E and F. Algorithm 1: Algorithm for accurate gradient estimation in ODE solver for free-form functions Define model, where f is a free-form function. Denote integration time as T. Forward (f, T, z 0, tolerance): Select initial step size h = h 0 (adaptively with adaptive step-size solver). time points = empty list While t < T: state = f.state dict, accept step = F alse While Not accept step: f.load state dict(state) with grad disabled: z new, error estimate = step(f, z, t, h) If error estimate < tolerance: accept step = T rue z = z new, t = t + h, time points.append(t) else: reduce stepsize h according to error estimate delete z new, error estimate and related local computation graph cache.save(time points) return z, cache Backward (f, T, z 0, tolerance, cache): Details of our method are summarized in Algorithm 1. We discuss its properties below: Summary of the algorithm During forward pass, the solver performs a numerical integration, with the stepsize adaptively varying with error estimation. During forward pass, the solver outputs the integrated value, and the evaluation time points {t i}. All middle activations are deleted to save memory. During backward pass, the solver re-builds the computation graph, by directly evaluating at saved time points, without adaptive searching. During backward pass, the solver performs a numerical version (Eq. 7) of reverse-time integration (Eq. 6). Support for free-form continuous dynamics There's no constraint on the form of f. Therefore, our algorithm is a generic method. Memory consumption analysis Suppose f has N f layers, the number of forward evaluation step is N t on average, and the evaluations to adaptively search for an optimal stepsize is K. A naive solver will take O(N f × N t × K), while our method consumes O(N f × N t) because all middle activations are deleted during forward pass, and we don't need to search for optimal stepsize in backward pass. If we perform step-wise checkpoint method, where we only store z(t i) for all t i, and compute the gradient ∂z(ti+1) ∂z(ti) for one t i at a time, then the memory consumption can be reduced to O(N f +N t). Since the solver can handle free-form functions, it can also handle restricted form invertible block (see below). In this case, we don't need to store z(t i), and the memory consumption can reduce to O(N f). More memory-efficient solver with invertible blocks Restricting the form of f to invertible blocks allows for O(N f) memory consumption. For invertible blocks, input x is split into two parts (x 1, x 2) of the same size (e.g. x has shape N × C, where N is batch size, C is channel number; we can split x into x 1 and x 2 with shape N × C 2). The forward and inverse of a bijective block can be denoted as: where the output of a bijective block is denoted as (y 1, y 2) with the same size as (x 1, x 2). F and G are any differentiable neural networks, whose output has the same shape as the input. ψ(α, β) is a differentiable bijective function w.r.t α when β is given; ψ −1 (α, β) is the inverse function of ψ. Theorem 1 If ψ(α, β) is a bijective function w.r.t α when β is given, then the block defined by Eq. 8 is a bijective mapping. Proof of Theorem 1 is given in appendix D. Based on this, we can apply different ψ functions for different tasks. Since x can be accurately reconstructed from y, there's no need to store activations, hence is memory-efficient. Details for back-prop without storing activation are in appendix B. We first introduce graph neural networks with discrete layers, then extend to the continuous case and introduce graph ordinary differential equations (GODE). As shown in Fig. 2, a graph is represented with nodes (marked with circles) and edges (solid lines). We assign a unique color to each node for ease of visualization. Current GNNs can generally be represented in a message passing scheme : where x u k represents states of the uth node in the graph at kth layer and e u,v represents the edge between nodes u and v. N (u) represents the set of neighbor nodes for node u. ζ represents a differentiable, permutation invariant operation such as mean, max or sum. γ (k) and φ (k) are differentiable functions parameterized by neural networks. For a specific node u, a GNN can be viewed as a 3-stage model, corresponding to Eq. 9-11: Message passing, where neighbor nodes v ∈ N (u) send information to node u, denoted by message (v,u). The message is generated from function φ(·), parameterized by a neural network. Message aggregation, where a node u aggregates all messages from its neighbors N (u), denoted as aggregation u. The aggregation function ζ is typically permutation invariant operations such as mean and sum, because graphs are invariant to permutation. Update, where the states of a node are updated according to its original states x u k−1 and aggregation of messages aggregation u, denoted as γ(·). We can convert a discrete-time GNN to continuous-time GNN by replacing f in Eq. 2 with the message passing process defined in Eq. 9 to 11, which we call graph ordinary differential equation (GODE). A diagram of GODE is shown in Fig. 2. Because GODE is an ODE in nature, it can capture highly non-linear functions, thus has the potential to outperform its discrete-layer counterparts. We demonstrate that the asymptotic stability of GODE could be related to the over-smoothing phenomena . It's demonstrated that graph convolution is a special case of Laplacian smoothing , which can be written as Y = (I − γD −1/2LD−1/2)X where X and Y are the input and output of a graph-conv layer respectively,Ã = A + I where A is the adjacency matrix, andD is the corresponding degree matrix ofÃ, and γ is a positive scaling constant. When modified from a discrete model to a continuous model, the continuous smoothing process is: Since all eigenvalues of the symmetrically normalized Laplacian are real and non-negative, then all eigenvalues of the above ODE are real and non-positive. Suppose all eigenvalues of the normalized Laplacian are non-zero. In this case, the ODE has only negative eigenvalues, hence the ODE above is asymptotically stable . Hence as time t grows sufficiently large, all trajectories are close enough. In the experiments, this suggests if integration time T is large enough, all nodes (from different classes) will have very similar features, thus the classification accuracy will drop. To evaluate our method on general NODE, we conducted experiments with a CNN-NODE on two benchmark image classification tasks (CIFAR10 and CIFAR100) . We also evaluated our method on benchmark graph datasets, including 2 bioinformatic graph classification datasets (MUTAG and PROTEINS), 2 social network graph classification datasets (IMDB-BINRAY, REDDIT-BINARY) , and 3 citation networks (Cora, CiteSeer and PubMed). For graph classification tasks, different from the experiment settings in , we input the raw dataset into our models without pre-processing. For node classification tasks, we performed transductive inference and strictly followed the train-validation-test split by , where less than 6% nodes are used as training examples. Details of datasets are summarized in appendix A. For image classification tasks, we directly modify a ResNet18 into its corresponding NODE model. For each block, the function is where f is a sequence of conv − bn − relu − conv − bn − relu layers. f is the same as residual branch in ResNet, and it can be replaced with any free-form functions. For tasks on graph datasets, GODE can be applied to any graph neural network by simply replacing f in Eq. 2 with corresponding structures (free-form functions), or replacing F, G in Eq. 8 with other structures (invertible blocks). To demonstrate that GODE is easily generalized to existing structures, we used several different GNN architectures, including the graph convolutional network (GCN) , graph attention network (GAT) (Veličković et al., 2017), graph network approximated with Chebyshev expansion (ChebNet) , and graph isomorphism network (GIN) . For a fair comparison, we trained GNNs with different depths of layers (1-3 middle layers, besides an initial layer to transform data into specified channels, and a final layer to generate prediction), and reported the best among all depths for each model structure. On the same task, different models use the same hyper-parameters on model structures, such as channel number. For graph classification tasks, we set the channel number of hidden layers as 32 for all models; for ChebNet, we set the number of hops as 16. For node classification tasks, we set the channel number as 16 for GCN and ChebNet, and set number of hops as 3 for ChebNet; for GAT, we used 8 heads, and set each head as 8 channels. For every GNN structure, we experimented with different number of hidden layers, and calculated the mean and variance of accuracy of 10 runs. We compared the adjoint method and direct back-propagation on the same network, and demonstrated direct back-prop generates higher accuracy. For CNN-NODE on classification tasks, we directly modify a ResNet18 into NODE18, and report resuls in Table. 1; for graph networks, we train a GODE model with a GCN to parameterize the derivative, and report in Table. 2. Empirical performance Direct back-propagation consistently outperformed the adjoint method for both tasks. This validates our analysis on the instability of the adjoint method, which is intuitively caused by the instability of the reverse-time ODE. On image classification tasks, compared to adjoint method, our training method reduces error rate of NODE18 from 19% (37%) to 5%(23%) on CIFAR10 (CIFAR100). Furthermore, NODE18 has the same number of parameters as ResNet18, but outperforms deeper networks such as ResNet101 on both datasets. Our method also consistently outperforms the adjoint method on several benchmark graph datasets, as shown in Table. 2. Robustness to ODE solvers We implemented adaptive ODE solvers of different orders, as shown in Table 1. HeunEuler, RK23, RK45 are of order 1, 2, 4 respectively, i.e., for each step forward in time f is evaluated 1, 2, 4 times respectively. During inference, using different solvers is equivalent to changing model depth (without re-training the network): for discrete-layer models, it generally causes huge error; for continuous models, we observe only around 1% increase in error rate. This suggests our method is robust to different orders of ODE solvers. Support for free-form functions Our method supports NODE and GODE models with free-form functions; for example, f in NODE18 in Table. 1 is a free-form function. We demonstrate that bijective blocks defined as Eq. 8 can be easily generalized: F and G are general neural networks, which can be adapted to different tasks; ψ(α, β) can be any differentiable bijective mapping w.r.t. α when β is given. We demonstrate two examples of ψ: Results for different ψ are reported in Table 3. Note that we experimented with different depths and reported the best accuracy for each model, and performed a paired t-test on from GODE and their discrete-layer counterparts. Most GODE models outperformed their corresponding discretelayer models significantly, validating the effectiveness of GODE; different ψ functions behaved similarly on our node classification tasks, indicating the continuous-time model is more important than coupling function ψ. We also validate the lower memory cost, with details in appendix B. Results for different models on graph classification tasks are summarized in Table 4. We experimented with different structures, including GCN, ChebNet and GIN; for corresponding GODE models (marked with ODE), we tested both free-form (marked with "free") and invertible block (marked with "INV"). We performed paired t-test comparing GODE and its discrete-layer counterparts. For most experiments, GODE models performed significantly better. This indicates the continuous process model might be important for graph models. For a NODE and GODE model, during inference, we test the influence of integration time. Results are summarized in Table. 5. When integration time is short, the network does not gather sufficient information from neighbors; when integration time is too long, the model is sensitive to over-smooth issue, as discussed in Sec. 4.2. We observe accuracy drop in both cases. We propose GODE, which enables us to model continuous diffusion process on graphs. We propose a memory-efficient direct back-propagation method to accurately determine the gradient for general free-form NODEs, and validate its superior performance on both image classification tasks and graph data. Furthermore, we related the over-smoothing of GNN to asymptotic stability of ODE. Our paper tackles the fundamental problem of gradient estimation for NODE; to our knowledge, it's the first paper to improve accuracy on benchmark tasks to comparable with state-of-the-art discrete layer models. It's an important step to apply NODE from theory to practice. A DATASETS We perform experiments on various datasets, including citation networks (Cora, CiteSeer, PubMed), social networks (COLLAB, IMDB-BINARY, REDDIT-BINARY), and bioinformatics datasets (MUTAG, PROTEINS). Details of each dataset are summarized in Table 1. We explain the structure and conduct experiments for the invertible block here. Structure of invertible blocks Structure of invertible blocks are shown in Fig. 1. We follow the work of with two important modifications: We generalize to a family of bijective blocks with different ψ in Eq. 8 in the main paper, while restrict the form of ψ to be sum. We propose a parameter state checkpoint method, which enables bijective blocks to be called more than once, while still generating accurate inversion. The algorithm is summarized in Algo. 2. We write the pseudo code for forward and backward function as in PyTorch. Note that we use "inversion" to represent reconstructing input from the output, and use "backward" to denote calculation of the gradient. To reduce memory consumption, in the forward function, we only keep the outputs y 1, y 2 and delete all other variables and computation graphs. In the backward function, we first "inverse" the block to calculate x 1, x 2 from y 1, y 2, then perform a local forward and calculate the gradient x1,x2]. In this section we demonstrate that our bijective block is memory efficient. We trained a GODE model with bijective blocks, and compared the memory consumption using our memory-efficient function as in Algo. 2 and a memory-inefficient method as in conventional backpropagation. Results were measured with a batchsize of 100 on MUTAG dataset. Depth Memory-efficient Conventional 10 2.2G 5.3G 20 2.6G 10.5G Table 2: Memory consumption of bijective blocks. "Conventional" represents storing activation of all layers in cache, "memory-efficient" represents our method in Algo. 2. Results are summarized in Table. 2. We measured the memory consumption with different depths, which is the number of ODE blocks. When depth increases from 10 to 20, the memory by conventional methods increases from 5.3G to 10.5G, while our memory-efficient version only increases from 2.2G to 2.6G. In theory, our bijective block takes O memory, because we only need to store the outputs in cache, while deleting activations of middle layers. For memory-efficient network, the slightly increased memory consumption is because states of F, G need to be cached; but this step takes up minimal memory compared to input data. Algorithm 2: Function for memory-efficient bijective blocks delete computation graphs generated by F and G return cache, y1, y2 Backward(cache, y1, y2, F, G, ψ, Proposition 1 For an ODE in the form dz(t) dt = f (z(t), t), denote the Jacobian of f as J f. If this ODE is stable both in forward-time and reverse-time, then Re(λ i (J f)) = 0 ∀i, where λ i (J f) is the ith eigenvalue of J f, and Re(λ) is the real part of λ. Proof Denote s = T − t, where T is the end time. Notice that the reverse-time in t is equivalent to forward-time in s. Therefore, we have forward-time ODE: and reverse-time ODE: Therefore, we have λ(J f) = −λ(J g). For both forward-time and reverse-time ODE to be stable, the eigenvalues of J need to have non-positive real part. Therefore The only solution is D PROOF FOR THEOREM 1 Theorem 1 For bijective block whose forward and reverse mappings are defined as If ψ(α, β) is a bijective function w.r.t α when β is given, then the block is a bijective mapping. Proof To prove the forward mapping is bijective, it is equivalent to prove the mapping is both injective and surjective. Injective We need to prove, if F orward(x 1, x 2) = F orward(x 3, x 4), then x 1 = x 3, x 2 = x 4. The assumption above is equivalent to Since ψ(α, β) is bijective w.r.t α when β is given, from Eq., we have x 1 = x 3. Similarly, condition on x 1 = x 3 and Eq., using bijective property of ψ, we have x 2 = x 4. Therefore, the mapping is injective. Given y 1, y 2, we construct Then for the forward function, given bijective property of ψ, apply F orward and Reverse defined in the proposition statement, z 1 = ψ(x 1, G(y 2)) = ψ ψ −1 y 1, G(y 2), G(y 2) = y 1 Therefore we construct x 1, x 2 s.t. F orward(x 1, x 2) = [y 1, y 2]. Therefore the mapping is surjective. Therefore is bijective. We use a figure to demonstrate the computation graph, and derive the gradient from the computation graph. The loss is L, forward pass is denoted with black arrows, gradient back-propagation is shown with red arrows. We use p to denote each path from θ to L, corresponding to all paths in red that goes from L to θ. In this section we derive the gradient of parameters in an neural-ODE model from an optimization perspective. Then we extend from continuous cases to discrete cases. Notations With the same notations as in the main paper, we use z(t) to denote hidden states z at time t. Denote parameters as θ, and input as x, target as y, and predicted output asŷ. Denote the loss as J(ŷ, y). Denote the integration time as 0 to T. Problem setup The continuous model is defined to follow an ODE: dz(t) dt = f (z(t), t, θ), s.t. z = x We assume f is differentiable, since f is represented by a neural network in our case. The forward pass is defined as:ŷ The loss function is defined as: J(ŷ, y) = J(z(T), y) We formulate the training process as an optimization problem: For simplicity, Eq. 15 only considers one ODE block. In the case of multiple blocks, z(T) is the input to the next ODE block. As long as we can derive dLoss dθ and dLoss dz when dLoss dz(T) is given, the same analysis here can be applied to the case with a chain of ODE blocks. We use the Lagrangian Multiplier Method to solve the problem defined in Eq. 15. For simplicity, only consider one example (can be easily extended to multiple examples cases), the Lagrangian is Karush-Kuhn-Tucker (KKT) conditions are necessary conditions for an solution to be optimal. In the following sections we start from the KKT condition and derive our . Derivative w.r.t. λ At optimal point, we have δL δλ = 0. Note that λ is a function of t, we derive the derivative from calculus of variation. Consider a cotninuous and differentiable perturbation λ(t) on λ(t), and a scalar, L now becomes a function of, It's easy to check the conditions for Leibniz integral rule, and we can switch integral and differentiation, thus: At optimal λ(t), dL d | =0 = 0 for all continuous differentiable λ(t). Therefore, dz(t) dt − f (z(t), t, θ) = 0, ∀t ∈ (0, T) From continuous to discrete case To derive corresponding in discrete cases, we need to replace all integration with finite sum. In discrete cases, the ODE condition turns into: from Eq. 31, we can get: Re-arranging terms we have: which is the discrete version of Eq. 29. Which also corresponds to our analysis in Eq. 10 and 11.

Represent the TLDR for finding the corresponding scientific article

Apply ordinary differential equation model on graph structured data

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose Janossy pooling, a method for learning deep permutation invariant functions designed to exploit relationships within the input sequence and tractable inference strategies such as a stochastic optimization procedure we call piSGD


Question: Represent the scientific article to find a short TLDR

Question: Unsupervised image-to-image translation is a recently proposed task of translating an image to a different style or domain given only unpaired image examples at training time. In this paper, we formulate a new task of unsupervised video-to-video translation, which poses its own unique challenges. Translating video implies learning not only the appearance of objects and scenes but also realistic motion and transitions between consecutive frames. We investigate the performance of per-frame video-to-video translation using existing image-to-image translation networks, and propose a spatio-temporal 3D translator as an alternative solution to this problem. We evaluate our 3D method on multiple synthetic datasets, such as moving colorized digits, as well as the realistic segmentation-to-video GTA dataset and a new CT-to-MRI volumetric images translation dataset. Our show that frame-wise translation produces realistic on a single frame level but underperforms significantly on the scale of the whole video compared to our three-dimensional translation approach, which is better able to learn the complex structure of video and motion and continuity of object appearance. Recent work on unsupervised image-to-image translation BID10 BID12 has shown astonishing on tasks like style transfer, aerial photo to map translation, day-to-night photo translation, unsupervised semantic image segmentation and others. Such methods learn from unpaired examples, avoiding tedious data alignment by humans. In this paper, we propose a new task of unsupervised video-to-video translation, i.e. learning a mapping from one video domain to another while preserving high-level semantic information of the original video using large numbers of unpaired videos from both domains. Many computer vision tasks can be formulated as video-to-video translation, e.g., semantic segmentation, video colorization or quality enhancement, or translating between MRI and CT volumetric data (illustrated in FIG0). Moreover, motion-centered tasks such as action recognition and tracking can greatly benefit from the development of robust unsupervised video-to-video translation methods that can be used out-of-the-box for domain adaptation. Since a video can be viewed as a sequence of images, one natural approach is to use an image-toimage translation method on each frame, e.g., applying a state-of-art method such as CycleGAN, CoGAN BID10 or UNIT BID12. Unfortunately, these methods cannot preserve continuity and consistency of a video when applied frame-wise. For example, colorization of an object may have multiple correct solutions for a single input frame, since some objects such as cars can have different colors. Therefore, there is no guarantee that an object would preserve its color if translation is performed on the frame level frame. In this paper, we propose to translate an entire video as a three-dimensional tensor to preserve its cross-frame consistency and spatio-temporal structure. We employ multiple datasets and metrics to evaluate the performance of our proposed video-to-video translation model. Our synthetic datasets include videos of moving digits of different colors and volumetric images of digits imitating medical scans. We also perform more realistic segmentation-to-RGB and colorization experiments on the GTA dataset BID14, and propose a new MRI-to-CT dataset for medical volumetric image translation, which to our knowledge is the first open medical dataset for unsupervised volumeto-volume translation. We propose the task of unsupervised video-to-video translation. Left: Results of MR-to-CT translation. Right: moving MNIST digits colorization. Rows show per-frame CycleGAN (2D) and our spatio-temporal extension (3D). Since CycleGAN takes into account information only from the current image, it produces reasonable on the image level but fails to preserve the shape and color of an object throughout the video. Best viewed in color. Figure 2: Results of GTA video colorization show that per-frame translation of videos does not preserve constant colours of objects within the whole sequence. We provide more and videos in the supplementary video: https://bit.ly/2R5aGgo. Best viewed in color. Our extensive experiments show that the proposed 3D convolutional model provides more accurate and stable video-to-video translation compared to framewise translation with various settings. We also investigate how the structure of individual batches affects the training of framewise translation models, and find that structure of a batch is very important for stable translation contrary to an established practice of shuffling training data to avoid overfitting in deep models BID3.To summarize, we make the following main contributions: 1) a new unsupervised video-to-video translation task together with both realistic and synthetic proof-of-concept datasets; 2) a spatiotemporal video translation model based on a 3D convnet that outperforms per-frame methods in Figure 3: Our model consists of two generator networks (F and G) that learn to translate input volumetric images from one domain to another, and two discriminator networks (D A and D B) that aim to distinguish between real and fake inputs. Additional cycle consistency property requires that the of translation to the other domain and back is equal to the input video, DISPLAYFORM0 all experiments, according to human and automatic metrics, and 3) an additional analysis of how performance of per-frame methods depends on the structure of training batches. In recent years, there has been increasing interest in unsupervised image-to-image translation. Aside from producing interesting graphics effects, it enables task-independent domain adaptation and unsupervised learning of per-pixel labels. Many recent translation models BID10 BID12 use the adversarial formulation initially proposed by BID2 as an objective for training generative probabilistic models. The main intuition behind an adversarial approach to domain translation is that the learned cross-domain mapping F: X → Y should translate source samples to fake target samples that are indistinguishable from actual target samples, in the sense that no discriminator from a fixed hypothesis space H should be capable of distinguishing them. Many recent advances in domain translation are due to the introduction of the cycle-consistency idea. Models with a cycle consistency loss aim to learn two mappings F (x) and G(y) such that not only are the translated source samples F (x) indistinguishable from the target, and G(y) are indistinguishable from source, but they are also inverses of each other, i.e. F (G(y)) = y, G(F (x)) = x. We also employ this idea and explore how well it generalizes to video-to-video translation. The cycle-consistency constraints might not restrict semantic changes as explored by BID4. There has been work on combining cycle-consistency with variational autoencoders that share latent space for source and target BID11, which ed in more visually pleasing . Both adversarial BID5 and non-adversarial BID1 supervised imageto-image translation models archive much better visual fidelity since samples in source and target datasets are paired, but their use is limited since we rarely have access to such aligned datasets for different domains. Adversarial video generation has also gained much traction over last years with frame-level models based on long short-term memory BID15 as well as spatiotemporal convolutions BID17, especially since adversarial terms seem to better avoid frame blurring BID13 than Euclidian distance. However, none of these works consider learning a conditional generative video model from unpaired data, i.e. cross domain video translation. Here, we propose this problem and a solution based on jointly translating the entire volume of video. We introduce a neural approach for video-to-video translation based on a conditional GAN BID2 ) that treats inputs and outputs as three-dimensional tensors. The network takes a volumetric image (e.g. a video) from domain A and produces a corresponding volume of the same shape in the domain B. The generator module aims to generate realistic volumes, while the discriminator aims to discriminate between the real and generated samples. Similarly to the CycleGAN method, we introduce two generator-discriminator pairs and add a cycle consistency loss ensuring that samples mapped from A to B and back to A are close to the originals. We implement the two generators, F and G, as 3D convolutional networks BID7 ) that follow the architecture described in BID8. The networks consist of three convolutional layers with 3D convolutional filters of shape 3 × 3 × 3, nine resudual blocks and two additional convolutional layers with stride The overall objective of the model consists of the adversarial loss L GAN and the cycle consistency loss L cyc. The adversarial L GAN loss forces both the generator networks to produce realistic videos and the discriminators to distinguish between real and fake samples from the domain in a min-max fashion, whereas L cyc ensures that each sample x ∼ p A translated into domain B and back is equal to the original and vice versa, i.e. G(F (x)) ≈ x (see Fig. 3).The adversarial loss L GAN is a log-likelihood of correct classification between real and synthesized volumes: DISPLAYFORM0 where the generator G is learned in a way that minimizes L GAN, while the discriminator D B aims to maximize it. The cycle consistency loss is the L 1 loss between the input volume and of the reverse translation: DISPLAYFORM1 The total objective can be written as follows: DISPLAYFORM2 Because we employ the cycle loss and the PatchGAN architecture also employed by CycleGAN, we refer to our model as 3D CycleGAN. More generally, we can consider other generator and discriminator implementations within the overall 3D convolutional framework for video-to-video translation. We used CycleGAN trained on randomly selected images (referred to as random CycleGAN) as a baseline method. We also considered two alternative training strategies for training frame-level CycleGAN baselines: CycleGAN trained on consecutive image sequences (sequential CycleGAN) and sequential CycleGAN with additional total variation loss (see Eq. 2) that penalizes radical change in the generated image sequence (const-loss CycleGAN). We compared the performance of these baselines with our approach that operates on three-dimensional inputs (3D CycleGAN).Random CycleGAN. The first strategy for training a CycleGAN is taking as an input 2D images selected randomly from image sequences available for training, which is the standard approach in deep learning. Data shuffling is known to reduce overfitting and speeds up the learning process BID3.Sequential CycleGAN. Since the order of frames is essential in sequential image data, we investigated the case when images are given to a 2D CycleGAN sequentially during the training phase (see Figure 4 : We compare three ways of forming batches used during training of a CycleGAN : (a) random frames from multiple videos, (b) sequential frames from a single video or (c) single 3D tensor consisting of consecutive frames from a single video. Contrary to the conventional wisdom, our experiments suggest that additional randomness in the batch structure induced in case (a) hurts the performance and convergence of the ing translation model. Fig. 4). In contrast to our expectation and the conventional wisdom, sequential CycleGAN often outperformed random CycleGAN in terms of image continuity and frame-wise translation quality. Const-loss CycleGAN. Since CycleGAN performs translation on the frame-level, it is not able to use information from previous frames and, as a , produces some undesired motion artifacts, such as too rapid change of object shape in consecutive frames, change of color or disappearing objects. To alleviate this problem, we tried to force the frame-level model to generate more consistent image sequences by directly adding a total variation penalty term to the loss function, as follows: DISPLAYFORM0 5 EXPERIMENTS Intuitively, translation models that operate on individual frames can not preserve continuity along the time dimension, which may in radical and inconsistent changes in shape, color and texture. In order to show this empirically, we used the GTA segmentation dataset for unsupervised segmentation-to-video translation. Since this task is very challenging even for still images, we created three more datasets that give more insight into pros and cons of different models. MRCT dataset. First, we evaluated the performance of our method on an MR (magnetic resonance) to CT (computed tomography) volumetric image translation task. We collected 225 volumetric MR images from LGG-1p19qDeletion dataset BID0 and 234 CT volumetric images from Head-Neck-PET-CT dataset BID16. Both datasets are licensed with Creative Commons Attribution 3.0 Unported License. Since images from these datasets represent different parts of the body, we chose parts of volume where body regions represented in the images overlap: from superciliary arches to lower jaw. Images of both modalities were manually cropped and resized to 30 × 256 × 256 shape. The final dataset is available for download on the website [TDB].Volumetric MNIST. Volumetric MNIST dataset was created using MNIST handwritten digits database BID9. From each image from MNIST we created a volumetric image imitating 3d scan domain using erosion transformation of two types, we called them "spherical" and "sandglass" domains (see FIG2). The task is to capture and translate the global intensity pattern over time while preserving image content (digit). The ing image volumes of shape 30 × 84 × 84 were used to train the models to transform digits of spherical type to sandglass type and vice versa. Colorization of moving MNIST digits. To test models' ability to preserve local information about the color of object, inspired by the Moving MNIST dataset introduced in BID15, we generated a dataset with moving digits of different colors. We used this dataset to train the models to translate from the original moving white digits to the same moving digits in color. GTA segmentation dataset. The Playing for Benchmarks dataset BID15 is a large collection of GTA gameplay recordings with a rich set of available annotations, and currently it is one of the default datasets for evaluation of image-to-image translation methods. Using the daylight driving and walking subsets 1 of this dataset we generated 1216 short clips of shape 30 × 192 × 108 and corresponding ground truth segmentation videos. Since this paper is focused on translation of Figure 5: Results of unsupervised GTA video-to-segmentation translation with different models. We observed that frame-level methods diverged significantly more frequently than 3D model (top). No information about ground truth pairs was used during training. Frame-wise translation (2D) produces plausible images, but does not preserve temporal consistency. Forming batches from consecutive frames during training (2D sequence) helps in reducing spatio-temporal artifacts and improves convergence. Additional penalty term on consecutive generated frames (const loss) further reduces motion artifacts at the expense of diversity of generated images. Our proposed 3D convolutional model (3D) produces outputs that are coherent in time, but have fewer details because network has to approximate a higher dimensional mapping (video-to-video instead of frame-to-frame) using same number of learnable weights. with global decay-then-rise and rise-then-decay intensity patterns respectively, during models were presented with pairs of videos containing different digits. Our experiments show that frame-level approaches are not able to learn this spatio-temporal pattern and hence cannot perform correct translation whereas our 3D method performs almost perfectly. Both sequence and sequence+const approaches were able to capture temporal pattern but did not learn shape correspondence.dense image sequences with a lot of inter-frame relations, we used the walking subset of this dataset for evaluation as it has slower camera movements and therefore higher effective frame rate. We performed experiments on all datasets mentioned above with four approaches: random, sequential, sequential+const and 3D CycleGAN. For each dataset we used the same 70% of data for training and 30% for performance evaluation. For datasets that contain ground truth pairs (GTA, volumetric MNIST, colored 3D MNIST) samples from both domains were split into train and test sets independently, i.e. often no corresponding pairs of images were present in the training data. We set the number of parameters to be the same for all models (∼ 90M). We also trained the models with number of learnable parameters proportional to the input size (∼ 200M for 3D model). Visual fidelity of the video generated by larger models did not improve, whereas segmentation quality increased significantly. We also report the performance of a large (∼ 200M) random 2D model for reference. For MRCT and GTA segmentation-to-RGB tasks, we ran human evaluation on amazon mechanical turk since these is no "gold" translation to compare with TAB3. In a series of randomized trials, participants were presented with multiple real samples from each domain and then were asked to choose the more realistic one of the outputs of two different models for same input. We also estimated the probability of choosing a video generated by each model over a real one, but only report these numbers for the MRCT domain pair since for segmentation-to-RGB they were below significance level. To help evaluate significance of differences in probabilities, we report a bootstrap estimate of the standard deviation of reported probabilities. For some domain pairs we actually have definitive ground truth answers. For rgb-to-segmentation translation we evaluated segmentation pixel accuracy and L 2 distance between stochastic matrices of pixel class transitions between frames for generated and ground truth segmentation videos with different label denoising levels TAB4. For volumetric MNIST dataset we also computed L 2 error TAB1 ). Since there is no single correct colorization in the colorization of moving digits task, we evaluated average standard deviation of non- colors within each generated image. Models are expected to colorize digits without changing their shapes, therefore we also evaluate L 2 error between original and translated shapes (FIG2) and ground truth videos (last row). Volumetric MNIST. Our experiments on volumetric MNIST show that standard CycleGAN is not able to capture the global motion patterns in the image sequence (see FIG2). Since videos from domain B have the same frames as videos from domain B but in different order, a model with random batches cannot this temporal pattern and outputs a slightly dimmed input. In contrast, 3D CycleGAN was able to learn the transformation almost perfectly. In contrast, sequential models learned the global phase pattern properly, but were unable to generate correct shapes. DISPLAYFORM0 The experiment on colorization of MNIST videos showed that the "random" model is able to colorize individual frames but cannot preserve the color throughout the whole sequence. The choice of batch selection, however, is important: the sequential and const-loss models learned to preserve the same color throughout the sequence even though they did not have access to previous frames. However, we should mention that all models that succeeded in this task collapsed to colorizing digits with a single (blue-green) color even though the training data had 20 different colors. The GTA segmentation-to-video translation with the 3D model produced smoother and more consistent videos compared to the framewise methods which produced undesirable artifacts such as shadow flickering and rapid deformation or disappearance of objects and road marking. Both sequence methods often moved static objects like road marking with camera. One of the drawbacks of the 3D model is that it does not tolerate rapid changes in the input and hence cannot deal with low frame-rate videos. The additional constraint on total variation ed in better visual fidelity and smoothness, but leaned towards rendering all objects of same semantic class using same texture, which reduces the variability of the outputs and fine details, but at the same time reduces the amount of spatio-temporal artifacts. The qualitative of GTA video colorization confirm that the spatio-temporal model produces more consistent and stable colorization (see FIG0).The experiments on MRI-to-CT translation showed that all per-frame translation methods produce image volumes that do not capture the real anatomy (e.g. shape of the skull, nasal path and eyes vary significantly within the neighboring frames), whereas the proposed 3D method gives a continuous and generally more realistic for both CT-MRI and GTA segmentation-to-video tasks TAB3. The CT-to-MRI task is harder since it requires "hallucinating" a lot of fine details and on this task 3D model outperformed random with a significant margin (bold numbers). On a simpler MRI-to-CT task random and 3D models performed similarly within the limits of statistical error. In contrast to the common practice, the sequential batch approach produced more realistic and continuous compared to the random batch choice. Supposedly this is due to the fact that images within the sequence are more similar than randomly selected images, and hence the magnitude of the sum of gradients might be higher ing in faster convergence. Of course, order of frames within sequential batch does not matter since all gradients are summed up during backward pass, but the similarity between images within a batch is important. We proposed a new computer vision task of unsupervised video-to-video translation as well as datasets, metrics and multiple baselines: multiple approaches to framewise translation using imageto-image CycleGAN and its spatio-temporal extension 3D CycleGAN. The of exhaustive experiments show that per-frame approaches cannot capture the essential properties of videos, such as global motion patterns and shape and texture consistency of translated objects. However, contrary to the previous practice, sequential batch selection helps to reduce motion artifacts.

Represent the short summary for finding the corresponding scientific article (long)

Proposed new task, datasets and baselines; 3D Conv CycleGAN preserves object properties across frames; batch structure in frame-level methods matters.

represent the tldr for finding the corresponding scientific article

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.