Question: represent the scientific article to find a short tldr

Question: Vanilla RNN with ReLU activation have a simple structure that is amenable to systematic dynamical systems analysis and interpretation, but they suffer from the exploding vs. vanishing gradients problem. Recent attempts to retain this simplicity while alleviating the gradient problem are based on proper initialization schemes or orthogonality/unitary constraints on the RNN’s recurrency matrix, which, however, comes with limitations to its expressive power with regards to dynamical systems phenomena like chaos or multi-stability. Here, we instead suggest a regularization scheme that pushes part of the RNN’s latent subspace toward a line attractor configuration that enables long short-term memory and arbitrarily slow time scales. We show that our approach excels on a number of benchmarks like the sequential MNIST or multiplication problems, and enables reconstruction of dynamical systems which harbor widely different time scales. Theories of complex systems in biology and physics are often formulated in terms of sets of stochastic differential or difference equations, i.e. as stochastic dynamical systems (DS). A long-standing desire is to retrieve these generating dynamical equations directly from observed time series data . A variety of machine and deep learning methodologies toward this goal have been introduced in recent years (; ; ; ; ; ;), many of them based on recurrent neural networks (RNN) which can universally approximate any DS (i.e., its flow field) under some mild conditions . However, vanilla RNN as often used in this context are well known for their problems in capturing long-term dependencies and slow time scales in the data . In DS terms, this is generally due to the fact that flexible information maintenance over long periods requires precise fine-tuning of model parameters toward'line attractor' configurations (Fig. 1), a concept first propagated in computational neuroscience for addressing animal performance in parametric working memory tasks (; ;). Line attractors introduce directions of zero-flow into the model's state space that enable long-term maintenance of arbitrary values (Fig. 1). Specially designed RNN architectures equipped with gating mechanisms and (linear) memory cells have been suggested for solving this issue . However, from a DS perspective, simpler models that can more easily be analyzed and interpreted in DS terms, and for which more efficient inference algorithms exist that emphasize approximation of the true underlying DS would be preferable. Recent solutions to the vanishing vs. exploding gradient problem attempt to retain the simplicity of vanilla RNN by initializing or constraining the recurrent weight matrix to be the identity , orthogonal or unitary . In this way, in a system including piecewise linear (PL) components like rectified-linear units (ReLU), line attractor dimensions are established from the start by construction or ensured throughout training by a specifically parameterized matrix decomposition. However, for many DS problems, line attractors instantiated by mere initialization procedures may be unstable and quickly dissolve during training. On the other hand, orthogonal or unitary constraints are too restrictive for reconstructing DS, and more generally from a computational perspective as well : For instance, neither 2) with flow field (grey) and nullclines (set of points at which the flow of one of the variables vanishes, in blue and red). Insets: Time graphs of z 1 for T = 30 000. A) Perfect line attractor. The flow converges to the line attractor from all directions and is exactly zero on the line, thus retaining states indefinitely in the absence of perturbations, as illustrated for 3 example trajectories (green) started from different initial conditions. B) Slightly detuned line attractor (cf.). The system's state still converges toward the'line attractor ghost' , but then very slowly crawls up within the'attractor tunnel' (green trajectory) until it hits the stable fixed point at the intersection of nullclines. Within the tunnel, flow velocity is smoothly regulated by the gap between nullclines, thus enabling arbitrary time constants. Note that along other, not illustrated dimensions of the system's state space the flow may still evolve freely in all directions. C) Simple 2-unit solution to the addition problem exploiting the line attractor properties of ReLUs in the positive quadrant. The output unit serves as a perfect integrator, while the input unit will only convey those input values to the output unit that are accompanied by a'1' in the second input stream (see 7.1.1 for complete parameters). chaotic behavior (that requires diverging directions) nor settings with multiple isolated fixed point or limit cycle attractors are possible. Here we therefore suggest a different solution to the problem, by pushing (but not strictly enforcing) ReLU-based, piecewise-linear RNN (PLRNN) toward line attractor configurations along some (but not all) directions in state space. We achieve this by adding special regularization terms for a subset of RNN units to the loss function that promote such a configuration. We demonstrate that our approach outperforms, or is en par with, LSTM and other, initialization-based, methods on a number of'classical' machine learning benchmarks . More importantly, we demonstrate that while with previous methods it was difficult to capture slow behavior in a DS that exhibits widely different time scales, our new regularization-supported inference efficiently captures all relevant time scales. Long-range dependency problems in RNN. Error gradients in vanilla RNN trained by some form of gradient descent, like back-propagation through time , tend to either explode or vanish due to the large product of derivative terms that from recursive application of the chain rule over time steps (; ;). Formally, RNN z t = F θ (z t−1, s t) are discrete time dynamical systems that tend to either converge, e.g. to fixed point or limit cycle attractors, or diverge (to infinity or as in chaotic systems) over time, unless parameters of the system are precisely tuned to create directions of zero-flow in the system's state space (Fig. 1), called line attractors (; ;). Convergence of the RNN in general implies vanishing and global divergence exploding gradients. To address this issue, RNN with gated memory cells have been specifically designed , but these are complicated and tedious to analyze from a DS perspective. observed that initialization of the recurrent weight matrix W to the identity in ReLU-based RNN may yield performance en par with LSTMs on standard machine learning benchmarks. For a ReLU with activity z t ≥ 0, zero bias and unit slope, this in the identity mapping, hence a line attractor configuration. expanded on this idea by initializing the recurrence matrix such that its largest absolute eigenvalue is 1, arguing that this would leave other directions in the system's state space free for computations other than memory maintenance. Later work enforced orthogonal (; ;) or unitary constraints on the recurrent weight matrix during training. While this appears to yield long-term memory performance superior to that of LSTMs, these networks are limited in their computational power . This may be a consequence of the fact that RNN with orthogonal recurrence matrix are quite restricted in the range of dynamical phenomena they can produce, e.g. chaotic attractors are not possible since diverging eigen-directions are disabled. Our approach therefore is to establish line attractors only along some but not all directions in state space, and to only push the RNN toward these configurations but not strictly enforce them, such that convergence or divergence of RNN dynamics is still possible. We furthermore implement these concepts through regularization terms in the loss functions, such that they are encouraged throughout training unlike when only established through initialization. Dynamical systems reconstruction. From a natural science perspective, the goal of reconstructing the underlying DS fundamentally differs from building a system that'merely' yields good ahead predictions, as in DS reconstruction we require that the inferred model can freely reproduce (when no longer guided by the data) the underlying attractor geometries and state space properties (see section 3.5, Fig. S2 ;). Earlier work using RNN for DS identification mainly focused on inferring the posterior over latent trajectories Z = {z 1, . . ., z T} given time series data X = {x 1, . . ., x T}, p(Z|X), and on ahead predictions , hence did not show that inferred models can generate the underlying attractor geometries on their own. Others attempt to approximate the flow field, obtained e.g. by numerical differentiation, directly through basis expansions or neural networks, but numerical derivatives are problematic for their high variance and other numerical issues (; ;). Some approaches assume the form of the DS equations basically to be given and focus on estimating the system's latent states and parameters, rather than approximating an unknown DS based on the observed time series information alone. In many biological systems like the brain the intrinsic dynamics are highly stochastic with many noise sources, like probabilistic synaptic release , such that models that do not explicitly account for dynamical process noise may be less suitable. Finally, some fully probabilistic models for DS reconstruction based on GRU (, cf.), LSTM , or radial basis function networks are not easily interpretable and amenable to DS analysis. Most importantly, none of these previous approaches considers the long-range dependency problem within more easily tractable RNN for DS reconstruction. Assume we are given two multivariate time series S = {s t} and X = {x t}, one we will denote as'inputs' (S) and the other as'outputs' (X). We will first consider the'classical' (supervised) machine learning setting where we wish to map S on X through a RNN with latent state equation z t = F θ (z t−1, s t), as for instance in the'addition problem' . In DS reconstruction, in contrast, we usually have a dense time series X from which we wish to infer (unsupervised) the underlying DS, where S may provide an additional forcing function or sparse experimental inputs or perturbations. The latent RNN we consider here takes the specific form where the input mapping, h ∈ R M ×1 a bias, and ε t a Gaussian noise term with diagonal covariance ma- This specific formulation is originally motivated by firing rate (population) models in computational neuroscience , where latent states z t may represent membrane voltages or currents, A the neurons' passive time constants, W the synaptic coupling among neurons, and φ(·) the voltage-to-rate transfer function. However, for a RNN in the form z t = W φ (z t−1) + h, note that the simple change of variables y t → W −1 (z t − h) will yield the more familiar form y t = φ (W y t−1 + h) . Besides its neuroscience motivation, note that by letting A = I, W = 0, h = 0, we get a strict line attractor system across the variables' whole support which we conjecture will be of advantage for establishing long short-term memory properties. Also we can solve for all of the system's fixed points analytically by solving the equations z * = (I − A − W D Ω) −1 h, with D Ω as defined in Suppl. 7.1.2, and can determine their stability from the eigenvalues of matrix A + W D Ω. We could do the same for limit cycles, in principle, which are fixed points of the r-times iterated map F r θ, although practically the number of configurations to consider increases exponentially as 2 M ·r. Finally, we remark that a discrete piecewise-linear system can, under certain conditions, be transformed into an equivalent continuous-time (ODE) piecewise-linear systemζ = G Ω (ζ(t), s(t)) (Suppl. 7.1.2,), in the sense that if ζ(t) = z t, then ζ(t + ∆t) = z t+1 after a defined time step ∆t. These are among the properties that make PLRNNs more amenable to rigorous DS analysis than other RNN formulations. We will assume that the latent RNN states z t are coupled to the actual observations x t through a simple observation model of the form in the case of real-valued observations x t ∈ R N ×1, where B ∈ R N ×M is a factor loading matrix and diag(Γ) ∈ R N + the diagonal covariance matrix of the Gaussian observation noise, or in the case of multi-categorical observations x i,t ∈ {0, 1}, i x i,t = 1. We start from a similar idea as , who initialized RNN parameters such that it performs an identity mapping for z i,t ≥ 0. However, 1) we use a neuroscientifically motivated network architecture (eq. 1) that enables the identity mapping across the variables whole support, z i,t ∈ [−∞, +∞], 2) we encourage this mapping only for a subset M reg ≤ M of units (Fig. S1), leaving others free to perform arbitrary computations, and 3) we stabilize this configuration throughout training by introducing a specific L 2 regularization for parameters A, W, and h in eq. 1. That way, we divide the units into two types, where the regularized units serve as a memory that tends to decay very slowly (depending on the size of the regularization term), while the remaining units maintain the flexibility to approximate any underlying DS, yet retaining the simplicity of the original RNN model (eq. 1). Specifically, the following penalty is added to the loss function (Fig. S1): While this formulation allows us to trade off, for instance, the tendency toward a line attractor (A → I, h → 0) vs. the sensitivity to other units' inputs (W → 0), for all experiments performed here a common value, τ A = τ W = τ h = τ, was assumed for the three regularization factors. For comparability with other approaches like LSTMs or iRNN , we will assume that the latent state dynamics eq. 1 are deterministic (i.e., Σ = 0), will take g(z t) = z t and Γ = I N in eq. 2 (leading to an implicit Gaussian assumption with identity covariance matrix), and will use stochastic gradient descent (SGD) for training to minimize the squared-error loss across R samples,, between estimated and actual outputs for the addition and multiplication problems, and the cross entropy loss i,T ) for sequential MNIST, to which penalty eq. 4 was added for the regularized PLRNN (rPLRNN). We used the Adam algorithm from the PyTorch package with a learning rate of 0.001, a gradient clip parameter of 10, and batch size of 16. In all cases, SGD is stopped after 100 epochs and the fit with the lowest loss across all epochs is chosen. For DS reconstruction we request that the latent RNN approximates the true generating system of equations, which is a taller order than learning the mapping S → X or predicting future values in a time series (cf. sect. 3.5). This point has important implications for the design of models, inference algorithms and performance metrics if the primary goal is DS reconstruction rather than'mere' time series forecasting. In this context we consider the fully probabilistic, generative RNN eq. 1. Together with eq. 2 (where we take g(z t) = φ(z t)) this gives the typical form of a nonlinear state space model with observation and process noise. We solve for the parameters θ = {A, W, C, h, Σ, B, Γ} by maximum likelihood, for which an efficient ExpectationMaximization (EM) algorithm has recently been suggested , which we will briefly summarize here. Since the involved integrals are not tractable, we start off from the evidence-lower bound (ELBO) to the log-likelihood which can be rewritten in various useful ways: In the E-step, given a current estimate θ * for the parameters, we seek to determine the posterior p θ (Z|X) which we approximate by a global Gaussian q(Z|X) instantiated by the maximizer (mode) Z * of p θ (Z|X) as an estimator of the mean, and the negative inverse Hessian around this maximizer as an estimator of the state covariance, i.e. since Z integrates out in p θ (X) (equivalently, this can be derived from a Laplace approximation to the log-likelihood, log p(X|θ) where L * is the Hessian evaluated at the maximizer). We solve this optimization problem by a fixed-point iteration scheme that efficiently exploits the model's piecewise linear structure (see Suppl. 7.1.3, ;). Using this approximate posterior for p θ (Z|X), based on the model's piecewise-linear structure most of the expectation values, and E z∼q [φ(z)φ(z) ], could be solved for (semi-)analytically (where z is the concatenated vector form of Z, as in Suppl. 7.1.3). In the Mstep, we seek θ *:= arg max θ L(θ, q *), assuming proposal density q * to be given from the E-step, which for a Gaussian observation model amounts to a simple linear regression problem (see Suppl. eq. 23). To force the PLRNN to really capture the underlying DS in its governing equations, we use a previously suggested (Koppe et al. 2019) stepwise annealing protocol that gradually shifts the burden of fitting the observations X from the observation model eq. 2 to the latent RNN model eq. 1 during training, the idea of which is to establish a mapping from latent states Z to observations X first, fixing this, and then enforcing the temporal consistency constraints implied by eq. 1 while accounting for the actual observations. Measures of prediction error. For the machine learning benchmarks we employed the same criteria as used for optimization (MSE or cross-entropy, sect. 3.3) as performance metrics, evaluated across left-out test sets. In addition, we report the relative frequency P correct of correctly predicted trials Agreement in attractor geometries. From a DS perspective, it is not sufficient or even sensible to judge a method's ability to infer the underlying DS purely based on some form of (ahead-)prediction error like the MSE defined on the time series itself (Ch.12 in). Rather, we require that the inferred model can freely reproduce (when no longer guided by the data) the underlying attractor geometries and state space properties. This is not automatically guaranteed for a model that yields agreeable ahead predictions on a time series. Vice versa, if the underlying attractor is chaotic, with a tiny bit of noise even trajectories starting from the same initial condition will quickly diverge and ahead-prediction errors are not even meaningful as a performance metric (Fig. S2A). To quantify how well an inferred PLRNN captured the underlying dynamics we therefore followed and used the Kullback-Leibler divergence between the true and reproduced probability distributions across states in state space, thus assessing the agreement in attractor geometries (cf. ;) rather than in precise matching of time series, where p true (x) is the true distribution of observations across state space (not time!), p gen (x|z) is the distribution of observations generated by running the inferred PLRNN, and the sum indicates a spatial discretization (binning) of the observed state space (see Suppl. 7.1.4 for more details). We emphasize thatp (k) gen (x|z) is obtained from freely simulated trajectories, i.e. drawn from the prior p(z), not from the inferred posteriorsp(z|x train). (The form ofp(z) is given by the dynamical model eq. 1 and has a'mixture of piecewise-Gaussians' structure, see.) In addition, to assess reproduction of time scales by the inferred PLRNN, we computed the average correlation between the power spectra of the true and generated time series. Fig. S1, with additional regularization term (eq. 4) during training LSTM Long Short-Term Memory 4 NUMERICAL EXPERIMENTS We compared the performance of our rPLRNN to other models on the following three benchmarks requiring long short-term maintenance of information (as in and): 1) The addition problem of time length T consists of 100 000 training and 10 000 test samples of 2 × T input series S = {s 1, . . ., s T}, where entries s 1,: ∈ are drawn from a uniform random distribution and s 2,: ∈ {0, 1} contains zeros except for two indicator bits placed randomly at times t 1 < 10 and t 2 < T /2. Constraints on t 1 and t 2 are chosen such that every trial requires a long memory of at least T /2 time steps. At the last time step T, the target output of the network is the sum of the two inputs in s 1,: indicated by the 1-entries in s 2,:, x target T = s 1,t1 + s 1,t2. 2) The multiplication problem is the same as the addition problem, only that the product instead of the sum has to be produced by the RNN as an output at time T, x target T = s 1,t1 ·s 1,t2. 3) The MNIST dataset consists of 60 000 training and 10 000 28 × 28 test images of hand written digits. To make this a time series problem, in sequential MNIST the images are presented sequentially, pixel-by-pixel, scanning lines from upper left to bottom-right, ing in time series of fixed length T = 784. On all three benchmarks we compare the performance of the rPLRNN (eq. 1) to several other models summarized in Table 1. To achieve a meaningful comparison, all models have the same number of hidden states M, except for the LSTM, which requires three additional parameters for each hidden state and hence has only M/4 hidden states, yielding the overall same number of trainable parameters as for the other models. In all cases, M = 40, which initial numerical exploration suggested to be a good compromise between model complexity (bias) and data fit (variance) (Fig. S3). Fig. 2 summarizes the for the machine learning benchmarks. As can be seen, on the addition and multiplication tasks, and in terms of either the MSE or percentage correct, our rPLRNN outperforms all other tested methods, including LSTMs. Indeed, the LSTM performs even significantly worse than the iRNN and the iPLRNN. The large error bars in Fig. 2 from the fact that the networks mostly learn these tasks in an all-or-none fashion, i.e. either learn the task and succeed in almost 100 percent of the cases or fail completely. The for the sequential MNIST problem are summarized in Fig. 2C. While in this case the LSTM outperforms all other methods, the rPLRNN is almost en par with it. In addition, the iPLRNN outperforms the iRNN. Similar were obtained for M = 100 units (M = 25, respectively, for LSTM; Fig. S6). While the rPLRNN in general outperformed the pure initialization-based models (iRNN, npRNN, iPLRNN), confirming that a line attractor subspace present at initialization may be lost throughout training, we conjecture that this difference in performance will become even more pronounced as noise levels or task complexity increase. Fig. 3: Reconstruction of a 2-time scale DS (biophysical bursting neuron model) in limit cycle regime. A) KL divergence (D KL) between true and generated state space distributions as a function of τ. Unstable (globally diverging) system estimates were removed. B) Average MSE between power spectra of true and reconstructed DS. C) Average normalized MSE between power spectra of true and reconstructed DS split according to low (≤ 50 Hz) and high (> 50 Hz) frequency components. Error bars = SEM in all graphs. D) Example of (best) generated time series (red=reconstruction with τ = 2 3). Here our goal was to examine whether our regularization approach would also help with the identification of DS that harbor widely different time scales. By tuning systems in the vicinity of line attractors, multiple arbitrary time scales can be realized in theory . To test this, we used a biophysically motivated (highly nonlinear) bursting cortical neuron model with one voltage and two conductance recovery variables (see), one slow and one fast (Suppl. 7.1.5). Reproduction of this DS is challenging since it produces very fast spikes on top of a slow nonlinear oscillation (Fig. 3D). Time series of standardized variables of length T = 1500 were generated from this model and provided as observations to the rPLRNN inference algorithm. rPLRNNs with M = {8 . . . 18} states were estimated, with the regularization factor varied within τ ∈ {0, 10 1, 10 2, 10 3, 10 4, 10 5}/1500. Fig. 3A confirms our intuition that stronger regularization leads to better DS reconstruction as assessed by the KL divergence between true and generated state distributions. This decrease in D KL is accompanied by a likewise decrease in the MSE between the power spectra of true (Suppl. eq. 27) and generated (rPLRNN) voltage traces as τ increased (Fig. 3B). Fig. 3D gives an example of voltage traces and gating variables freely simulated (i.e., sampled) from the generative rPLRNN trained with τ = 2 3, illustrating that our model is in principle capable of capturing both the stiff spike dynamics and the slower oscillations in the second gating variable at the same time. Fig. 3C provides more insight into how the regularization worked: While the high frequency components (> 50 Hz) related to the repetitive spiking activity hardly benefitted from increasing τ, there was a strong reduction in the MSE computed on the power spectrum for the lower frequency range (≤ 50 Hz), suggesting that increased regularization helps to map slowly evolving components of the dynamics. In this work we have introduced a simple solution to the long short-term memory problem in RNN that on the one hand retains the simplicity and tractability of vanilla RNN, yet on the other hand does not curtail the universal computational capabilities of RNN and their ability to approximate arbitrary DS (; ;). We achieved this by adding regularization terms to the loss function that encourage the system to form a'memory subspace', that is, line attractor dimensions which would store arbitrary values for, if unperturbed, arbitrarily long periods. At the same time we did not rigorously enforce this constraint which has important implications for capturing slow time scales in the data: It allows the RNN to slightly depart from a perfect line attractor, which has been shown to constitute a general dynamical mechanism for regulating the speed of flow and thus the learning of arbitrary time constants that are not naturally included qua RNN design (; 2004). This is because as we come infinitesimally close to a line attractor and thus a bifurcation in the system's parameter space, the flow along this direction becomes arbitrarily slow until it vanishes completely in the line attractor configuration (Fig. 1). Moreover, part of the RNN's latent space was not regularized at all, leaving the system enough degrees of freedom for realizing arbitrary computations or dynamics. We showed that the rPLRNN is en par with or outperforms initialization-based approaches and LSTMs on a number of classical benchmarks, and, more importantly, that the regularization strongly facilitates the identification of challenging DS with widely different time scales in PLRNN-based algorithms for DS reconstruction. Future work will explore a wider range of DS models and empirical data with diverse temporal and dynamical phenomena. Another future direction may be to replace the EM algorithm by black-box variational inference, using the re-parameterization trick for gradient descent (; ;). While this would come with better scaling in M, the number of latent states (the scaling in T is linear for EM as well, see), the EM used here efficiently exploits the model's piecewise linear structure in finding the posterior over latent states and computing the parameters (see Suppl. 7.1.3). It may thus be more accurate and suitable for smaller-scale problems where high precision is required, as often encountered in neuroscience or physics. 7 SUPPLEMENTARY MATERIAL 7.1 SUPPLEMENTARY TEXT 7.1.1 Simple exact PLRNN solution for addition problem The exact PLRNN parameter settings (cf. eq. 1) for solving the addition problem with 2 units (cf. Fig. 1C) are as follows: Under some conditions we can translate the discrete into an equivalent continuous time PLRNN. Using D Ω(t) as defined below (7.1.3) for a single time step t, we can rewrite (ignoring the noise term and inputs) PLRNN eq. 1 in the form where Ω(t):= {m|z m,t > 0} is the set of all unit indices with activation larger 0 at time t. To convert this into an equivalent (in the sense defined in eq. 11) system of (piecewise) ordinary differential equations (ODE), we need to find parameters W Ω and h, such that where ∆t is the time step with which the empirically observed time series X was sampled. From these conditions it follows that for each of the s ∈ {1, . . ., 2 M} subregions (orthants) defined by fixed index sets Ω s ⊆ {1, . . ., M} we must have where we assume that D Ω s is constant for one time step, i.e. between 0 and ∆t. We approach this by first solving the homogeneous system using the general ansatz for systems of linear ODEs, where we have used z 0 = k c k v k on lines 15 and 16. Hence we can infer matrix W Ω s from the eigendecomposition of matrix W Ω s, by lettingλ k = 1 ∆t log λ k, where λ k are the eigenvalues of W Ω s, and reassembling We obtain the general solution for the inhomogeneous case by requiring that for all fixed points z * = F (z *) of the map eq. 9 we have G(z *) = 0. Using this we obtaiñ Assuming inputs s t to be constant across time step ∆t, we can apply the same transformation to input matrix C. Fig. S5 illustrates the discrete to continuous PLRNN conversion for a nonlinear oscillator. Note that in the above derivations we have assumed that matrix W Ω s can be diagonalized, and that all its eigenvalues are nonzero (in fact, W Ω s should not have any negative real eigenvalues). In general, not every discrete-time PLRNN can be converted into a continuous-time ODE system in the sense defined above. For instance, we can have chaos in a 1d nonlinear map, while we need at least a 3d ODE system to create chaos . Here we briefly outline the fixed-point-iteration algorithm for solving the maximization problem in eq. 6 (for more details see ;). Given a Gaussian latent PLRNN and a Gaussian observation model, the joint density p(X, Z) will be piecewise Gaussian, hence eq. 6 piecewise quadratic in Z. Let us concatenate all state variables across m and t into one long column vector z = (z 1,1, . . ., z M,1, . . ., z 1,T, . . ., z M,T), arrange matrices A, W into large M T × M T block tri-diagonal matrices, define d Ω:= 1 z1,1>0, 1 z2,1>0,..., 1 z M,T >0 as an indicator vector with a 1 for all states z m,t > 0 and zeros otherwise, and D Ω:= diag(d Ω) as the diagonal matrix formed from this vector. Collecting all terms quadratic, linear, or constant in z, we can then write down the optimization criterion in the form In essence, the algorithm now iterates between the two steps: 2. Given fixed z *, recompute D Ω until either convergence or one of several stopping criteria (partly likelihood-based, partly to avoid loops) is reached. The solution may afterwards be refined by one quadratic programming step. Numerical experiments showed this algorithm to be very fast and efficient . At z *, an estimate of the state covariance is then obtained as the inverse negative Hessian, In the M-step, using the proposal density q * from the E-step, the solution to the maximization problem θ *:= arg max θ L(θ, q *), can generally be expressed in the form where, for the latent model, eq. 1, α t = z t and β t:= z t−1, φ(z t−1), s t, 1 ∈ R 2M +K+1, and for the observation model, eq. 2, α t = x t and β t = g (z t). The measure D KL introduced in the main text for assessing the agreement in attractor geometries only works for situations where the ground truth p true (X) is known. , here we would like to briefly indicate how a proxy for D KL may be obtained in empirical situations where no ground truth is available. Reasoning that for a well reconstructed DS the inferred posterior p inf (z|x) given the observations should be a good representative of the prior generative dynamics p gen (z), one may use the Kullback-Leibler divergence between the distribution over latent states, obtained by sampling from the prior density p gen (z), and the (data-constrained) posterior distribution p inf (z|x) (where z ∈ R M ×1 and x ∈ R N ×1), taken across the system's state space: As evaluating this integral is difficult, one could further approximate p inf (z|x) and p gen (z) by Gaussian mixtures across trajectories, i.e., where the mean and covariance of p(z t |x 1:T) and p(z l |z l−1) are obtained by marginalizing over the multivariate distributions p(Z|X) and p gen (Z), respectively, yielding E[z t |x 1:T], E[z l |z l−1], and covariance matrices Var(z t |x 1:T) and Var(z l |z l−1). Supplementary eq. 24 may then be numerically approximated through Monte Carlo sampling by For high-dimensional state spaces, for which MC sampling becomes challenging, there is luckily a variational approximation of eq. 24 available : where the KL divergences in the exponentials are among Gaussians for which we have an analytical expression. The neuron model used in section 4.2 is described by σ(V) = 1 +.33e where C m refers to the neuron's membrane capacitance, the g • to different membrane conductances, E • to the respective reversal potentials, and m, h, and n are gating variables with limiting values given by Different parameter settings in this model lead to different dynamical phenomena, including regular spiking, slow bursting or chaos (see with parameters in the chaotic regime (blue curves) and with simple fixed point dynamics in the limit (red line). Although the system has vastly different limiting behaviors (attractor geometries) in these two cases, as visualized in the state space, the agreement in time series initially seems to indicate a perfect fit. B) Same as in A) for two trajectories drawn from exactly the same DS (i.e., same parameters) with slightly different initial conditions. Despite identical dynamics, the trajectories immediately diverge, ing in a high MSE. Dash-dotted grey lines in top graphs indicate the point from which onward the state space trajectories were depicted.

Represent the TLDR for finding the corresponding scientific article

We develop a new optimization approach for vanilla ReLU-based RNN that enables long short-term memory and identification of arbitrary nonlinear dynamical systems with widely differing time scales.

represent the tldr for finding the corresponding scientific article

We show that neural networks operate by changing topologly of a data set and explore how architectural choices effect this change.


Question: Represent the scientific article to find a short TLDR

Question: In the field of Generative Adversarial Networks (GANs), how to design a stable training strategy remains an open problem. Wasserstein GANs have largely promoted the stability over the original GANs by introducing Wasserstein distance, but still remain unstable and are prone to a variety of failure modes. In this paper, we present a general framework named Wasserstein-Bounded GAN (WBGAN), which improves a large family of WGAN-based approaches by simply adding an upper-bound constraint to the Wasserstein term. Furthermore, we show that WBGAN can reasonably measure the difference of distributions which almost have no intersection. Experiments demonstrate that WBGAN can stabilize as well as accelerate convergence in the training processes of a series of WGAN-based variants. Over the past few years, Generative Adversarial Networks (GANs) have shown impressive in many generative tasks. They are inspired by the game theory, that two models compete with each other: a generator which seeks to produce samples from the same distribution as the data, and a discriminator whose job is to distinguish between real and generated data. Both models are forced stronger simultaneously during the training process. GANs are capable of producing plausible synthetic data across a wide diversity of data modalities, including natural images (; ;), natural language (; ;), music;; ), etc. Despite their success, it is often difficult to train a GAN model in a fast and stable way, and researchers are facing issues like vanishing gradients, training instability, mode collapse, etc. This has led to a proliferation of works that focus on improving the quality of GANs by stabilizing the training procedure (; ; ; ; ;). In particular, introduced a variant of GANs based on the Wasserstein distance, and releases the problem of gradient disappearance to some extent. However, WGANs limit the weight within a range to enforce the continuity of Lipschitz, which can easily cause over-simplified critic functions . To solve this issue, proposed a gradient penalty method termed WGAN-GP, which replaces the weight clipping in WGANs with a gradient penalty term. As such, WGAN-GP provides a more stable training procedure and succeeds in a variety of generating tasks. Based on WGAN-GP, more works (; ; ; ; ; ; adopt different forms of gradient penalty terms to further improve training stability. However, it is often observed that such gradient penalty strategy sometimes generate samples with unsatisfying quality, or even do not always converge to the equilibrium point . In this paper, we propose a general framework named Wasserstein-Bounded GAN (WBGAN), which improve the stability of WGAN training by bounding the Wasserstein term. The highlight is that the instability of WGANs also resides in the dramatic changes of the estimated Wasserstein distance during the initial iterations. Many previous works just focused on improving the gradient penalty term for stable training, while they ignored the bottleneck of the Wasserstein term. The proposed training strategy is able to adaptively enforce the Wasserstein term within a certain value, so as to balance the Wasserstein loss and gradient penalty loss dynamically and make the training process more stable. WBGANs are generalized, which can be instantiated using different kinds of bound estimations, and incorporated into any variant of WGANs to improve the training stability and accelerate the convergence. Specifically, with Sinkhorn distance for bound estimation, we test three representative variants of WGANs (WGAN-GP , WGANdiv , and WGAN-GPReal ) on the CelebA dataset . As shown in Fig. 1 Wasserstein GANs (WGANs). WGANs were primarily motivated by unstable training caused by the gradient vanishing problem of the original GANs . They proposed to use 1-Wasserstein distance W 1 (P r, P g) to measure the difference between P r and P g, the real and generated distributions, given that W 1 (P r, P g) is continuous everywhere and differentiable almost everywhere under mild assumptions. The objective of WGAN is formulated using the KantorovichRubinstein duality : where L 1 is the function space of all D satisfying the 1-Lipschitz constraint D L ≤ 1. D is a critic and G is the generator, both of which are parameterized by a neural network. Under an optimal critic, minimizing the objective with respect to G is to minimize W 1 (P r, P g). To enforce the 1-Lipschitz constraint on the critic, WGAN used a weight clipping on the critic to constrain the weights within a compact range, [−c, c], which guarantees the set of critic functions is a subset of the k-Lipschitz functions for some k. With weight clipping, the critic tends to learn over-simplified functions , which may lead to unsatisfying .;;; proposed different forms of gradient penalty as a regularization term, so that a generalized loss function with respect to the critic can be written as: where ] stands for the Wasserstein term, and GP for the gradient penalty term. L D is actually posing a tradeoff between these two objectives. Wasserstein Distance between Empirical Distributions. In practice, we approximate W 1 (P r, P g) using W 1 P r,P g, whereP r andP g denote the empirical version of P r and P g with N samples, Here, y i is randomly sampled from the real image dataset, and δ yi is the Dirac delta function at location y i. Computing W 1 P r,P g is a typical problem named discrete optimal transport. We denote B as the set of probabilistic couplings between two empirical distributions defined as: where 1 N is a N -dimensional all-one vector. Then we have W 1 (P r,P g) = min γ∈B Γ, C F, where ·, · F is the Frobenius dot-product and C is the cost matrix, with each element C i,j = c(G(z i), y j ) denoting the cost to move a probability mass from G(z i) to y j. The optimal coupling is the solution of this minimization problem: Γ 0 = arg min Γ∈B Γ, C F. The Sinkhorn Algorithm. Despite Wasserstein distance has appealing theoretical properties in measuring the difference between distributions, its computational costs for linear programming are often high in particular when the problem size becomes large. To alleviate this burden, Sinkhorn distance was proposed to approximate Wasserstein distance: where U α (P r,P g) is a subset of B defined in Eq. 3: where H(·) is the entropy defined as H(Γ) = − N i,j=1 Γ i,j log Γ i,j and H(P r) = − N n=1p n logp n wherep n is the probability of the n-th sample. Compared to Wasserstein distance, Sinkhorn distance restricts the search space of joint probabilities to those with sufficient smoothness. To compute Sinkhorn distance, a Lagrange multiplier was used: can be computed with a much cheaper cost than the original Wasserstein distance using matrix scaling algorithms. For λ > 0, the solution Γ λ is unique and has the form Γ λ = diag(u)K diag(v), where K is the element-wise exponential of −λC. u and v are two non-negative vectors uniquely defined up to a multiplicative factor (. W is often referred to as the Wasserstein term, which is unbounded during the training process. In a wide range of WGAN's variants such as WGAN-GP , the critic defined by L D is to maximize the Wasserstein term W while satisfying the gradient penalty GP. However, in practice, we find that W often rises rapidly to a tremendous value which is far from rational during the initial training procedure. A possible reason may lie in that the critic function does not satisfy the Lipschitz constraint during the initial training stage. As shown in Fig. 2, this leads to dramatic instability in optimization and finally in unsatisfying performance in image generation. Our idea is thus straightforward, i.e., setting an upper-bound for W. The modified critic loss function is written as: Our formulation brings a benefit to the numerical stability of the Wasserstein term. In practice, it remains comparable to the other term, λ GP · GP, so that both W and GP can be optimized in a'mild' manner, i.e., without any one of them dominating or being ignored during training. Note that the Ď W term cannot be chosen arbitrarily. Setting it too small, Ď W will limit the capacity of the critic function, ing in a poor generation. Setting it too large, there will be no effect of bounding the W term. The proposed bounded strategy is a general framework. We name it general in two folds: First, WBGAN can be applied to almost all gradient penalty based WGANs, such as WGAN-GP , WGAN-GPReal , etc. Moreover, there are different ways to estimate the value of Ď W. For example, the linear programming was applied successfully to some existing WGANs like WGAN-TS. In what follows, we present an example which uses Sinkhorn distance to estimate Ď W, while we believe other ways of estimation are also possible. In this section, we give an instantiation, Sikhorn distance , to effectively compute the bounded term Ď W. The motivation of using Sinkhorn distance lies in that in theory, the Wasserstein term of WGAN will eventually converge to the 1-Wasserstein distance between the real distribution P r and the generated distribution P g ). Therefore, we can use the 1-Wasserstein distance between the empirical distributions,P r andP g, as the upper-bound Ď W. Since the computation of Wasserstein distance involves a large linear programming which Algorithm 1 WBGAN with Sinkhorn distance Require: learning rate α, batch size M, the number of iterations of the critic per generator iteration N critic, weight of gradient penalty λ GP, weight of Sinkhorn distance λ s, initial parameters θ and φ 0, other hyper-parameters; 1: while φ t has not converged do 2: Sample a batch {z (m) } M m=1 ∼ P z of prior samples; 5: 7: end for 10: Sample a batch {z (m) } M m=1 ∼ P z of prior samples; 11: 12: φ t+1 ← Adam(L φt, φ t, α, β 1, β 2); 14: end while Ensure: trained parameters θ and φ T (converged). suffers heavy computational costs, we replace it by Sinkhorn distance instead -the Sinkhorn distance betweenP r andP g can be computed using Sinkhorn's matrix scaling algorithm , which is orders of magnitude faster than the linear programming solvers. Mathematically, consider a generator function G φ (z) that produces samples by transforming noise input z drawn from a simple distribution P z, e.g., Gaussian distribution. D θ stands for a critic function parameterized by θ. The objective of the critic is: where d λ (P r,P g) is the Sinkhorn distance defined in Eq. 6. On the other hand, given a fixed critic function D θ, considering that Sinkhorn distance allows gradient back-propagation , we can find the optimal generator G φ by solving: where λ s is a balancing hyper-parameter, which we set λ s = 0.5 in this paper. In Algorithm 1, we summarize the flowchart of training WBGAN with Sinkhorn distance. ) be a separable metric space. P(X) denotes the set of Borel probability measures. P p (X) denotes the set of all µ ∈ P(X) such that X d(x, y) p dµ(x) < +∞ for some y ∈ X. We can suppose real data distribution P r, generated data distribution P g and their empirical distributionP r andP g all in P p (X). Proposition 1. Let P r and P g be real data distribution and generated data distribution. Suppose that P r andP g are empirical measures of P r and P g. Then we have 0 Proof. Please refer to Appendix A. Proposition 1 tells us that as E[W 1 (P r,P g)] → 0, W 1 (P r, P g) is forced to 0. has pointed out that if λ is chosen large enough, d λ (P r,P g) coincides with W 1 (P r,P g). So, it is reasonable to use d λ (P r,P g) to constrain the Wasserstein term. Most GANs measure the distance between distributions based on probability divergence. We will prove that the Eq. 8 is indeed a valid divergence. First, we have the following definition. Definition 1. Given probability measures p and q, D is a functional of p and q. If D satisfies the following properties: then we say D is a probability divergence between p and q. Remark 1. The following W (P r, P g) satisfies the Definition 1 and is therefore a probability divergence. where L 1 is the 1-Lipschitz constraint. Please see the proof and detailed discussion in. This is the objective of critic used by WGAN. Remark 2. Equation 8 satisfies the Definition 1 and is a probability divergence. Proof. The proof is given in Appendix B. Remark 3. Consider two distributions P r (x) = δ(x − α), P g (x) = δ(x − β) that have no intersection (α = β). δ is the Dirac delta function. In such an extreme case, Eq. 8 can still be optimized by gradient descent. Proof. The proof is in Appendix C Remark 2 tells us that Eq. 8 is a valid divergence. Since the real data distribution is supported by lowdimensional manifolds, the supports of generated distribution and real data distribution are unlikely to have a non-negligible intersection. Remark 3 shows that compared to the standard GAN , WBGAN can continuously measure the difference between two distributions, even if there is almost no intersection between the distributions. To verify that WBGAN is a generalized approach, we select three variants of WGAN, namely, WGAN-GP , WGAN-div and WGAN-GPReal (gradient penalty on real data only) as our baselines. By adding bound constraints to these WGAN variants, we obtain the counterparts WBGAN-GP, WBGAN-div, and WBGANGPReal, respectively. Two different network architectures are used, i.e., DCGAN and BigGAN . For DCGAN, we directly output the activation before the sigmoid layer. BigGAN is a conditional GAN architecture, in which class conditioning is passed to generator by supplying it with class-conditional gains and biases in the batch normalization layer (; de ; . In addition, the discriminator is conditioned by using the cosine similarity between its features and a set of learned class embedding. We use the spectral norm in BigGAN, but for the sake of simplicity, we do not use the self-attention module (; . Other hyper-parameters and the network architecture of BigGAN simply follow the original paper. We choose the Fréchet Inception Distance (FID) for quantitative evaluation, which has been proven to be more consistent with individual assessment in evaluating the fidelity and variation of the generated image samples. We first investigate mid-resolution image generation on the CelebA dataset , a large-scale face image dataset with more than 200K face images. During training, we crop 108 × 108 face from the original images and then resize them to 64 × 64. FID Stability. We first use DCGAN to build our generator and discriminator. Training curves are shown in Fig. 2, and quantitative are summarized in Table 1. Each approach is executed for 5 times and the average is reported. All FID curves are obtained from generators directly without using the moving average strategy (; ; ; Yazıcıet al., 2018) to avoid over-smoothing the FID curves, such that we can diagnose the underlying oscillating properties of different methods during training. One can see that WBGAN-based counterparts improve the stability during training, and achieve superior performance over the WGAN-based baselines. We emphasize that the converged FID values reported by WBGAN-div and WBGANGPReal are lower than those reported by WGAN-div and WGAN-GPReal. In particular, WGAN-div suffers several FID fluctuation unexpectedly, and WGAN-GPReal has not ever achieved FID convergence during the entire training process. Regarding WGAN-GP, although the final FID is slightly better than that of WBGAN-GP (6.76 vs. 7.32), we observe a much slower convergence rate in Fig. 2(a). For the generated face images by different approaches, please refer to We also investigate a stronger backbone by replacing the network with BigGAN, a conditional GAN architecture that uses spectral normalization on both generator and discriminator. We set the number of labels to be 1 since the CelebA dataset only contains face images. Training curves are shown in Fig. 3 and quantitative are summarized in Table 1. Among three WGAN-based methods, only WGAN-GP achieves convergence, but its convergence speed and the FID value are inferior to those reported by WBGAN-GP. In opposite, both WGAN-div and WGAN-GPReal fails to converge while the counterparts equipped with WB-GAN perform well. For the generated face images by different approaches, please refer to Fig. 11 and Fig. 12 in Appendix F for details. Epoch Wasserstein Loss and Generator Loss Stability. Next, we evaluate the stability of WBGAN in terms of the Wasserstein term and generator loss. In Fig. 4, we evaluate the impact on WGAN-GP (DCGAN on CelebA). One can see that, after the bound is applied, the Wasserstein term W is stablized especially during the start of training. Due to space limit, more using BigGAN on CelebA are provided in Appendix E. In addition, we compute a new term named the generator loss, which is defined as Fig. 6 shows the curves of this statistics during the starting iterations. Compared to WGAN-based approaches, WBGAN-based approaches produce more stable G loss terms, which verifies that the training process of GAN becomes more stable. Ablation Study. Before continuing to high-resolution experiments, we conduct an ablation study to investigate the contribution made by different components of WBGAN. The backbone network is DCGAN, and the dataset is CelebA. We compare four configurations, i.e., WGAN-GP, with the original loss term used in WGAN-GP; WGAN-GP+D-bound, which adds a bound (Sinkhorn distance) to the Wasserstein term of the critic D of WGAN-GP; WGAN-GP+G-Sinkhorn, which adds Sinkhorn distance to the loss function of the generator G in WGAN-GP; and WGAN-GP+D-bound+G-Sinkhorn, which is equivalent to the final WBGAN-GP, with Sinkhorn distance added to both critic D and generator G. Fig. 5 plots the FID curves of all four settings. One can see that, although the FID curves of WGAN-GP and WGAN-GP+G-Sinkhorn descend quickly in the first 10 epochs, they begin to fluctuate between 20 to 40 epochs. On the other hand, when WGAN-GP is combined with D-bound, FID is able to descend smoothly (without fluctuation), showing that it is the bounded constraint that stablizes the training process. Finally, by integrating both D-bound and G-Sinkhorn into WGAN-GP, the FID curve descends not only smoothly but also fast, which is what we desire in real-world applications. In this section, we evaluate our approach on higher-resolution (128 × 128) images. We use the CelebA-HQ dataset , and use BigGAN as the backbone. As the target become larger (128 × 128), the number of images we can feed into a single batch becomes smaller. Since we are using an empirical way of estimating Sinkhorn distance, it becomes less accurate in the scenario of small batch size and large image size. In other words, it is no longer the best choice to use Sinkhorn distance to estimate the upper-bound Ď W. Returning to our generalized formulation, Eq. 7, we note that other forms of bound to constrain the critic. Here we consider a very simple bound, which is also based on empirical study. Note that the baseline methods, though not converging very well, can finally arrive at a stablized W value. Heuristically, we use this constant value (there is no need to be accurate) as the bound, which is 10 for WGAN-GP, 5 for WGAN-div and 3 for WGAN-GPReal, respectively. In Appendix D, we provide the curves of the Wasserstein term for these baselines, which lead to our estimation. FID curves and quantitative using these constant bounds are shown in Fig. 7 and Table 2, respectively. We find that WBGAN-GP produces a similar convergence rate with WGAN-GP, WBGAN-div is slightly better than WGAN-div, and WBGAN-GPReal outperforms WGAN-GPReal and produces the best . For the generated face images by different approaches, please refer to Fig. 13 and Fig. 14 in Appendix F for details. Discussions. From the above experiments, we can see that Sinkhorn distance is just one way of upper-bound estimation. In case that it becomes less accurate, we can freely replace it with other types of estimation. Besides the constant bound used above, there also exist other examples, such as the two-step computation of the exact Wasserstein distance. However, it is still a challenge to estimate the Wasserstein distance between high-resolution (1024 × 1024) image distributions efficiently. Nevertheless, the most important deliveries of our work are that a bounded Wasserstein term can bring benefits on training stability, and that we can use it to a wide range of frameworks based on WGAN. This paper introduced a general framework called WBGANs, which can be applied to a variety of WGAN variants to stabilize the training process and improve the performance. We clarify that WBGANs can stabilize the Wasserstein term at the beginning of the iterations, which is beneficial for smoother convergence of WGAN-based methods. We present an instantiated bound estimation method via Sinkhorn distance and give a theoretical analysis on it. It remains an open topic on how to set a better bound for higher resolution image generation tasks. Proof. Suppose µ, υ 1, υ 2 ∈ P p (X), t 1, t 2 ≥ 0, t 1 + t 2 = 1, then there exist γ 1 (x, y) and γ 2 (x, y) with marginals (µ, υ 1) and (µ, υ 2) satisfying: Let υ = t 1 υ 1 + t 2 υ 2, γ(x, y) = t 1 γ 1 (x, y) + t 2 γ 2 (x, y), then γ(x, y) has marginals (µ, υ). We can derive: This can be extended to a general form: where are the independent empirical measures drawn fromP g. From Eq. 15, we can get According to the strong law of large numbers, we can derive that with probability 1, 1 n n i=1P gi → P g as n → ∞ (assuming P g has finite first moments). Since W 1 is continuous in P p (X), we can derive that with probability 1, W 1 (P r, 1 n n i=1P gi) → W 1 (P r, P g) as n → ∞. By the law of large numbers again, with probability 1, 1 n n i=1 W 1 (P r,P gi) → E[W 1 (P r,P g)] as n → ∞. Thus we can deduce that: Similarly, supposeP ri (1 ≤ i ≤ n) are the independent empirical measures drawn fromP r. Since the symmetry of Wasserstein distance, we can deduce that: Therefore, combining Eq. 17 and Eq. 18, we can get B PROOF OF REMARK 2 where d λ (P r,P g) ≥ 0 is the Sinkhorn distance defined in Eq. 6. Next, if P r = P g, then we have L θ (P r, P g) = 0. So we only need to show L θ (P r, P g) > 0 if Applying this into Eq. 8 leads to L θ (P r, P g) = max Since P r = P g, we know that d λ (P r,P g) > 0. Therefore, we have L θ (P r, P g) > 0 while P r = P g. We finish the proof. Proof. Let P r (x) = δ(x − α), P g (x) = δ(x − β) and α = β, then we have We know that Wasserstein distance W (P r, P g) = max D θ ∈L1 D θ (α) − D θ (β). Since P r, P g are Dirac distributions, then we have W (P r, P g) = d λ (P r,P g). Combining this into Eq. 22 leads to L θ (P r, P g) = d λ (P r,P g). Considering that Sinkhorn distance d λ (P r,P g)

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Propose an improved framework for WGANs and demonstrate its better performance in theory and practice.

represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Generative models such as Variational Auto Encoders (VAEs) and Generative Adversarial Networks (GANs) are typically trained for a fixed prior distribution in the latent space, such as uniform or Gaussian. After a trained model is obtained, one can sample the Generator in various forms for exploration and understanding, such as interpolating between two samples, sampling in the vicinity of a sample or exploring differences between a pair of samples applied to a third sample. In this paper, we show that the latent space operations used in the literature so far induce a distribution mismatch between the ing outputs and the prior distribution the model was trained on. To address this, we propose to use distribution matching transport maps to ensure that such latent space operations preserve the prior distribution, while minimally modifying the original operation. Our experimental validate that the proposed operations give higher quality samples compared to the original operations. Generative models such as Variational Autoencoders (VAEs) BID6 and Generative Adversarial Networks (GANs) BID3 have emerged as popular techniques for unsupervised learning of intractable distributions. In the framework of Generative Adversarial Networks (GANs) BID3, the generative model is obtained by jointly training a generator G and a discriminator D in an adversarial manner. The discriminator is trained to classify synthetic samples from real ones, whereas the generator is trained to map samples drawn from a fixed prior distribution to synthetic examples which fool the discriminator. Variational Autoencoders (VAEs) BID6 are also trained for a fixed prior distribution, but this is done through the loss of an Autoencoder that minimizes the variational lower bound of the data likelihood. For both VAEs and GANs, using some data X we end up with a trained generator G, that is supposed to map latent samples z from the fixed prior distribution to output samples G(z) which (hopefully) have the same distribution as the data. In order to understand and visualize the learned model G(z), it is a common practice in the literature of generative models to explore how the output G(z) behaves under various arithmetic operations on the latent samples z. In this paper, we show that the operations typically used so far, such as linear interpolation BID3, spherical interpolation , vicinity sampling and vector arithmetic BID12, cause a distribution mismatch between the latent prior distribution and the of the operations. This is problematic, since the generator G was trained on a fixed prior and expects to see inputs with statistics consistent with that distribution. We show that this, somewhat paradoxically, is also a problem if the support of ing (mismatched) distribution is within the support of a uniformly distributed prior, whose points all have equal likelihood during training. To address this, we propose to use distribution matching transport maps, to obtain analogous latent space operations (e.g. interpolation, vicinity sampling) which preserve the prior distribution of the latent space, while minimally changing the original operation. In Figure 1 we showcase how our proposed technique gives an interpolation operator which avoids distribution mismatch when interpolating between samples of a uniform distribution. The points of the (red) matched trajectories samples from prior linear matched (ours) spherical (a) Uniform prior: Trajectories of linear interpolation, our matched interpolation and the spherical interpolation . Figure 1: We show examples of distribution mismatches induced by the previous interpolation schemes when using a uniform prior in two dimensions. Our matched interpolation avoids this with a minimal modification to the linear trajectory, traversing through the space such that all points along the path are distributed identically to the prior.are obtained as minimal deviations (in expectation of l 1 distance) from the the points of the (blue) linear trajectory. In the literature there are dozens of papers that use sample operations to explore the learned models. BID0 use linear interpolation between neighbors in the latent space to study how well deep vs shallow representations can disentangle the latent space of Contractive Auto Encoders (CAEs) BID14.In the seminal GAN paper of BID3, the authors use linear interpolation between latent samples to visualize the transition between outputs of a GAN trained on MNIST. BID2 linearly interpolate the latent codes of an auto encoder trained on a synthetic chair dataset. BID12 also linearly interpolate between samples to evaluate the quality of the learned representation. Furthermore, motivated by the semantic word vectors of , they explore using vector arithmetic on the samples to change semantics such as adding a smile to a generated face. BID13 use linear interpolation to explore their proposed GAN model which operates jointly in the visual and textual domain. BID1 combine GANs and VAEs for a neural photo editor, using masked interpolations to edit an embedded photo in the latent space. While there are numerous works performing operations on samples, most of them have ignored the problem of distribution mismatch, such as the one presented in Figure 1d. BID6 and BID9 sidestep the problem when visualizing their models, by not performing operations on latent samples, but instead restrict the latent space to 2-d and uniformly sample the percentiles of the distribution on a 2-d grid. This way, the samples have statistics that are consistent with the prior distribution. However, this approach does not scale up to higher dimensions -whereas the latent spaces used in the literature can have hundreds of dimensions. Related to our work, experimentally observe that there is a distribution mismatch between the distance to origin for points drawn from uniform or Gaussian distribution and points obtained with linear interpolation, and propose to use a so-called spherical linear interpolation to reduce the mismatch, obtaining higher quality interpolated samples. However, the proposed approach has no theoretical guarantees. DISPLAYFORM0 Vicinity sampling Table 1: Examples of interesting sample operations which need to be adapted if we want the distribution of the y to match the prior distribution. If the prior is Gaussian, our proposed matched operation simplifies to a proper re-scaling factor (see third column) for additive operations. DISPLAYFORM1 In this work, we propose a generic method to fully preserve the desired prior distribution when using sample operations. The approach works as follows: we are given a'desired' operation, such as linear interpolation y = tz 1 + (1 − t)z 2, t ∈. Since the distribution of y does not match the prior distribution of z, we search for a warping f: DISPLAYFORM2 has the same distribution as z. In order to have the modificationỹ as faithful as possible to the original operation y, we use optimal transform maps BID17; 2008) to find a minimal modification of y which recovers the prior distribution z. Figure 1a, where each pointỹ of the matched curve is obtained by warping a corresponding point y of the linear trajectory, while not deviating too far from the line. With implicit models such as GANs BID3 and VAEs BID6, we use the data X, drawn from an unknown random variable x, to learn a generator G: DISPLAYFORM0 with respect to a fixed prior distribution p z, such that G(z) approximates x. Once the model is trained, we can sample from it by feeding latent samples z through G.We now bring our attention to operations on latent samples DISPLAYFORM1 We give a few examples of such operations in Table 1.Since the inputs to the operations are random variables, their output y = κ(z 1, · · ·, z k) is also a random variable (commonly referred to as a statistic). While we typically perform these operations on realized (i.e. observed) samples, our analysis is done through the underlying random variable y. The same treatment is typically used to analyze other statistics over random variables, such as the sample mean, sample variance and test statistics. In Table 1 we show example operations which have been commonly used in the literature. As discussed in the Introduction, such operations can provide valuable insight into how the trained generator G changes as one creates related samples y from some source samples. The most common such operation is the linear interpolation, which we can view as an operation DISPLAYFORM2 where z 1, z 2 are latent samples from the prior p z and y t is parameterized by t ∈. Now, assume z 1 and z 2 are i.i.d, and let Z 1, Z 2 be their (scalar) first components with distribution p Z. Then the first component of y t is Y t = tZ 1 + (1 − t)Z 2, and we can compute: DISPLAYFORM3 Since (1 + 2t(t − 1)) = 1 for all t ∈ \ {0, 1}, it is in general impossible for y t to have the same distribution as z, which means that distribution mismatch is inevitable when using linear interpolation. A similar analysis reveals the same for all of the operations in Table 1.This leaves us with a dilemma: we have various intuitive operations (see Table 1) which we would want to be able to perform on samples, but their ing distribution p yt is inconsistent with the distribution p z we trained G for. Due to the curse of dimensionality, as empirically observed by , this mismatch can be significant in high dimensions. We illustrate this in FIG1, where we plot the distribution of the squared norm y t 2 for the midpoint t = 1/2 of linear interpolation, compared to the prior In order to address the distribution mismatch, we propose a simple and intuitive strategy for constructing distribution preserving operators, via optimal transport: Strategy 1 (Optimal Transport Matched Operations). 2. We analytically (or numerically) compute the ing (mismatched) distribution p y 3. We search for a minimal modificationỹ = f (y) (in the sense that E y [c(ỹ, y)] is minimal with respect to a cost c), such that distribution is brought back to the prior, i.e. pỹ = p z.The cost function in step 3 could e.g. be the euclidean distance c(x, y) = x − y, and is used to measure how faithful the modified operator,ỹ = f (κ(z 1, · · ·, z k)) is to the original operator k. Finding the map f which gives a minimal modification can be challenging, but fortunately it is a well studied problem from optimal transport theory. We refer to the modified operationỹ as the matched version of y, with respect to the cost c and prior distribution p z.For completeness, we introduce the key concept of optimal transport theory in a simplified setting, i.e. assuming probability distributions are in euclidean space and skipping measure theoretical formalism. We refer to Villani (2003; 2008) and BID17 for a thorough and formal treatment of optimal transport. The problem of step above was first posed by Monge and can more formally be stated as: Problem 1 (Problem 1.1). Given probability distributions p x, p y, with domains X, Y respectively, and a cost function c: X × Y → R +, we want to minimize DISPLAYFORM0 We refer to the minimizer f * X → Y of (MP) (if it exists), as the optimal transport map from p x to p y with respect to the cost c. However, the problem remained unsolved until a relaxed problem was studied by:Problem 2 (Problem 1.2). Given probability distributions p x, p y, with domains X, Y respectively, and a cost function c: X × Y → R +, we want to minimize DISPLAYFORM1 where (x, y) ∼ p x,y, x ∼ p x, y ∼ p y denotes that (x, y) have a joint distribution p x,y which has (previously specified) marginals p x and p y.We refer to the joint p x,y which minimizes (KP) as the optimal transport plan from p x to p y with respect to the cost c. The key difference is to relax the deterministic relationship between x and f (x) to a joint probability distribution p x,y with marginals p x and p y for x and y. In the case of Problem 1, the minimization might be over the empty set since it is not guaranteed that there exists a mapping f such that f (x) ∼ y. In contrast, for Problem 2, one can always construct a joint density p x,y with p x and p y as marginals, such as the trivial construction where x and y are independent, i.e. DISPLAYFORM2 Note that given a joint density p x,y (x, y) over X × Y, we can view y conditioned on x = x for a fixed x as a stochastic function f (x) from X to Y, since given a fixed x do not get a specific function value f (x) but instead a random variable f (x) that depends on x, with f (x) ∼ y|x = x with density p y (y|x = x):= px,y(x,y)px (x). In this case we have (x, f (x)) ∼ p x,y, so we can view the Problem KP as a relaxation of Problem MP where f is allowed to be a stochastic mapping. While the relaxed problem of Kantorovich (KP) is much more studied in the optimal transport literature, for our purposes of constructing operators it is desirable for the mapping f to be deterministic as in (MP).To this end, we will choose the cost function c such that the two problems coincide and where we can find an analytical solution f or at least an efficient numerical solution. In particular, we note that most operators in Table 1 are all pointwise, such that if the points z i have i.i.d. components, then the y will also have i.i.d. components. If we combine this with the constraint for the cost c to be additive over the components of x, y, we obtain the following simplification: Theorem 1. Suppose p x and p y have i.i.d components and c over DISPLAYFORM3 Consequently, the minimization problems (MP) and (KP) turn into d identical scalar problems for the distributions p X and p Y of the components of x and y: DISPLAYFORM4 such that an optimal transport map T for (MP-1-D) gives an optimal transport map f for (MP) by pointwise application of T, i.e. f (x) (i):= T (x (i) ), and an optimal transport plan p X,Y for DISPLAYFORM5 Proof. See Appendix. Fortunately, under some mild constraints, the scalar problems have a known solution: Theorem 2 (Theorem 2.9 in BID17). Let h: R → R + be convex and suppose the cost C takes the form C(x, y) = h(x − y). Given an continuous source distribution p X and a target distribution p Y on R having a finite optimal transport cost in (KP-1-D), then defines an optimal transport map from DISPLAYFORM6 DISPLAYFORM7 is the Cumulative Distribution Function (CDF) of X and F DISPLAYFORM8 ≥ y} is the pseudo-inverse of F Y. Furthermore, the joint distribution of (X, T mon X→Y (X)) defines an optimal transport plan for (KP-1-D).The mapping T mon X→Y (x) in Theorem 2 is non-decreasing and is known as the monotone transport map from X to Y. It is easy to verify that T mon X→Y (X) has the distribution of Y, in particular DISPLAYFORM9 Now, combining Theorems 1 and 2, we obtain a concrete realization of the Strategy 1 outlined above. We choose the cost c such that it admits to Theorem 1, such as c(x, y):= x − y 1, and use an operation that is pointwise, so we just need to compute the monotone transport map in. That is, if z has i.i.d components with distribution p Z, we just need to compute the component distribution p Y of the y of the operation, the CDFs F Z, F Y and obtain DISPLAYFORM10 as the component-wise modification of y, i.e.ỹ DISPLAYFORM11 In FIG2 we show the monotone transport map for the linear interpolation y = tz 1 + (1 − t)z 2 for various values of t. The detailed calculations and examples for various operations are given in Appendix 5.3, for both Uniform and Gaussian priors. The Gaussian case has a particularly simple ing transport map for additive operations, where it is just a linear transformation through a scalar multiplication, summarized in the third column of Table 1. To validate the correctness of the matched operators obtained above, we numerically simulate the distributions for toy examples, as well as prior distributions typically used in the literature. Priors vs. interpolations in 2-D For Figure 1, we sample 1 million pairs of points in two dimension, from a uniform prior (on [−1, 1] 2 ), and estimate numerically the midpoint distribution of linear interpolation, our proposed matched interpolation and the spherical interpolation of. It is reassuring to see that the matched interpolation gives midpoints which are identically distributed to the prior. In contrast, the linear interpolation condenses more towards the origin, forming a pyramid-shaped distribution (the of convolving two boxes in 2-d). Since the spherical interpolation of follows a great circle with varying radius between the two points, we see that the ing distribution has a "hole" in it, "circling" around the origin for both priors. distribution of the squared norm of the midpoints. We see there is a dramatic difference between vector lengths in the prior and the midpoints of linear interpolation, with only minimal overlap. We also show the spherical (SLERP) interpolation of which has a matching first moment, but otherwise also induces a distribution mismatch. In contrast, our matched interpolation, fully preserves the prior distribution and perfectly aligns. We note that this setting (d = 100, uniform or Gaussian) is commonly used in the literature. In this section we will present some concrete examples for the differences in generator output dependent on the exact sample operation used to traverse the latent space of a generative model. To this end, the generator output for latent samples produced with linear interpolation, SLERP (spherical linear interpolation) of and our proposed matched interpolation will be compared. Please refer to Table 1 for an overview of the operators used in this Section. Setup We used DCGAN BID12 generative models trained on LSUN bedrooms , CelebA BID7 and LLD BID15, an icon dataset, to qualitatively evaluate. For LSUN, the model was trained for two different output resolutions, providing 64 × 64 pixel and a 128×128 pixel output images (where the latter is used in figures containing larger sample images). The models for LSUN and the icon dataset where both trained on a uniform latent prior distribution, while for CelebA a Gaussian prior was used. The dimensionality of the latent space is 100 for both LSUN and CelebA, and 512 for the model trained on the icon model. Furthermore we use improved Wasserstein GAN (iWGAN) with gradient penalty BID4 ) trained on CIFAR-10 at 32 × 32 pixels with a 128-dimensional Gaussian prior to produce the inception scores presented in Section 3.3. We begin with the classic example of 2-point interpolation: FIG3 shows three examples per dataset for an interpolation between 2 points in latent space. Each example is first done via linear interpolation, then SLERP and finally matched interpolation. In FIG5 in the Appendix we show more densely sampled examples. FIG3 that linear interpolation produces inferior with generally more blurry, less saturated and less detailed output images. SLERP and matched interpolation are slightly different, however it is not visually obvious which one is superior. Differences between the various interpolation methods for CelebA FIG3 ) are much more subtle to the point that they are virtually indistinguishable when viewed side-by-side. This is not an inconsistency though: while distribution mismatch can cause large differences, it can also happen that the model generalizes well enough that it does not matter. In all cases, the point where the interpolation methods diverge the most, is at the midpoint of the interpolation where t = 0.5. Thus we provide 25 such interpolation midpoints in Figures 5 (LLD icons) and 6 (LSUN) for direct comparison. 3.69 ± 0.10 3.91 ± 0.10 2.04 ± 0.04 Table 2: Inception scores on LLD-icon, LSUN, CIFAR-10 and CelebA for the midpoints of various interpolation operations. Scores are reported as mean ± standard deviation (relative change in %). highlights the very apparent loss of detail and increasing prevalence of artifacts towards the midpoint in the linear version, compared to SLERP compared and our matched interpolation. Vicinity sampling Furthermore we provide two examples for vicinity sampling in Figures 9 and 10. Analogous to the previous observations, the output under a linear operator lacks definition, sharpness and saturation when compared to both spherical and matched operators. Random walk An interesting property of our matched vicinity sampling is that we can obtain a random walk in the latent space by applying it repeatedly: we start at a point y 0 = z drawn from the prior, and then obtain point y i by sampling a single point in the vicinity of y i−1, using some fixed'step size'.We show an example of such a walk in FIG8, using = 0.5. As a of the repeated application of the vicinity sampling operation, the divergence from the prior distribution in the non-matched case becomes stronger with each step, ing in completely unrecognizable output images on the LSUN and LLD icon models. Even for the CelebA model where differences where minimal before, they are quite apparent in this experiment. The random walk thus perfectly illustrates the need for respecting the prior distribution when performing any operation in latent space, as the adverse effects can cumulate through the repeated application of operators that do not comply to the prior distribution. We quantitatively confirm the observations of the previous section by using the Inception score BID16. In Table 2 we compare the Inception score of our trained models (i.e. using random samples from the prior) with the score when sampling midpoints from the 2-point and 4-point interpolations described above, reporting mean and standard deviation with 50,000 samples, as well as relative change to the original model scores if they are significant. Compared to the original scores of the trained models, our matched operations are statistically indistinguishable (as expected) while the linear interpolation gives a significantly lower score in all settings (up to 29% lower). As observed for the quality visually, the SLERP heuristic gives similar scores to the matched operations. We have shown that the common latent space operations used for Generative Models induce distribution mismatch from the prior distribution the models were trained for. This problem has been mostly ignored by the literature so far, partially due to the belief that this should not be a problem for uniform priors. However, our statistical and experimental analysis shows that the problem is real, with the operations used so far producing significantly lower quality samples compared to their inputs. To address the distribution mismatch, we propose to use optimal transport to minimally modify (in l 1 distance) the operations such that they fully preserve the prior distribution. We give analytical formulas of the ing (matched) operations for various examples, which are easily implemented. The matched operators give a significantly higher quality samples compared to the originals, having the potential to become standard tools for evaluating and exploring generative models. We note that the analysis here can bee seen as a more rigorous version of an observation made by , who experimentally show that there is a significant difference between the average norm of the midpoint of linear interpolation and the points of the prior, for uniform and Gaussian distributions. Suppose our latent space has a prior with DISPLAYFORM0 In this case, we can look at the squared norm DISPLAYFORM1 From the Central Limit Theorem (CLT), we know that as d → ∞, DISPLAYFORM2 in distribution. Thus, assuming d is large enough such that we are close to convergence, we can approximate the distribution of z 2 as N (dµ Z 2, dσ 2 Z 2). In particular, this implies that almost all points lie on a relatively thin spherical shell, since the mean grows as O(d) whereas the standard deviation grows only as O(DISPLAYFORM3 We note that this property is well known for i.i.d Gaussian entries (see e.g. Ex. 6.14 in MacKay FORMULA5). For Uniform distribution on the hypercube it is also well known that the mass is concentrated in the corner points (which is consistent with the claim here since the corner points lie on a sphere).Now consider an operator such as the midpoint of linear interpolation, y = DISPLAYFORM4 In this case, we can compute: DISPLAYFORM5 Thus, the distribution of y 2 can be approximated with N (DISPLAYFORM6 . Therefore, y also mostly lies on a spherical shell, but with a different radius than z. In fact, the shells will intersect at regions which have a vanishing probability for large d. In other words, when looking at the squared norm y 2, y 2 is a (strong) outlier with respect to the distribution of z 2. Proof. We will show it for the Kantorovich problem, the Monge version is similar. Starting from (KP), we compute DISPLAYFORM0 where the inequality in is due to each term being minimized separately. DISPLAYFORM1 where p X,Y has marginals p X and p Y. In this case P d (X, Y) is a subset of all joints p x,y with marginals p x and p y, where the pairs (DISPLAYFORM2) are constrained to be i.i.d. Starting again from can compute: DISPLAYFORM3 where the inequality in is due to minimizing over a smaller set. Since the two inequalities above are in the opposite direction, equality must hold for all of the expressions above, in particular: DISPLAYFORM4 Thus, (KP) and (KP-1-D) equal up to a constant, and minimizing one will minimize the other. Therefore the minimization of the former can be done over p X,Y with p x,y (x, y) = DISPLAYFORM5 In the next sections, we illustrate how to compute the matched operations for a few examples, in particular for linear interpolation and vicinity sampling, using a uniform or a Gaussian prior. We picked the examples where we can analytically compute the uniform transport map, but note that it is also easy to compute F DISPLAYFORM0 and (F Y (y)) numerically, since one only needs to estimate CDFs in one dimension. Since the components of all random variables in these examples are i.i.d, for such a random vector x we will implicitly write X for a scalar random variable that has the distribution of the components of x. When computing the monotone transport map T mon X→Y, the following Lemma is helpful. Lemma 1 (Theorem 2.5 in BID17). Suppose a mapping g(x) is non-decreasing and maps a continuous distribution p X to a distribution p Y, i.e. DISPLAYFORM1 then g is the monotone transport map T mon X→Y.According to Lemma 1, an alternative way of computing T mon X→Y is to find some g that is nondecreasing and transforms p X to p Y. Suppose z has uniform components Z ∼ Uniform(−1, 1). In this case, p Z (z) = 1/2 for −1 < z < 1. Now let y t = tz 1 + (1 − t)z 2 denote the linear interpolation between two points z 1, z 2, with component distribution p Yt. Due to symmetry we can assume that t > 1/2, since p Yt = p Y1−t. We then obtain p Yt as the convolution of p tZ and p (1−t)Z, i.e. p Yt = p tZ * p (1−t)Z. First we note that p tZ = 1/(2t) for −t < z < t and p (1−t)Z = 1/(2(1 − t)) for −(1 − t) < z < 1 − t. We can then compute: DISPLAYFORM0 if − t + (1 − t) < y < t − (1 − t) −y + 1 if t − (1 − t) < y < 1 0 if 1 < yThe CDF F Yt is then obtained by computing Since p Z (z) = 1/2 for |z| < 1, we have F Z (z) =. For vicinity sampling, we want to obtain new points z 1, ·, z k which are close to z. We thus define DISPLAYFORM1 where u i also has uniform components, such that each coordinate of z i differs at most by from z. By identifying tZ i = tZ + (1 − t)U i with t = 1/(1 +), we see that tZ i has identical distribution to the linear interpolation Y t in the previous example. Thus g t (Z i):= T mon Yt→Z (tZ i) will have the distribution of Z, and by Lemma1 is then the monotone transport map from Z i to Z. Suppose z has components Z ∼ N (0, σ 2). In this case, we can compute linear interpolation as before, y t = tz 1 + (1 − t)z 2. Since the sum of Gaussians is Gaussian, we get, Y t ∼ N (0, t 2 σ 2 + (1 − t) 2 σ 2 ). Now, it is easy to see that with a proper scaling factor, we can adjust the variance of Y t back to σ 2. That is, By adjusting the vicinity sampling operation to DISPLAYFORM0 where e i ∼ N, we can similarly find the monotone transport map g (y) = 1 √ 1+ 2 y. Another operation which has been used in the literature is the "analogy", where from samples z 1, z 2, z 3, one wants to apply the difference between z 1 and z 2, to z 3. The transport map is then g(y) =

Represent the TLDR for finding the corresponding scientific article.

Operations in the GAN latent space can induce a distribution mismatch compared to the training distribution, and we address this using optimal transport to match the distributions.

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.