Question: Represent the article for finding a single-sentence TLDR!

Question: Many real-world sequential decision-making problems can be formulated as optimal control with high-dimensional observations and unknown dynamics. A promising approach is to embed the high-dimensional observations into a lower-dimensional latent representation space, estimate the latent dynamics model, then utilize this model for control in the latent space. An important open question is how to learn a representation that is amenable to existing control algorithms? In this paper, we focus on learning representations for locally-linear control algorithms, such as iterative LQR (iLQR). By formulating and analyzing the representation learning problem from an optimal control perspective, we establish three underlying principles that the learned representation should comprise: 1) accurate prediction in the observation space, 2) consistency between latent and observation space dynamics, and 3) low curvature in the latent space transitions. These principles naturally correspond to a loss function that consists of three terms: prediction, consistency, and curvature (PCC). Crucially, to make PCC tractable, we derive an amortized variational bound for the PCC loss function. Extensive experiments on benchmark domains demonstrate that the new variational-PCC learning algorithm benefits from significantly more stable and reproducible training, and leads to superior control performance. Further ablation studies give support to the importance of all three PCC components for learning a good latent space for control. Decomposing the problem of decision-making in an unknown environment into estimating dynamics followed by planning provides a powerful framework for building intelligent agents. This decomposition confers several notable benefits. First, it enables the handling of sparse-reward environments by leveraging the dense signal of dynamics prediction. Second, once a dynamics model is learned, it can be shared across multiple tasks within the same environment. While the merits of this decomposition have been demonstrated in low-dimensional environments , scaling these methods to high-dimensional environments remains an open challenge. The recent advancements in generative models have enabled the successful dynamics estimation of high-dimensional decision processes (; ;). This procedure of learning dynamics can then be used in conjunction with a plethora of decision-making techniques, ranging from optimal control to reinforcement learning (RL) (; ; ; ; ; ; ;). One particularly promising line of work in this area focuses on learning the dynamics and conducting control in a low-dimensional latent embedding of the observation space, where the embedding itself is learned through this process (; ; ;). We refer to this approach as learning controllable embedding (LCE). There have been two main approaches to this problem: 1) to start by defining a cost function in the high-dimensional observation space and learn the embedding space, its dynamics, and reward function, by interacting with the environment in a RL fashion , and 2) to first learn the embedding space and its dynamics, and then define a cost function in this low-dimensional space and conduct the control . This can be later combined with RL for extra fine-tuning of the model and control. In this paper, we take the second approach and particularly focus on the important question of what desirable traits should the latent embedding exhibit for it to be amenable to a specific class of control/learning algorithms, namely the widely used class of locally-linear control (LLC) algorithms? We argue from an optimal control standpoint that our latent space should exhibit three properties. The first is prediction: given the ability to encode to and decode from the latent space, we expect the process of encoding, transitioning via the latent dynamics, and then decoding, to adhere to the true observation dynamics. The second is consistency: given the ability to encode a observation trajectory sampled from the true environment, we expect the latent dynamics to be consistent with the encoded trajectory. Finally, curvature: in order to learn a latent space that is specifically amenable to LLC algorithms, we expect the (learned) latent dynamics to exhibit low curvature in order to minimize the approximation error of its first-order Taylor expansion employed by LLC algorithms. Our contributions are thus as follows: We propose the Prediction, Consistency, and Curvature (PCC) framework for learning a latent space that is amenable to LLC algorithms and show that the elements of PCC arise systematically from bounding the suboptimality of the solution of the LLC algorithm in the latent space. We design a latent variable model that adheres to the PCC framework and derive a tractable variational bound for training the model. To the best of our knowledge, our proposed curvature loss for the transition dynamics (in the latent space) is novel. We also propose a direct amortization of the Jacobian calculation in the curvature loss to help training with curvature loss more efficiently. Through extensive experimental comparison, we show that the PCC model consistently outperforms E2C and RCE on a number of control-from-images tasks, and verify via ablation, the importance of regularizing the model to have consistency and low-curvature. We are interested in controlling the non-linear dynamical systems of the form s t+1 = f S (s t, u t) + w, over the horizon T. In this definition, s t ∈ S ⊆ R ns and u t ∈ U ⊆ R nu are the state and action of the system at time step t ∈ {0, . . ., T − 1}, w is the Gaussian system noise, and f S is a smooth non-linear system dynamics. We are particularly interested in the scenario in which we only have access to the high-dimensional observation x t ∈ X ⊆ R nx of each state s t (n x n s). This scenario has application in many real-world problems, such as visual-servoing , in which we only observe high-dimensional images of the environment and not its underlying state. We further assume that the high-dimensional observations x have been selected such that for any arbitrary control sequence U = {u t} T −1 t=0, the observation sequence {x t} T t=0 is generated by a stationary Markov process, i.e., x t+1 ∼ P (·|x t, u t), ∀t ∈ {0, . . ., T − 1}. A common approach to control the above dynamical system is to solve the following stochastic optimal control (SOC) problem that minimizes expected cumulative cost: where c t: X × U → R ≥0 is the immediate cost function at time t, c T ∈ R ≥0 is the terminal cost, and x 0 is the observation at the initial state s 0. Note that all immediate costs are defined in the observation space X, and are bounded by c max > 0 and Lipschitz with constant c lip > 0. For example, in visualservoing, (SOC1) can be formulated as a goal tracking problem , where we control the robot to reach the goal observation x goal, and the objective is to compute a sequence of optimal open-loop actions U that minimizes the cumulative tracking error Since the observations x are high dimensional and the dynamics in the observation space P (·|x t, u t) is unknown, solving (SOC1) is often intractable. To address this issue, a class of algorithms has been recently developed that is based on learning a low-dimensional latent (embedding) space Z ⊆ R nz (n z n x) and latent state dynamics, and performing optimal control there. This class that we refer to as learning controllable embedding (LCE) throughout the paper, include recently developed algorithms, such as E2C , RCE , and SOLAR . The main idea behind the LCE approach is to learn a triplet, (i) an encoder E: X → P(Z); (ii) a dynamics in the latent space F: Z × U → P(Z); and (iii) a decoder D: Z → P(X). These in turn can be thought of as defining a (stochastic) mapping P: X × U → P(X) of the form P = D • F • E. We then wish to solve the SOC in latent space Z: such that the solution of (SOC2), U 3 PCC MODEL: A CONTROL PERSPECTIVE As described in Section 2, we are primarily interested in solving (SOC1), whose states evolve under dynamics P, as shown at the bottom row of Figure 1 (a) in (blue). However, because of the difficulties in solving (SOC1), mainly due to the high dimension of observations x, LCE proposes to learn a mapping P by solving (SOC2) that consists of a loss function, whose states evolve under dynamics F (after an initial transition by encoder E), as depicted in Figure 1 (b), and a regularization term. The role of the regularizer R 2 is to account for the performance gap between (SOC1) and the loss function of (SOC2), due to the discrepancy between their evolution paths, shown in Figures 1(a)(blue) and 1(b)(green). The goal of LCE is to learn P of the particular form P = D • F • E, described in Section 2, such that the solution of (SOC2) has similar performance to that of (SOC1). In this section, we propose a principled way to select the regularizer R 2 to achieve this goal. Since the exact form of (SOC2) has a direct effect on learning P, designing this regularization term, in turn, provides us with a recipe (loss function) to learn the latent (embedded) space Z. In the following subsections, we show that this loss function consists of three terms that correspond to prediction, consistency, and curvature, the three ingredients of our PCC model. Note that these two SOCs evolve in two different spaces, one in the observation space X under dynamics P, and the other one in the latent space Z (after an initial transition from X to Z) under dynamics F. Unlike P and F that only operate in a single space, X and Z, respectively, P can govern the evolution of the system in both X and Z (see Figure 1 (c)). Therefore, any recipe to learn P, and as a the latent space Z, should have at least two terms, to guarantee that the evolution paths ed from P in X and Z are consistent with those generated by P and F. We derive these two terms, that are the prediction and consistency terms in the loss function used by our PCC model, in Sections 3.1 and 3.2, respectively. While these two terms are the of learning P in general SOC problems, in Section 3.3, we concentrate on the particular class of LLC algorithms (e.g., iLQR ) to solve SOC, and add the third term, curvature, to our recipe for learning P. Figures 1(a)(blue) and 1(c)(red) show the transition in the observation space under P and P, where x t is the current observation, and x t+1 andx t+1 are the next observations under these two dynamics, respectively. Instead of learning a P with minimum mismatch with P in terms of some distribution norm, we propose to learn P by solving the following SOC: whose loss function is the same as the one in (SOC1), with the true dynamics replaced by P. In Lemma 1 (see Appendix A.1, for proof), we show how to set the regularization term R 3 in (SOC3), such that the control sequence ed from solving (SOC3), U In Section 3.1, we provided a recipe for learning P (in form of D • F • E) by introducing an intermediate (SOC3) that evolves in the observation space X according to dynamics P. In this section we first connect (SOC2) that operates in Z with (SOC3) that operates in X. For simplicity and without loss generality, assume the initial cost c 0 (x, u) is zero. 4 Lemma 2 (see Appendix A.2, for proof) suggests how we shall set the regularizer in (SOC2), such that its solution performs similarly to that of (SOC3), under their corresponding dynamics models. Lemma 2. Let (U * 3, P * 3) be a solution to (SOC3) and (U * 2, P * 2) be a solution to (SOC2) with and Similar to Lemma 1, in Eq. 2, the expectation is over the state-action stationary distribution of the policy used to generate the training samples. Moreover, are the probability over the next latent state z, given the current observation x and action u, in (SOC2) and (SOC3) (see the paths x t → z t →z t+1 and x t → z t →z t+1 →x t+1 →ẑ t+1 in Figures 1(b)(green) and 1(c)(red)). Therefore R 2 (P) can be interpreted as the measure of discrepancy between these models, which we term as consistency loss. Although Lemma 2 provides a recipe to learn P by solving (SOC2) with the regularizer, unfortunately this regularizer cannot be computed from the data -that is of the form (x t, u t, x t+1) -because the first term in the D KL requires marginalizing over current and next latent states (z t andz t+1 in Figure 1 (c)). To address this issue, we propose to use the (computable) regularizer in which the expectation is over (x, u, x) sampled from the training data. Corollary 1 (see Appendix A.3, for proof) bounds the performance loss ed from using R 2 (P) instead of R 2 (P), and shows that it could be still a reasonable choice. Corollary 1. Let (U * 3, P * 3) be a solution to (SOC3) and (U * 2, P * 2) be a solution to (SOC2) with R 2 (P) and and λ 2 defined by and. Then, we have L(U * where R 3 ( P) and R 2 (P) are defined by and. Then, we have 3.3 LOCALLY-LINEAR CONTROL IN THE LATENT SPACE AND CURVATURE REGULARIZATION In Sections 3.1 and 3.2, we derived a loss function to learn the latent space Z. This loss function, that was motivated by the general SOC perspective, consists of two terms to enforce the latent space to not only predict the next observations accurately, but to be suitable for control. In this section, we focus on the class of locally-linear control (LLC) algorithms (e.g., iLQR), for solving (SOC2), and show how this choice adds a third term, that corresponds to curvature, to the regularizer of (SOC2), and as a , to the loss function of our PCC model. The main idea in LLC algorithms is to iteratively compute an action sequence to improve the current trajectory, by linearizing the dynamics around this trajectory, and use this action sequence to generate the next trajectory (see Appendix B for more details about LLC and iLQR). This procedure implicitly assumes that the dynamics is approximately locally linear. To ensure this in (SOC2), we further restrict the dynamics P and assume that it is not only of the form P = D • F • E, but F, the latent space dynamics, has low curvature. One way to ensure this in (SOC2) is to directly impose a penalty over the curvature of the latent space transition function where w is a Gaussian noise. Consider the following SOC problem: where R 2 is defined by; U is optimized by a LLC algorithm, such as iLQR; R LLC (P) is given by, where = (z, u) ∼ N (0, δ 2 I), δ > 0 is a tunable parameter that characterizes the "diameter" of latent state-action space in which the latent dynamics model has low curvature., where 1/X is the minimum non-zero measure of the sample distribution w.r.t. X, and 1 − η ∈ is a probability threshold. Lemma 4 (see Appendix A.5, for proof and discussions on how δ affects LLC performance) shows that a solution of (SOC-LLC) has similar performance to a solution of (SOC1, and thus, (SOC-LLC) is a reasonable optimization problem to learn P, and also the latent space Z. Lemma 4. Let (U * LLC, P * LLC) be a LLC solution to (SOC-LLC) and U * 1 be a solution to (SOC1). Suppose the nominal latent state-action trajectory {(z z z t, u u u t)} T −1 t=0 satisfies the condition: t=0 is the optimal trajectory of (SOC2). Then with proba- In practice, instead of solving (SOC-LLC) jointly for U and P, we treat (SOC-LLC) as a bi-level optimization problem, first, solve the inner optimization problem for P, i.e., where R 3 (P) = −E x,u,x [log P (x |x, u)] is the negative log-likelihood, 5 and then, solve the outer optimization problem, min U L(U, F *,c, z 0), where P * = D * • F * • E *, to obtain the optimal control sequence U *. Solving (SOC-LLC) this way is an approximation, in general, but is justified, when the regularization parameter λ LLC is large. Note that we leave the regularization parameters (λ p, λ c, λ cur) as hyper-parameters of our algorithm, and do not use those derived in the lemmas of this section. Since the loss for learning P * in (PCC-LOSS) enforces (i) prediction accuracy, (ii) consistency in latent state prediction, and (iii) low curvature over f Z, through the regularizers R 3, R 2, and R LLC, respectively, we refer to it as the prediction-consistency-curvature (PCC) loss. The PCC-Model objective in (PCC-LOSS) introduces the optimization problem min P λ p R 3 (P) + λ c R 2 (P) + λ cur R LLC (P). To instantiate this model in practice, we de- In this section, we propose a variational approximation to the intractable negative log-likelihood R 3 and batch-consistency R 2 losses, and an efficient approximation of the curvature loss R LLC. The negative log-likelihood 6 R 3 admits a variational bound via Jensen's Inequality, which holds for any choice of recognition model Q. For simplicity, we assume the recognition model employs bottom-up inference and thus factorizes as Q(z t,ẑ t+1 |x t, x t+1, u t) = Q(ẑ t+1 |x t+1)Q(z t |ẑ t+1, x t, u t). The main idea behind choosing a backward-facing model is to allow the model to learn to account for noise in the underlying dynamics. We estimate the expectations in via Monte Carlo simulation. To reduce the variance of the estimator, we decompose R 3,NLE-Bound further into log P (ẑt+1 | zt, ut), and note that the Entropy H(·) and Kullback-Leibler D KL (· ·) terms are analytically tractable when Q is restricted to a suitably chosen variational family (i.e. in our experiments, Q(ẑ t+1 | x t+1) and Q(z t |ẑ t+1, x t, u t) are factorized Gaussians). The derivation is provided in Appendix C.1. Interestingly, the consistency loss R 2 admits a similar treatment. We note that the consistency loss seeks to match the distribution ofẑ t+1 | x t, u t with z t+1 | x t+1, which we represent below as Here, P (ẑ t+1 | x t, u t) is intractable due to the marginalization of z t. We employ the same procedure as in to construct a tractable variational bound We now make the further simplifying assumption that ). This allows us to rewrite the expression as which is a subset of the terms in. See Appendix C.2 for a detailed derivation. In practice we use a variant of the curvature loss where Taylor expansions and gradients are evaluated When n z is large, evaluation and differentiating through the Jacobians can be slow. To circumvent this issue, the Jacobians evaluation can be amortized by treating the Jacobians as the coefficients of the best linear approximation at the evaluation point. This leads to a new amortized curvature loss where A and B are function approximators to be optimized. Intuitively, the amortized curvature loss seeks-for any given (z, u)-to find the best choice of linear approximation induced by A(z, u) and B(z, u) such that the behavior of F µ in the neighborhood of (z, u) is approximately linear. In this section, we highlight the key differences between PCC and the closest previous works, namely E2C and RCE. A key distinguishing factor is PCC's use of a nonlinear latent dynamics model paired with an explicit curvature loss. In comparison, E2C and RCE both employed "locally-linear dynamics" of the form z = A(z,ū)z + B(z,ū)u + c(z,ū) wherez andū are auxiliary random variables meant to be perturbations of z and u. When contrasted with, it is clear that neither A and B in the E2C/RCE formulation can be treated as the Jacobians of the dynamics, and hence the curvature of the dynamics is not being controlled explicitly. Furthermore, since the locally-linear dynamics are wrapped inside the maximum-likelihood estimation, both E2C and RCE conflate the two key elements prediction and curvature together. This makes controlling the stability of training much more difficult. Not only does PCC explicitly separate these two components, we are also the first to explicitly demonstrate theoretically and empirically that the curvature loss is important for iLQR. Furthermore, RCE does not incorporate PCC's consistency loss. Note that PCC, RCE, and E2C are all Markovian encoder-transition-decoder frameworks. Under such a framework, the sole reliance on minimizing the prediction loss will in a discrepancy between how the model is trained (maximizing the likelihood induced by encoding-transitioning-decoding) versus how it is used at test-time for control (continual transitioning in the latent space without ever decoding). By explicitly minimizing the consistency loss, PCC reduces the discrapancy between how the model is trained versus how it is used at test-time for planning. Interestingly, E2C does include a regularization term that is akin to PCC's consistency loss. However, as noted by the authors of RCE, E2C's maximization of pair-marginal log-likelihoods of (x t, x t+1) as opposed to the conditional likelihood of x t+1 given x t means that E2C does not properly minimize the prediction loss prescribed by the PCC framework. In this section, we compare the performance of To generate our training and test sets, each consists of triples (x t, u t, x t+1), we: sample an underlying state s t and generate its corresponding observation x t, sample an action u t, and obtain the next state s t+1 according to the state transition dynamics, add it a zero-mean Gaussian noise with variance σ 2 I ns, and generate corresponding observation x t+1.To ensure that the observation-action data is uniformly distributed (see Section 3), we sample the state-action pair (s t, u t) uniformly from the state-action space. To understand the robustness of each model, we consider both deterministic (σ = 0) and stochastic scenarios. In the stochastic case, we add noise to the system with different values of σ and evaluate the models' performance under various degree of noise. Each task has underlying start and goal states that are unobservable to the algorithms, instead, the algorithms have access to the corresponding start and goal observations. We apply control using the iLQR algorithm (see Appendix B), with the same cost function that was used by RCE and E2C, namely,c(z where z goal is obtained by encoding the goal observation, and Q = κ · I nz, R = I nu . Details of our implementations are specified in Appendix D.3. We report performance in the underlying system, specifically the percentage of time spent in the goal region 10 . A Reproducible Experimental Pipeline In order to measure performance reproducibility, we perform the following 2-step pipeline. For each control task and algorithm, we train 10 models independently, and solve 10 control tasks per model (we do not cherry-pick, but instead perform a total of 10 × 10 = 100 control tasks). We report statistics averaged over all the tasks (in addition, we report the best performing model averaged over its 10 tasks). By adopting a principled and statistically reliable evaluation pipeline, we also address a pitfall of the compared baselines where the best model needs to be cherry picked, and training variance was not reported. 7 Code will become available with the camera-ready version. 8 For the RCE implementation, we directly optimize the ELBO loss in Equation of the paper. We also tried the approach reported in the paper on increasing the weights of the two middle terms and then annealing them to 1. However, in practice this method is sensitive to annealing schedule and has convergence issues. 9 See a control demo on the TORCS simulator at https://youtu.be/GBrgALRZ2fw 10 Another possible metric is the average distance to goal, which has a similar behavior. 18.8 ± 2.1 9.1 ± 1.5 13.1 ± 1.9 11.5 ± 1.8 Results Table 1 shows how PCC outperforms the baseline algorithms in the noiseless dynamics case by comparing means and standard deviations of the means on the different control tasks (for the case of added noise to the dynamics, which exhibits similar behavior, refer to Appendix E.1). It is important to note that for each algorithm, the performance metric averaged over all models is drastically different than that of the best model, which justifies our rationale behind using the reproducible evaluation pipeline and avoid cherry-picking when reporting. Figure 2 depicts 2 instances (randomly chosen from the 10 trained models) of the learned latent space representations on the noiseless dynamics of Planar and Inverted Pendulum tasks for PCC, RCE, and E2C models (additional representations can be found in Appendix E.2). Representations were generated by encoding observations corresponding to a uniform grid over the state space. Generally, PCC has a more interpretable representation of both Planar and Inverted Pendulum Systems than other baselines for both the noiseless dynamics case and the noisy case. Finally, in terms of computation, PCC demonstrates faster training with 64% improvement over RCE, and 2% improvement over E2C. Ablation Analysis On top of comparing the performance of PCC to the baselines, in order to understand the importance of each component in (PCC-LOSS), we also perform an ablation analysis on the consistency loss (with/without consistency loss) and the curvature loss (with/without curvature loss, and with/without amortization of the Jacobian terms). Table 2 shows the ablation analysis of PCC on the aforementioned tasks. From the numerical , one can clearly see that when consistency loss is omitted, the control performance degrades. This corroborates with the theoretical in Section 3.2, which indicates the relationship of the consistency loss and the estimation error between the next-latent dynamics prediction and the next-latent encoding. This further implies that as the consistency term vanishes, the gap between control objective function and the model training loss is widened, due to the accumulation of state estimation error. The control performance also decreases when one removes the curvature loss. This is mainly attributed to the error between the iLQR control algorithm and (SOC2). Although the latent state dynamics model is parameterized with neural networks, which are smooth, without enforcing the curvature loss term the norm of the Hessian (curvature) might still be high. This also confirms with the analysis in Section 3.3 about sub-optimality performance and curvature of latent dynamics. Finally, we observe that the performance of models trained without amortized curvature loss are slightly better than with their amortized counterpart, however, since the amortized curvature loss does not require computing gradient of the latent dynamics (which means that in stochastic optimization one does not need to estimate its Hessian), we observe relative speed-ups in model training with the amortized version (speed-up of 6%, 9%, and 15% for Planar System, Inverted Pendulum, and Cartpole, respectively). In this paper, we argue from first principles that learning a latent representation for control should be guided by good prediction in the observation space and consistency between latent transition and the embedded observations. Furthermore, if variants of iterative LQR are used as the controller, the low-curvature dynamics is desirable. All three elements of our PCC models are critical to the stability of model training and the performance of the in-latent-space controller. We hypothesize that each particular choice of controller will exert different requirement for the learned dynamics. A future direction is to identify and investigate the additional bias for learning an effective embedding and latent dynamics for other type of model-based control and planning methods. where D TV is the total variation distance of two distributions. The first inequality is based on the of the above lemma, the second inequality is based on Pinsker's inequality , and the third inequality is based on Jensen's inequality of (·) function. Now consider the expected cumulative KL cost: t=0 KL(P (·|x t, u t)|| P (·|x t, u t)) | P, x 0 with respect to some arbitrary control action sequence {u t} T −1 t=0. Notice that this arbitrary action sequence can always be expressed in form of deterministic policy u t = π (x t, t) with some nonstationary state-action mapping π. Therefore, this KL cost can be written as: where the expectation is taken over the state-action occupation measure t=0 P(x t = x, u t = u|x 0, U) of the finite-horizon problem that is induced by data-sampling policy U. The last inequality is due to change of measures in policy, and the last inequality is due to the facts that (i) π is a deterministic policy, (ii) dU (u t) is a sampling policy with lebesgue measure 1/U over all control actions, (iii) the following bounds for importance sampling factor holds: To conclude the first part of the proof, combining all the above arguments we have the following inequality for any model P and control sequence U: For the second part of the proof, consider the solution of (SOC3), namely (U * 3, P * 3). Using the optimality condition of this problem one obtains the following inequality: Using the in and, one can then show the following chain of inequalities: where U * 1 is the optimizer of (SOC1) and (U * 3, P * 3) is the optimizer of (SOC3). Therefore by letting λ 3 = √ 2T 2 · c max U and R 3 (P) = E x,u KL(P (·|x, u)|| P (·|x, u)) and by combining all of the above arguments, the proof of the above lemma is completed. A.2 PROOF OF LEMMA 2 For the first part of the proof, at any time-step t ≥ 1, for any arbitrary control action sequence {u t} T −1 t=0, and any model P, consider the following decomposition of the expected cost:. Now consider the following cost function: E[c(x t−1, u t−1) + c(x t, u t) | P, x 0 ] for t > 2. Using the above arguments, one can express this cost as By continuing the above expansion, one can show that where the last inequality is based on Jensen's inequality of (·) function. For the second part of the proof, following similar arguments as in the second part of the proof of Lemma 1, one can show the following chain of inequalities for solution of (SOC3) and (SOC2): where the first and third inequalities are based on the first part of this Lemma, and the second inequality is based on the optimality condition of problem (SOC2). This completes the proof. To start with, the total-variation distance D TV x ∈X d P (x |x, u)E(·|x)||(F • E)(·|x, u) can be bounded by the following inequality using triangle inequality: where the second inequality follows from the convexity property of the D TV -norm (w.r.t. convex weights E(·|x), ∀x ). Then by Pinsker's inequality, one obtains the following inequality: We now analyze the batch consistency regularizer: and connect it with the inequality in. Using Jensen's inequality of convex function x log x, for any observation-action pair (x, u) sampled from U τ, one can show that Therefore, for any observation-control pair (x, u) the following inequality holds: By taking expectation over (x, u) one can show that is the lower bound of the batch consistency regularizer. Therefore, the above arguments imply that The inequality is based on the property that Equipped with the above additional , the rest of the proof on the performance bound follows directly from the from Lemma 2, in which here we further upper-bound A.4 PROOF OF LEMMA 3 For the first part of the proof, at any time-step t ≥ 1, for any arbitrary control action sequence {u t} T −1 t=0 and for any model P, consider the following decomposition of the expected cost: Under review as a conference paper at ICLR 2020 Now consider the following cost function: E[c(x t−1, u t−1) + c(x t, u t) | P, x 0 ] for t > 2. Using the above arguments, one can express this cost as Continuing the above expansion, one can show that where the last inequality is based on the fact that and is based on Jensen's inequality of (·) function. For the second part of the proof, following similar arguments from Lemma 2, one can show the following inequality for the solution of (SOC3) and (SOC2): where the first and third inequalities are based on the first part of this Lemma, and the second inequality is based on the optimality condition of problem (SOC2). This completes the proof. A Recap of the Result: Let (U * LLC, P * LLC) be a LLC solution to (SOC-LLC) and U * 1 be a solution to (SOC1). Suppose the nominal latent state-action pair {(z z z t, u u u t)} T −1 t=0 satisfies the condition: (z z z t, u u u t) ∼ N ((z * 2,t, u * 2,t), δ 2 I), where {(z * 2,t, u * 2,t} T −1 t=0 is the optimal trajectory of problem (SOC2). Then with probability 1 − η, we have L(U * Discussions of the effect of δ on LLC Performance: The of this lemma shows that when the nominal state and actions are δ-close to the optimal trajectory of (SOC2), i.e., at each time step (z z z t, u u u t) is a sample from the Gaussian distribution centered at (z * 2,t, u * 2,t) with standard deviation δ, then one can obtain a performance bound of LLC algorithm that is in terms of the regularization loss R LLC. To quantify the above condition, one can use Mahalanobis distance to measure the distance of (z z z t, u u u t) to distribution N ((z * 2,t, u * 2,t), δ 2 I), i.e., we want to check for the condition: for any arbitrary error tolerance > 0. While we cannot verify the condition without knowing the optimal trajectory {(z * 2,t, u * 2,t)} T −1 t=0, the above condition still offers some insights in choosing the parameter δ based on the trade-off of designing nominal trajectory {(z z z t, u u u t)} T −1 t=0 and optimizing R LLC. When δ is large, the low-curvature regularization imposed by the R LLC regularizer will cover a large portion of the state-action space. In the extreme case when δ → ∞, R LLC can be viewed as a regularizer that enforces global linearity. Here the trade-off is that the loss R LLC is generally higher, which in turn degrades the performance bound of the LLC control algorithm in Lemma 4. On the other hand, when δ is small the low-curvature regularization in R LLC only covers a smaller region of the latent state-action space, and thus the loss associated with this term is generally lower (which provides a tighter performance bound in Lemma 4). However the performance will only hold when (z z z t, u u u t) happens to be close to (z * 2,t, u * 2,t) at each time-step t ∈ {0, . . ., T − 1}. Proof: For simplicity, we will focus on analyzing the noiseless case when the dynamics is deterministic (i.e., Σ w = 0). Extending the following analysis for the case of non-deterministic dynamics should be straight-forward. First, consider any arbitrary latent state-action pair (z, u), such that the corresponding nominal state-action pair (z z z, u u u) is constructed by z z z = z − δz, u u u = u − δu, where (δz, δu) is sampled from the Gaussian distribution N (0, δ 2 I). (The random vectors are denoted as (δz, δu)) By the two-tailed Bernstein's inequality , for any arbitrarily given η ∈ one has the following inequality with probability 1 − η: The second inequality is due to the basic fact that variance is less than second-order moment of a random variable. On the other hand, at each time step t ∈ {0, . . ., T − 1} by the Lipschitz property of the immediate cost, the value function V t (z) = min U t: is also Lipchitz with constant (T − t + 1)c lip. Using the Lipschitz property of V t+1, for any (z, u) and (δz, δu), such that (z z z, u u u) = (z − δz, u − δu), one has the following property: Therefore, at any arbitrary state-action pair (z,ũ), for z z z = z − δz, and u u u =ũ − δu with Gaussian sample (δz, δu) ∼ N (0, δ 2 I), the following inequality on the value function holds w.p. 1 − η: which further implies Now letũ * be the optimal control w.r.t. Bellman operator T t [V t+1](z) at any latent statez. Based on the assumption of this lemma, at each statez the nominal latent state-action pair (z z z, u u u) is generated by perturbing (z,ũ *) with Gaussian sample (δz, δu) ∼ N (0, δ 2 I) that is in form of z z z =z − δz, u u u =ũ − δu. Then by the above arguments the following chain of inequalities holds w.p. 1 − η: Recall the LLC loss function is given by Also consider the Bellman operator w.r.t. latent SOC:, and the Bellman operator w.r.t. LLC:. Utilizing these definitions, the inequality in can be further expressed as This inequality is due to the fact that all latent states are generated by the encoding observations, i.e., z ∼ E(·|x), and thus by following analogous arguments as in the proof of Lemma 1, one has Therefore, based on the dynamic programming that bounds the difference of value function w.r.t. different Bellman operators in finite-horizon problems (for example see Theorem 1.3 in), the above inequality implies the following bound in the value function, w.p. 1 − η: Notice that here we replace η in the in with η/T. In order to prove, we utilize for each t ∈ {0, . . ., T − 1}, and this replacement is the of applying the Union Probability bound (to ensure holds with probability 1 − η). Therefore the proof is completed by combining the above with that in Lemma 3. We follow the same control scheme as in. Namely, we use the iLQR solver to plan in the latent space. Given a start observation x start and a goal observation x goal, corresponding to underlying states {s start, s goal}, we encode the observations to retrieve z start and z goal. Then, the procedure goes as follows: we initialize a random trajectory (sequence of actions), feed it to the iLQR solver and apply the first action from the trajectory the solver outputs. We observe the next observation returned from the system (closed-loop control), and feed the updated trajectory to the iLQR solver. This procedure continues until the it reaches the end of the problem horizon. We use a receding window approach, where at every planning step the solver only optimizes for a fixed length of actions sequence, independent of the problem horizon. Consider the latent state SOC problem At each time instance t ∈ {0, . . ., T} the value function of this problem is given by Recall that the nonlinear latent space dynamics model is given by: where F µ (z t, u t) is the deterministic dynamics model and F σ F σ is the covariance of the latent dynamics system noise. Notice that the deterministic dynamics model F µ (z t, u t) is smooth, and therefore the following Jacobian terms are well-posed: By the Bellman's principle of optimality, at each time instance t ∈ {0, . . ., T − 1} the value function is a solution of the recursive fixed point equation where the state-action value function at time-instance t w.r.t. state-action pair (z t, u t) = (z, u) is given by In the setting of the iLQR algorithm, assume we have access to a trajectory of latent states and actions that is in form of {(z z z t, u u u t, z z z t+1)} T −1 t=0. At each iteration, the iLQR algorithm has the following steps: 1. Given a nominal trajectory, find an optimal policy w.r.t. the perturbed latent states 2. Generate a sequence of optimal perturbed actions that locally improves the cumulative cost of the given trajectory 3. Apply the above sequence of actions to the environment and update the nominal trajectory 4. Repeat the above steps with new nominal trajectory Denote by δz t = z t − z z z t and δu t = u t − u u u t the deviations of state and control action at time step t respectively. Assuming that the nominal next state z z z t+1 is generated by the deterministic transition F µ (z z z t, u u u t) at the nominal state and action pair (z z z t, u u u t), the first-order Taylor series approximation of the latent space transition is given by To find a locally optimal control action sequence u * t = π * δz,t (δz t) + u u u t, ∀t, that improves the cumulative cost of the trajectory, we compute the locally optimal perturbed policy (policy w.r.t. perturbed latent state) {π * δz,t (δz t)} T −1 t=0 that minimizes the following second-order Taylor series approximation of Q t around nominal state-action pair (z z z t, u u u t), ∀t ∈ {0, . . ., T − 1}: where the first and second order derivatives of the Q−function are given by and the first and second order derivatives of the value functions are given by Notice that the Q-function approximation Q t in is quadratic and the matrix is positive semi-definite. Therefore the optimal perturbed policy π * δz,t has the following closed-form solution: where the controller weights are given by Furthermore, by putting the optimal solution into the Taylor expansion of the Q-function Q t, we get where the closed-loop first and second order approximations of the Q-function are given by. Notice that at time step t the optimal value function also has the following form: Therefore, the first and second order differential value functions can be V t,z (z z z t, u u u t) = Q * t,21 (z z z t, u u u t), V t,zz (z z z t, u u u t) = Q * t,22 (z z z t, u u u t), and the value improvement at the nominal state z z z t at time step t is given by While iLQR provides an effective way of computing a sequence of (locally) optimal actions, it has two limitations. First, unlike RL in which an optimal Markov policy is computed, this algorithm only finds a sequence of open-loop optimal control actions under a given initial observation. Second, the iLQR algorithm requires the knowledge of a nominal (latent state and action) trajectory at every iteration, which restricts its application to cases only when real-time interactions with environment are possible. In order to extend the iLQR paradigm into the closed-loop RL setting, we utilize the concept of model predictive control (MPC) and propose the following iLQR-MPC procedure. Initially, given an initial latent state z 0 we generate a single nominal trajectory: {(z z z t, u u u t, z z z t+1)} We derive the bound for the conditional log-likelihood log P (x t+1 |x t, u t). log P (x t+1 |x t, u t) = log zt,ẑt+1 Where (a) holds from the log function concavity, (b) holds by the factorization Q(z t,ẑ t+1 |x t, x t+1, u t) = Q(ẑ t+1 |x t+1)Q(z t |ẑ t+1, x t, u t), and (c) holds by a simple decomposition to the different components. We derive the bound for the consistency loss Consistency (P). Where (a) holds by the assumption that Q(ẑ t+1 | x t+1) = P (z t+1 | x t+1), (b) holds from the log function concavity, and (c) holds by a simple decomposition to the different components. In the following sections we will provide the description of the data collection process, domains, and implementation details used in the experiments. To generate our training and test sets, each consists of triples (x t, u t, x t+1), we: sample an underlying state s t and generate its corresponding observation x t, sample an action u t, and obtain the next state s t+1 according to the state transition dynamics, add it a zero-mean Gaussian noise with variance σ 2 I ns, and generate it's corresponding observation x t+1.To ensure that the observation-action data is uniformly distributed (see Section 3), we sample the state-action pair (s t, u t) uniformly from the state-action space. To understand the robustness of each model, we consider both deterministic (σ = 0) and stochastic scenarios. In the stochastic case, we add noise to the system with different values of σ and evaluate the models' performance under various degree of noise. Planar System In this task the main goal is to navigate an agent in a surrounded area on a 2D plane , whose goal is to navigate from a corner to the opposite one, while avoiding the six obstacles in this area. The system is observed through a set of 40 × 40 pixel images taken from the top view, which specifies the agent's location in the area. Actions are two-dimensional and specify the x − y direction of the agent's movement, and given these actions the next position of the agent is generated by a deterministic underlying (unobservable) state evolution function. Start State: one of three corners (excluding bottom-right). Goal State: bottom-right corner. Agent's Objective: agent is within Euclidean distance of 2 from the goal state. Inverted Pendulum -SwingUp & Balance This is the classic problem of controlling an inverted pendulum from 48 × 48 pixel images. The goal of this task is to swing up an under-actuated pendulum from the downward resting position (pendulum hanging down) to the top position and to balance it. The underlying state s t of the system has two dimensions: angle and angular velocity, which is unobservable. The control (action) is 1-dimensional, which is the torque applied to the joint of the pendulum. To keep the Markovian property in the observation (image) space, similar to the setting in E2C and RCE, each observation x t contains two images generated from consecutive time-frames (from current time and previous time). This is because each image only shows the position of the pendulum and does not contain any information about the velocity. Start State: Pole is resting down (SwingUp), or randomly sampled in ±π/6 (Balance). Agent's Objective: pole's angle is within ±π/6 from an upright position. CartPole This is the visual version of the classic task of controlling a cart-pole system . The goal in this task is to balance a pole on a moving cart, while the cart avoids hitting the left and right boundaries. The control (action) is 1-dimensional, which is the force applied to the cart. The underlying state of the system s t is 4-dimensional, which indicates the angle and angular velocity of the pole, as well as the position and velocity of the cart. Similar to the inverted pendulum, in order to maintain the Markovian property the observation x t is a stack of two 80 × 80 pixel images generated from consecutive time-frames. Start State: Pole is randomly sampled in ±π/6. Agent's Objective: pole's angle is within ±π/10 from an upright position. 3-link Manipulator -SwingUp & Balance The goal in this task is to move a 3-link manipulator from the initial position (which is the downward resting position) to a final position (which is the top position) and balance it. In the 1-link case, this experiment is reduced to inverted pendulum. In the 2-link case the setup is similar to that of arcobot , except that we have torques applied to all intermediate joints, and in the 3-link case the setup is similar to that of the 3-link planar robot arm domain that was used in the E2C paper, except that the robotic arms are modeled by simple rectangular rods (instead of real images of robot arms), and our task success criterion requires both swing-up (manipulate to final position) and balance. 12 The underlying (unobservable) state s t of the system is 2N -dimensional, which indicates the relative angle and angular velocity at each link, and the actions are N -dimensional, representing the force applied to each joint of the arm. The state evolution is modeled by the standard Euler-Lagrange equations . Similar to the inverted pendulum and cartpole, in order to maintain the Markovian property, the observation state x t is a stack of two 80 × 80 pixel images of the N -link manipulator generated from consecutive time-frames. In the experiments we will evaluate the models based on the case of N = 2 (2-link manipulator) and N = 3 (3-link manipulator). Start State: 1 st pole with angle π, 2 nd pole with angle 2π/3, and 3 rd pole with angle π/3, where angle π is a resting position. Agent's Objective: the sum of all poles' angles is within ±π/6 from an upright position. TORCS Simulaotr This task takes place in the TORCS simulator (specifically in michegan f1 race track, only straight lane). The goal of this task is to control a car so it would remain in the middle of the lane. We restricted the task to only consider steering actions (left / right in the range of [−1, 1]), and applied a simple procedure to ensure the velocity of the car is always around 10. We pre-processed the observations given by the simulator (240 × 320 RGB images) to receive 80 × 80 binary images (white pixels represent the road). In order to maintain the Markovian property, the observation state x t is a stack of two 80 × 80 images (where the two images are 7 frames apart -chosen so that consecutive observation would be somewhat different). The task goes as follows: the car is forced to steer strongly left (action=1), or strongly right (action=-1) for the initial 20 steps of the simulation (direction chosen randomly), which causes it to drift away from the center of the lane. Then, for the remaining horizon of the task, the car needs to recover from the drift, return to the middle of the lane, and stay there. Start State: 20 steps of drifting from the middle of the lane by steering strongly left, or right (chosen randomly). Agent's Objective: agent (car) is within Euclidean distance of 1 from the middle of the lane (full width of the lane is about 18). In the following we describe architectures and hyper-parameters that were used for training the different algorithms. All the algorithms were trained using: • Batch size of 128. • ADAM with α = 5 · 10 −4, β 1 = 0.9, β 2 = 0.999, and = 10 −8. • L 2 regularization with a coefficient of 10 −3. • Additional VAE loss term given by VAE t = −E q(z|x) [log p(x|z)] + D KL (q(z|x) p(z)), where p(z) ∼ N. The term was added with a very small coefficient of 0.01. We found this term to be important to stabilize the training process, as there is no explicit term that governs the scale of the latent space. • λ from the loss term of E2C was tuned using a parameter sweep in {0.25, 0.5, 1}, and was chosen to be 0.25 across all domains, as it performed the best independently for each domain. PCC training specifics: • λ p was set to 1 across all domains. • λ c was set to be 7 across all domains, after it was tuned using a parameter sweep in {1, 3, 7, 10} on the Planar system. • λ cur was set to be 1 across all domains without performing any tuning. • {z,ū}, for the curvature loss, were generated from {z, u} by adding Gaussian noise N (0, 0.1 2), where σ = 0.1 was set across all domains without performing any tuning. • Motivated by , we added a deterministic loss term in the form of cross entropy between the output of the generative path given the current observation and action (while taking the means of the encoder output and the dynamics model output) and the observation of the next state. This loss term was added with a coefficient of 0.3 across all domains after it was tuned using a parameter sweep over {0.1, 0.3, 0.5} on the Planar system. • E ADDITIONAL E.1 PERFORMANCE ON NOISY DYNAMICS Table 3 shows for the noisy cases. 1.2 ± 0.6 0.6 ± 0.3 17.9 ± 3.1 5.5 ± 1.2 6.1 ± 0.9 44.7 ± 3.6 Planar2 0.4 ± 0.2 1.5 ± 0.9 14.5 ± 2.3 1.7 ± 0.5 15.5 ± 2.6 29.7 ± 2.9 Pendulum1 6.4 ± 0.3 23.8 ± 1.2 16.4 ± 0.8 8.1 ± 0.4 36.1 ± 0.3 29.5 ± 0.2 Cartpole1 8.1 ± 0.6 6.6 ± 0.4 9.8 ± 0.7 20.3 ± 11 16.5 ± 0.4 17.9 ± 0.8 3-link1 0.3 ± 0.1 0 ± 0 0.5 ± 0.1 1.3 ± 0.2 0 ± 0 1.8 ± 0.3 Under review as a conference paper at ICLR 2020 The following figures depicts 5 instances (randomly chosen from the 10 trained models) of the learned latent space representations for both the noiseless and the noisy planar system from PCC, RCE, and E2C models.

represent the tldr for finding the corresponding scientific article.

Learning embedding for control with high-dimensional observations

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: The interplay between inter-neuronal network topology and cognition has been studied deeply by connectomics researchers and network scientists, which is crucial towards understanding the remarkable efficacy of biological neural networks. Curiously, the deep learning revolution that revived neural networks has not paid much attention to topological aspects. The architectures of deep neural networks (DNNs) do not resemble their biological counterparts in the topological sense. We bridge this gap by presenting initial of Deep Connectomics Networks (DCNs) as DNNs with topologies inspired by real-world neuronal networks. We show high classification accuracy obtained by DCNs whose architecture was inspired by the biological neuronal networks of C. Elegans and the mouse visual cortex. Recent advancements in neural network models have emerged through research in network architectures (; ;), optimization (; ;), and generalization techniques (; ;), with convolutional layers inspired from receptive fields and functional architecture in cats' visual cortex. However, the field of deep neural networks, with all its neuro-biologically inspired building blocks, has mostly left the topology story out. 1 Curiously, in the Cambrian explosion of neural network architectures in the post-AlexNet era, none seem to be inspired by the ideas prevalent in the domain of brain connectomics. The field of neuroscience was drawn into the network sciences when introduced the small-world network model, an example of which is the neuronal network of the nematode Caenorhabditis elegans (C. Elegans) 3. This idea was a sharp departure from the literature at the time as it considered a network model which was neither completely regular nor completely random -a model of complex networks. Complex networks with small world topology serve as an attractive model for the organization of brain anatomical and functional networks because a small-world topology can support both segregated/specialized and distributed/integrated information processing . Interestingly, while applications of complex networks based modeling have been well explored by the neuroscience community, they have been largely unexplored by the machine learning community as an avenue for designing and understanding deep learning architectures. Bridging the gap between neural connectomics and deep learning, we propose initial findings of designing neural network wiring based on connectomic structures as an intersection between network sciences, neuroscience and deep learning, and test our findings on image classification. Small-World Networks. By rewiring regular networks to introduce higher entropy and disorder, Watts and Strogatz proposed small-world networks with high clustering and small average path length. The model is analogous to six degrees of separation in the small-world phenomenon . Small-world networks are present in C. Elegans's connectome, power grid networks and protein interactions . Small-world models in neuroscience. Human brain structural and functional networks follow small-world configuration and this small-world model captures individual cognition and exhibits physiological basis . In the field of development psychology, literature shows that small-world modules and hubs are present during the mid-gestation period, and early brain network topology can predict later behavioral and cognitive performance . Erdos-Renyi (ER) (Erdős & Rényi, 1960), Barabasi-Albert (BA) (Albert & Barabási, 2002), and Watts-Strogatz (WS) models, Xie et. al. applied these graphs for image classification and showed that randomly wired neural networks achieve competitive accuracy on the ImageNet benchmark . Successes of ResNets and DenseNets in their performance as the first convolutional neural network (CNNs) that surpasses human-level performance on ImageNet were largely attributed to creative wiring patterns, with skip connections between multiple localized convolutional layers that is analogous to long-range connections across dense localized clusters, akin to small-world networks in neuronal networks. Inspired by small-world structures in deep CNNs, we construct DCNs based on biological neuronal network patterns, and determine their effectiveness in image classification 4. Figure 1: Wired graphs adopt connectomics structure from mouse and C. Elegans connectomes. Colored nodes indicate localized clusters in DCNs with small-world structures. We obtain small-world models of the mouse visual cortex 5 and C. Elegans connectomes . C. Elegans neuronal network includes 2D spatial positions of the rostral ganglia neurons 6, while the mouse primary visual cortex was characterized with electron microscopy. We treat the neuronal networks of both C. Elegans and the mouse visual cortex as undirected graphs which we convert to directed acyclic graphs (DAGs) in the same manner as by randomly assigning vertices indices, and set the edge to point from the smaller index to the larger index, which enforces a partial ordering on vertices. We introduce source and sink nodes by connecting vertices with indegree 0 and outdegree 0 respectively to ensure singular input and singular output through the DAG graph. The source broadcasts copies to the input nodes, and the sink performs an unweighted average on output nodes. With the exception of our choices in graph topology and number of layers, we inherit our architecture from the "small regime" RandWire architecture described in , which includes two conv layers, followed by randomly wired modules, and a fully connected softmax layer. Our modifications to this include the use of one conv layer instead of two, and the use of a single'random wiring' module as opposed to the three. Our single'random wiring' module consists of one of the topologies described in the previous subsection. As per the RandWire architecture, each node in the graph performs a weighted sum of the input, followed by a ReLU-conv-BN triplet transformation with a 3x3 separable convolution. Copies of the processed output are then broadcast to other nodes. We train for 50 epochs with Adam optimizer with learning rate of 0.001, batch size of 32 and with half-period cosine learning rate decay. For the first conv block before the DAG, we use a 2D conv with kernel size of 3, stride of 2 and padding of 1, followed by BN and ReLU. We evaluated a model with one conv block and a fully connected layer without any DAG as a baseline. Furthermore, we evaluated the C. Elegans and mouse visual cortex DCNs and compared these with the best graph structure in , the Watts-Strogatz network. These were evaluated on MNIST, while the C. Elegans model was also evaluated on Fashion-MNIST and KMNIST. In the generation of random graphs, Xie et. al. found that WS graphs performed better than ER and BA graphs. We thus compared DCNs with a simple convolutional graph-free CNN, and that with WS graphs as a baseline, and we observe that biologically wired DCNs outperform baselines without graphs, and with WS graphs. It could be argued that the accuracy improvement could be attributed to the increased number of parameters, so we further conduct experiments where we freeze the graph, thus keeping the same number of parameters as the graph-free CNN baseline. While the frozen WS model performs worse than the baseline, the Mouse visual cortex model performs better than the baseline even when frozen, suggesting that the graph topology is significant and independent of the number of parameters. For the C. Elegans DCN, we evaluated the performance across different datasets and showed consistently competitive on MNIST, KMNIST, and Fashion MNIST. Figure 3 shows the distribution of our compiled from 10 training trials on each dataset. The mean test accuracy on MNIST was 99%, while we achieved 93% on KMNIST, and 90% on Fashion MNIST. We demonstrated initial findings from applying networks inspired by real-world neuronal network topologies to deep learning. Our experiments show the trainability of a DNN based on the neuronal network C.Elegans and the visual cortex of a mouse with and without freezing the parameters of the graph modules, which outperforms WS graphs with good theoretical small-world properties. In future work, we will examine more principled methods for constructing a DAG from these networks and examine the impact of spectral properties of the graph topologies used both in the architectures we proposed and in the architecture proposed by , while extending to other connectomes.

Represent the TLDR for finding the corresponding scientific article

Initial findings in the intersection of network neuroscience and deep learning. C. Elegans and a mouse visual cortex learn to recognize handwritten digits.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: represent the scientific article to find a short tldr

Question: Convolutional neural networks and recurrent neural networks are designed with network structures well suited to the nature of spacial and sequential data respectively. However, the structure of standard feed-forward neural networks (FNNs) is simply a stack of fully connected layers, regardless of the feature correlations in data. In addition, the number of layers and the number of neurons are manually tuned on validation data, which is time-consuming and may lead to suboptimal networks. In this paper, we propose an unsupervised structure learning method for learning parsimonious deep FNNs. Our method determines the number of layers, the number of neurons at each layer, and the sparse connectivity between adjacent layers automatically from data. The ing models are called Backbone-Skippath Neural Networks (BSNNs). Experiments on 17 tasks show that, in comparison with FNNs, BSNNs can achieve better or comparable classification performance with much fewer parameters. The interpretability of BSNNs is also shown to be better than that of FNNs. Deep neural networks have made breakthroughs in all kinds of machine learning tasks BID13 BID22, specifically with convolutional neural networks (CNNs) for tasks with spacial data BID17 and recurrent neural networks (RNNs) for tasks with sequential data. One of the key reasons for the effectiveness of CNNs and RNNs is the well-designed network structures together with the parameter sharing schemes. For example, in the convolution layers of CNNs, each neuron is connected to a local region in the input volume instead of all the input neurons. Besides, the neurons in the same channel share the same set of weights. This design utilizes the local and "stationary" properties of spacial data and consequently forms effective feature extractors. In addition, it also prevents CNNs from having an exploding number of parameters when the networks become deeper and deeper. However, in practice, there are also many data which are neither spacial nor sequential, and hence the only applicable neural networks are the standard feed-forward neural networks (FNNs). In contrast to CNN and RNN, FNN's network structure is simple. It consists of multiple layers of neurons and each layer is fully connected to the next layer up, without considering any correlations in data or among neurons. The network structure has two main shortcomings. The first is that, there can be high connection redundancies. As the number of layers and the number of neuron at each layer increase, the number of parameters increases quickly, which can cause severe overfitting. The other shortcoming is that, ignoring all the correlations existing in data weakens the model's strength (as a feature extractor) and hurts the model's interpretability. We are interested in learning parsimonious deep feed-forward neural networks. The goal is to learn FNNs which contain as few parameters as possible. Parsimonious FNNs are desirable for several reasons. Firstly, fewer parameters can ease overfitting. Secondly, parsimonious FNNs require less storage and computation than FNNs, which makes it possible to be run on devices like mobile phones. Lastly, parsimonious FNNs can have very flexible and different structures from each other depending on the specific tasks and data. This would help the models fit the data well and also have good interpretability. In general, it is desirable to solve a problem using the simplest model possible because it implies a good understanding of the problem. connections (x − h 1, h 1 − h 2) form the Backbone path. The narrow fully-connected layers (x − h 3, h 1 − h 3, h 2 − h 3) are the Skip-paths. The number of units at h 3 is relatively smaller than that at x, h 1 and h 2.Learning parsimonious FNNs is challenging mainly because we need to determine the sparse connectivity between layers. Network pruning is a potential way to achieve this. However, it requires to start from a network which is much larger than necessary for the task at hand. This can cause a lot of computations wasted on those useless connections. In addition, network pruning is not able to learn the number of units and number of layers. In this paper, we assume that data are generated by a sparse probabilistic model with multiple layers of latent variables, and view the feed-forward network to be built as a way to approximate the relationships between the observed variables and the top-level latent variables in the probabilistic model. The level 1 latent variables induce correlations among the observed variables. Therefore, it is possible to determine them by analysing how the observed variables are correlated. Similarly, by analysing how the level 1 latent variables are correlated, we can determine the level 2 latent variables, and so on. We empirically show that our method can significantly reduce the number of parameters in FNNs, and the ing model still achieves better or comparable than FNNs in 17 classification tasks. Network Structure Learning One early attempt to learn network structure for FNNs is the approach based on constructive algorithms BID1 BID3 BID18. These algorithms start from a small network and gradually add new neurons to the network until some stopping criterion are met (e.g. no more performance gain is observed). They require manuallydesigned strategies to decide how to connect new neurons to the existing network. Besides, each time when new neurons are introduced, the network needs to be retrained completely or partially. Lately, BID0 proposes to learn the structure of deep belief networks by using cascading Indian buffet process, which is very time-consuming. In BID6, the authors propose a structure learning method, based on hierarchical latent tree analysis BID21 BID5, for RBM-like models. The method automatically determines the number of hidden units and the sparse connections between layers. However, it is not tested on deep models and in supervised learning tasks. Recently, reinforcement learning BID2 BID34 and genetic algorithms BID27 BID31 are also applied to learning complex structures for CNNs. Generally, these methods require tens of thousands of full training runs before giving a feasible network structure, which is prohibitive for many applications. Network Pruning In contrast to constructive algorithms, network pruning starts from a large network and prune connections or neurons to achieve structure learning. Optimal Brain Damage BID8 and Optimal Brain Surgeon BID12 prune connections based on the Hessian matrix of the loss function. Recently, BID11 proposes to conduct pruning by iteratively pruning connections with absolute weight value smaller than a threshold and retraining the network. One drawback of the method is that the retraining process is time-consuming. BID10 proposes Dynamic Network Surgery which conducts parameter learning and connection pruning simultaneously and avoids the retraining process. Moreover, it also allows mistakenly pruned connections to be rebuilt in subsequent training. Similar to connection pruning, neurons pruning methods are proposed and tested in BID28; BID20. The main drawback of all these pruning methods is that, they require to start from a network which is larger than necessary for the task at hand. This causes some wasted computations on the useless connections or neurons. In addition, the number of layers is still set manually instead of learned from data. In this section, we present a method for learning parsimonious deep FNNs. The method is called Parsimonious Structure Analysis (PSA). PSA learns a model which contains two parts as shown in FIG0. The first is the main part of the model, called the Backbone. It is a wide, deep but sparse feed-forward path in the network. The second part is the Skip-paths. It consists of multiple narrow paths, each of which is a fully-connected layer. We call the ing model Backbone-Skippath Neural Network (BSNN). We will introduce how PSA learns the Backbone and the Skip-paths in Section 3.1 and Section 3.2 respectively. Structure learning for neural networks is challenging since generally the features in data do not always have apparent relationships as the units in convolutional networks. In a convolutional layer, units in a feature map are only connected to a group of units strongly correlated in the spacial dimension at the layer below. This significantly reduces the number of parameters in CNNs and is essential if we want to learn a very sparse structure. The same intuition can be applied to general data other than images in feed-forward neural networks. A hidden unit, detecting one particular feature such as co-occurrence pattern, should only be connected to a group of units that are strongly correlated in the layer below. However, unlike CNNs where the spatial correlation is apparent, the correlations of units in feed-forward neural networks are not easy to discover. In PSA, we propose to apply Hierarchical Latent Tree Analysis (HLTA) BID21 BID5 to identify the co-occurrence patterns among units and construct hidden units to explain the co-occurrence patterns. PSA treats the input features as a set of isolated random variables as in Figure 3 (a). Although no apparent spacial or sequential relationships exist among the variables, PSA seeks to discover the correlations among the variables and groups the highly correlated ones together. It starts from finding two most correlated variables to form one group and keeps expanding the group if necessary. Let S denotes the set of observed variables which haven't been included into any variable groups. PSA firstly computes the mutual information between each pair of observed variables. Then it picks the pair in S with the highest mutual information and uses them as the seeds of a new variable group G. New variables from S are then added to G one by one in descending order of their mutual information with variables already in G. Each time when a new variable is added into G, PSA builds two models (M 1 and M 2) with G as the observed variables. The two models are the best models with one single latent variable and two latent variables respectively, as shown in Figure 2. PSA computes the BIC scores of the two models and tests whether the following condition is met: DISPLAYFORM0 where D is the dataset and δ is a threshold which is usually set at 3 BID5. When the condition is met, the two latent variable model M 2 is not significantly better than the one latent variable model M 1. Correlations among variables in G are still well modeled using a single latent variable. Then PSA keeps on adding new variables to G. If the test fails, PSA takes the subtree in M 2 which doesn't contain the newly added variable and identifies the observed variables in it as a finalized variable group. The group is then removed from S. And the above process is repeated on S until all the variables in S are partitioned into disjoint groups. An efficient algorithm progressive EM ) is used to estimate the parameters in M 1 and M 2.As shown in Figure 2 (b), after the above process, all the observed variables are partitioned into disjoint groups such that the variables in each group are strongly correlated and their correlations can be explained using a single latent variables. Then PSA introduces a latent variable for each group and computes the mutual information among the latent variables. After that, it links up the Figure 4: Expanding the tree structure for the Backbone path: A three-layer structure is first learned (left). New connections are added to all the layers according to empirical conditional mutual information (middle). The connections between variables at the top layer are removed and the structure is finalized (right). latent variables to form a Chow-Liu tree BID7. The is a latent tree model BID26 BID32, as shown in Figure 2 (c). Parameter estimation for the model is done using the EM algorithm. Since the model is tree-structured, EM is efficient in this process. While the above procedure gives us a one-layer network, we seek to build deep model to capture the long-range correlations among variables. We perform the construction of deep structure in a layerwise manner. Using the obtained one-layer model, PSA converts the latent variables into observed ones through data completion. With this, another layer of latent variables can be learned in the same manner as the first layer by grouping the first-layer latent variables and linking up the groups, as in Figure 2 (d). Then the two models can be stacked up to form a three-layer network, with the latent variables in the higher layer capturing longer-range correlations of the observed variables. This procedure can be recursively conducted to build deep hierarchy until the number of variables at the top layer falls below a threshold K. And it in a hierarchical latent tree model BID21 BID5. While the above deep structure captures the most important correlations among the observed variables, the tree structure might cause underfitting for discovering non-trivial correlations. Thus we introduce additional links to model the salient interactions that are not captured by the tree model. For each latent variable V l at level l, PSA considers adding connections to link it to more nodes at level l − 1. To do so, PSA considers how closely V l is related to each node V l−1 at level l − 1 given the parent variable Z of V l−1. The strength of correlation is measured using the conditional mutual information: DISPLAYFORM0 The top N nodes with the highest I(V l, V l−1 |Z) are then connected to V l. After expanding the connections for all the layers, PSA removes the links among the variables at the top layer and uses the ing structure for the Backbone. The process of expanding tree structure is illustrated in Figure 4. Although the Backbone path is deep and wide, its sparsity can easily lead to model which cannot capture global features. For example, suppose there is an essential feature which is correlated to all the input features. When the Backbone path is very sparse, even after multiple layers of projections, it is still unlikely that there will be a feature in the model which is projected from all the input features. To tackle the above problem, we introduce Skip-paths to our BSNN. FIG0 shows the whole model structure of BSNN. The path from x to h 2 illustrates the Backbone path whose sparse structure is learned using the method we propose. To complement the the model's power of extracting features, narrow Skip-paths (x − h 3, h 1 − h 3, h 2 − h 3) are added to the model. The Skip-paths take all the feature layers in the Backbone as input and compress them to layers with a small number of units through fully-connected projections. After the structure for the Backbone path and the Skip-paths are determined, a classification layer or regression layer can then be added to the top of all the paths, utilizing all the features extracted. The network can then be trained using back-propagation algorithms as in normal neural networks. In experiment, we evaluate our method in 17 classification tasks. We consider applications where the data is neither spacial nor sequential. Unlike CNNs or RNNs where the structure is designed to exploit spatial or sequential correlation, few effort has been put to learn the structure of feedfoward neural networks, which have highly redudant parameters and is prone to overfit. Our proposed method learns the structure of feedforward neural network from data. It significantly reduces the model complexity and parameters while achieving better or comparable classification performance, and leads to models which are more interpretable. TAB0 gives a summary of all the datasets used in the experiment. We choose 12 tasks for chemical compounds classification and 5 tasks for text classification. All the datasets are published by previous researchers and are available to the public. 1 There are about 12,000 environmental chemical compounds in the dataset, each represented as its chemical structure. The tasks are to predict 12 different toxic effects for the chemical compounds. We treat them as 12 binary classification tasks. We filter out sparse features which are present in fewer than 5% compounds, and rescale the remaining 1,644 features to zero mean and unit variance. The dataset contains a training set and a test set, and we randomly sample 500 compounds from training data to build the validation set. All the experiments are run for three times and we report the average AUC together with the standard deviations. 2 We use 5 text classification datasets from BID33. After removing stop words, the top 10,000 frequent words in each dataset are selected as the vocabulary respectively and each document is represented as bag-of-words over the vocabulary. The validation set is randomly sampled from the original training samples. We run all the experiments for three times and report the average classification accuracies with the standard deviations. We compare our model with standard feed-forward neural networks (FNNs) and sparse neural networks whose weak connections are pruned (Pruned FNNs) in the 17 classification tasks. The models involved in the experiment are as follows:• BSNN: Backbone-Skippath Neural Network is the ing model of our method PSA. For all the tasks, we keep only 5% of the connections in the Backbone path and limit the number of units in the narrow Skip-paths to 100.• FNN: Feed-forward Neural Network is a standard fully-connected neural network. It is mainly composed of linear layers and activation functions. Each hidden unit is connected to all neurons in the previous layer. Information flows from low layers to high layers in a feed-forward manner.• Pruned FNN: Pruned Feed-forward Neural Network is trained by using the method proposed in BID11. We Firstly train a fully-connected FNN from scratch, and then prune out the weak connections with small absolute weight values. The pruned network is then retrained from the initial training phase by keeping the surviving weight parameters. We learn the structure of BSNNs using PSA. The number of layers, the number of hidden units in each layer and the sparse connections between adjacent layers are automatically determined. After structure learning, we train the sparse model from scratch by random initialization of weights. As for FNNs, we treat the number of hidden units and number of layers as hyper-parameters of network and determine the best structure by grid-search over all the combinations using validation data. TAB1 shows the space of network structures considered. Following the method in BID16, both "rectangle " and "conic" network shapes are tested. In FNNs with rectangle shape, all the hidden layers have constant number of units. FNNs with conic shape start with the given number of hidden units and decrease it layer by layer in a geometric progression manner towards the output layer. For Pruned FNNs, we take the best FNNs as the initial model and perform pruning as in BID11. The pruned model is then retrained for final model. We implement all the experiments using PyTroch 3 which is a flexible deep learning framework. We use ReLUs BID25 BID9 as the non-linear activation functions in all the networks. Dropout BID14 BID29 ) with rate 0.5 is applied after each non-linear projection. We use Adam BID15 as the optimizer to optimize the training objective function. During training, we select models by monitoring validation loss. Codes will be released after the paper is accepted to the conference. TAB2 shows the classification of BSNNs and FNNs on Tox21 dataset. The structures of FNNs are tuned individually for each task. It is clear that BSNNs achieve better AUC scores on 10 out of the 12 classification tasks. Even when it is not better, the average AUC value of BSNNs, e.g. on task SR.MMP, is also very close to that of FNNs. More importantly, BSNNs always contain much fewer parameters than FNNs, with the ratios of parameter number ranging from 7% to 40.11%. TAB3 shows the of BSNNs and FNNs over the 5 text classification tasks. Although BSNNs contain much fewer parameters than FNNs, BSNNs still achieve higher classification accuracy in the first two tasks, and comparable accuracy in the remaining tasks. Note that the ratios of parameter number ranges from 6.25% to 32.07%. This again confirms that our method learns good parsimonious deep models which can achieve high classification performance with much fewer parameters than standard FNNs. To validate our assumption that the backbone path in BSNNs captures most of the information in data and acts as a main part of the model, we remove the narrow skip-paths in BSNNs and train the model to test its performance in classification tasks. TAB4 shows the . As we can see from the , the backbone path alone already achieves AUC scores or accuracies which are only slightly worse than BSNNs. Note that the number of parameters in the sparse path is even much smaller than BSNNs. Compared with FNNs, the number of parameters is only 2% 11%, significantly smaller than that of FNNs. However, without the backbone, the performance of the model will be significantly worse due to the insufficient capability of the other narrow path. The not only show the importance of the backbone path in BSNNs, but also shows that our structure learning method in the backbone path is effective enough. To further show the effectiveness of our structure learning method, we introduce a new model called BSNN-FC. For each specific task, the structure of BSNN-FC is completely the same as that of BSNN, except that the layers in the sparse Backbone path are changed to fully-connected layers. We train BSNN-FC for all the tasks in Tox21 dataset and the are shown in TAB5. From the table we can see that, although BSNN keeps only 5% of the connections in the sparse path, it gives classification which are very similar to that of BSNN-FC. It shows that our structure learning method successfully removes the useless connections in BSNN-FC.We also compare BSNNs with Pruned FNNs whose weak connections are pruned using the method in BID11. We start from the fully pretrained FNNs reported in TAB2, and prune the connections with the smallest absolute weight values. After pruning, the number of remaining parameters in each FNN is the same as that in the corresponding BSNN for the same task. The comparison between BSNNs and pruned FNNs is shown in TAB5. Again BSNNs give higher AUC scores than pruned FNNs in 10 of the 12 classification tasks. Next we compare the interpretability of BSNNs with FNNs and Pruned FNNs on the text datasets. Here is how we interpret hidden units. We feed the data to the networks and do forward propagation to get the values of the hidden units corresponding to each data sample. Then for each hidden unit, we sort the words in descending order of the correlations between the words and the hidden unit. The top 10 words with the highest correlations are chosen to characterize the hidden unit. Following BID6, we measure the interpretability of a hidden unit by considering how similar pairs of words in the top-10 list are. The similarity between two words is determined using a word2vec model BID23 b) trained on part of the Google News datasets, where each word is mapped to a high dimensional vector. The similarity between two words is defined as the cosine similarity of the two corresponding vectors. High similarity suggests that the two words appear in similar contexts. The interpretability score of a hidden unit is defined as the compactness of its characterizing words and is computed as the average similarity of all pairs of words. The interpretability score of a model is defined as the average of interpretability scores of all hidden units. Table 7 reports the interpretability scores of BSNNs, FNNs and Pruned FNNs for different datasets. Sogounews dataset is not included in the experiment since its vocabulary are Chinese pingyin characters and most of them do not appear in the Google News word2vec model. We measure the interpretability scores by considering the top-layer hidden units. For the fair of comparison, all models have approximately the same number of top-layer hidden units. As it can be seen that BSNNs significantly outperform the FNNs and Pruned FNNs in most cases and is comparable if not better, showing superior coherency and compactness in the characterizations of the hidden units and thus better model interpretability. Pruned FNNs, on the other hand, reduce the interpretability of FNNs with the pruning strategy. Table 8 shows the qualitative interpretability by presenting the characterization words of hidden units with high interpretability scores in BSNNs. The hidden units are very meaningful for different datasets. For example, in Yelp Review dataset, the first hidden unit represents negative opinions on food with words "tasteless" and"flavorless"; the second hidden unit is more related to food like "paprika", "crust" and "unagi". In DBPedia, the first hidden unit is found out to have closer relationship with music, while the second one is more closely related to sport. Similar phenomena can be found in the rest of the table. This shows that the proposed BSNNs, with the statistical property, have better model interpretability and make a step further towards understandable deep learning models. Structure learning for deep neural network is a challenging and interesting research problem. We have proposed an unsupervised structure learning method which utilizes the correlation information in data for learning parsimonious deep feed-forward networks. In comparison with standard FNN, although the ing model of our method contains much fewer parameters, it achieves better or comparable classification performance in all kinds of tasks. Our method is also shown to learn models with better interpretability, which is also an important problem in deep learning. In the future, we will generalize our method to other networks like RNNs and CNNs.

An unsupervised structure learning method for Parsimonious Deep Feed-forward Networks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.