Question: Represent the article for finding a single-sentence TLDR

Question: Reinforcement learning algorithms, though successful, tend to over-fit to training environments, thereby hampering their application to the real-world. This paper proposes $\text{W}\text{R}^{2}\text{L}$ -- a robust reinforcement learning algorithm with significant robust performance on low and high-dimensional control tasks. Our method formalises robust reinforcement learning as a novel min-max game with a Wasserstein constraint for a correct and convergent solver. Apart from the formulation, we also propose an efficient and scalable solver following a novel zero-order optimisation method that we believe can be useful to numerical optimisation in general. We empirically demonstrate significant gains compared to standard and robust state-of-the-art algorithms on high-dimensional MuJuCo environments Reinforcement learning (RL) has become a standard tool for solving decision-making problems with feedback, and though significant progress has been made, algorithms often over-fit to training environments and fail to generalise across even slight variations of transition dynamics . Robustness to changes in transition dynamics is a crucial component for adaptive and safe RL in real-world environments. Motivated by real-world applications, recent literature has focused on the above problems, proposing a plethora of algorithms for robust decisionmaking (; ;). Most of these techniques borrow from game theory to analyse, typically in a discrete state and actions spaces, worst-case deviations of agents' policies and/or environments, see;;; and references therein. These methods have also been extended to linear function approximators , and deep neural networks showing (modest) improvements in performance gain across a variety of disturbances, e.g., action uncertainties, or dynamical model variations. In this paper, we propose a generic framework for robust reinforcement learning that can cope with both discrete and continuous state and actions spaces. Our algorithm, termed Wasserstein Robust Reinforcement Learning (WR 2 L), aims to find the best policy, where any given policy is judged by the worst-case dynamics amongst all candidate dynamics in a certain set. This set is essentially the average Wasserstein ball around a reference dynamics P 0. The constraints makes the problem well-defined, as searching over arbitrary dynamics can only in non-performing system. The measure of performance is the standard RL objective, the expected return. Both the policy and the dynamics are parameterised; the policy parameters θ k may be the weights of a deep neural network, and the dynamics parameters φ j the parameters of a simulator or differential equation solver. The algorithm performs estimated descent steps in φ space and -after (almost) convergence -performs an update of policy parameters, i.e., in θ space. Since φ j may be high-dimensional, we adapt a zero'th order sampling method based extending to make estimations of gradients, and in order to define the constraint set which φ j is bounded by, we generalise the technique to estimate Hessians (Proposition 2). We emphasise that although access to a simulator with parameterisable dynamics is required, the actual reference dynamics P 0 need not be known explicitly nor learnt by our algorithm. Put another way, we are in the "RL setting", not the "MDP setting" where the transition probability matrix is known a priori. The difference is made obvious, for example, in the fact that we cannot perform dynamic programming, and the determination of a particular probability transition can only be estimated from sampling, not retrieved explicitly. Hence, our algorithm is not model-based in the traditional sense of learning a model to perform planning. We believe our contribution is useful and novel for two main reasons. Firstly, our framing of the robust learning problem is in terms of dynamics uncertainty sets defined by Wasserstein distance. Whilst we are not the first to introduce the Wasserstein distance into the context of MDPs (see, e.g., or), we believe our formulation is amongst the first suitable for application to the demanding application-space we desire, that being, high-dimensional, continuous state and action spaces. Secondly, we believe our solution approach is both novel and effective (as evidenced by experiments below, see Section 5), and does not place a great demand on model or domain knowledge, merely access to a simulator or differentiable equation solver that allows for the parameterisation of dynamics. Furthermore, it is not computationally demanding, in particular, because it does not attempt to build a model of the dynamics, and operations involving matrices are efficiently executable using the Jacobian-vector product facility of automatic differentiation engines. A Markov decision process (MDP) 1 is denoted by M = S, A, P, R, γ, where S ⊆ R d denotes the state space, A ⊆ R n the action space, P: S × A × S → is a state transition probability describing the system's dynamics, R: S × A → R is the reward function measuring the agent's performance, and γ ∈ specifies the degree to which rewards are discounted over time. At each time step t, the agent is in state s t ∈ S and must choose an action a t ∈ A, transitioning it to a new state s t+1 ∼ P (s t+1 |s t, a t), and yielding a reward R(s t, a t). A policy π: S × A → is defined as a probability distribution over state-action pairs, where π(a t |s t) represents the density of selecting action a t in state s t. Upon subsequent interactions with the environment, the agent collects a trajectory τ of state-action pairs. The goal is to determine an optimal policy π by solving: where p π (τ) denotes the trajectory density function, and R Total (τ) the return, that is, the total accumulated reward: with µ 0 (·) denoting the initial state distribution. We make use of the Wasserstein distance to quantify variations from a reference transition density P 0 (·). The latter being a probability distribution, one may consider other divergences, such as Kullback-Leibler (KL) or total variation (TV). Our main intuition for choosing Wasserstein distance is explained below, but we note that it has a number of desirable properties: Firstly, it is symmetric (W p (µ, ν) = W p (ν, µ), a property K-L lacks). Secondly, it is well-defined for measures with different supports (which K-L also lacks). Indeed, the Wasserstein distance is flexible in the forms of the measures that can be compared -discrete, continuous or a mixture. Finally, it takes into account the underlying geometry of the space the distributions are defined on, which can encode valuable information. It is defined as follows: Let X be a metric space with metric d(·, ·). Let C(X) be the space of continuous functions on X and let M(X) be the set of probability measures on X. Let µ, ν ∈ M(X). Let K(µ, ν) be the set of couplings between µ, ν: That is, the set of joint distributions κ ∈ M(X × X) whose marginals agree with µ and ν respectively. Given a metric (serving as a cost function) d(·, ·) for X, the p'th Wasserstein distance W p (µ, ν) for p ≥ 1 between µ and ν is defined as: (in this paper, and mostly for computational convenience, we use p = 2, though other values of p are applicable). The desirable properties of Wasserstein distance aside, our main intuition for choosing it is described thus: Per the definition, constraining the possible dynamics to be within an -Wasserstein ball of a reference dynamics P 0 (·) means constraining it in a certain way. Wasserstein distance has the form mass × distance. If this quantity is constrained to be less than a constant, then if the mass is large, the distance is small, and if the distance is large, the mass is small. Intuitively, when modelling the dynamics of a system, it may be reasonable to concede that there could be a systemic erroror bias -in the model, but that bias should not be too large. It is also reasonable to suppose that occasionally, the behaviour of the system may be wildly different to the model, but that this should be a low-probability event. If the model is frequently wrong by a large amount, then there is no use in it. In a sense, the Wasserstein ball formalises these assumptions. Due to the continuous nature of the state and action spaces considered in this work, we resort to deep neural networks to parameterise policies, which we write as π θ (a t |s t), where θ ∈ R d1 is a set of tunable hyper-parameters to optimise. For instance, these policies can correspond to multilayer perceptrons for MuJoCo environments, or to convolutional neural networks in case of highdimensional states depicted as images. Exact policy details are ultimately application dependent and, consequently, provided in the relevant experiment sections. In principle, one can similarly parameterise dynamics models using deep networks (e.g., LSTMtype models) to provide one or more action-conditioned future state predictions. Though appealing, going down this path led us to agents that discover worst-case transition models which minimise training data but lack any valid physical meaning. For instance, original experiments we conducted on CartPole ended up involving transitions that alter angles without any change in angular velocities. More importantly, these effects became more apparent in high-dimensional settings where the number of potential minimisers increases significantly. It is worth noting that we are not the first to realise such an artifact when attempting to model physics-based dynamics using deep networks. Authors in remedy these problems by introducing Lagrangian mechanics to deep networks, while others (; argue the need to model dynamics given by differential equation structures directly. Though incorporating physics-based priors to deep networks is an important and challenging task that holds the promise of scaling model-based reinforcement learning for efficient solvers, in this paper we rather study an alternative direction focusing on perturbing differential equation solvers and/or simulators with respect to the dynamic specification parameters φ ∈ R d2 . Not only would such a consideration reduce the dimension of parameter spaces representing transition models, but would also guarantee valid dynamics due to the nature of the simulator. Though tackling some of the above problems, such a direction arrives with a new set of challenges related to computing gradients and Hessians of black-box solvers. In Section 4, we develop an efficient and scalable zero-order method for valid and accurate model updates. Unconstrained Loss Function: Having equipped agents with the capability of representing policies and perturbing transition models, we are now ready to present an unconstrained version of WR 2 L's loss function. Borrowing from robust optimal control, we define robust reinforcement learning as an algorithm that learns best-case policies under worst-case transitions: where p φ θ (τ) is a trajectory density function parameterised by both policies and transition models, i.e., θ and φ, respectively: specs vector and diff. solver Under review as a conference paper at ICLR 2020 At this stage, it should be clear that our formulation, though inspired from robust optimal control, is, truthfully, more generic as it allows for parameterised classes of transition models without incorporating additional restrictions on the structure or the scope by which variations are executed 2. Constraints & Complete Problem Definition: Clearly, the problem in Equation 5 is ill-defined due to the arbitrary class of parameterised transitions. To ensure well-behaved optimisation objectives, we next introduce constraints to bound search spaces and ensure convergence to feasible transition models. For a valid constraint set, our method assumes access to samples from a reference dynamics model P 0 (·|s, a), and bounds learnt transitions in an -Wasserstein ball around P 0 (·|s, a), i.e., the set defined as: where ∈ R + is a hyperparameter used to specify the "degree of robustness" in a similar spirit to maximum norm bounds in robust optimal control. It is worth noting, that though we have access to samples from a reference simulator, our setting is by no means restricted to model-based reinforcement learning in an MDP setting. That is, our algorithm operates successfully given only traces from P 0 accompanied with its specification parameters, e.g., pole lengths, torso masses, etc. -a more flexible framework that does not require full model learners. Though defining a better behaved optimisation objective, the set in Equation 6 introduces infinite number of constraints when considering continuous state and/or actions spaces. To remedy this problem, we target a relaxed version that considers a constraint of average Wasserstein distance bounded by a hyperparameter: The sampling (s, a) in the expectation is done as follows: We sample trajectories using reference dynamics P 0 and a policy π that chooses actions uniformly at random (uar). Then (s, a) pairs are sampled uar from those collected trajectories. For a given pair (s, a), W 2 2 (P φ (·|s, a), P 0 (·|s, a)) is approximated through the empirical distribution: we use the state that followed (s, a) in the collected trajectories as a data point. Estimating Wasserstein distance using empirical data is standard, see, e.g., Peyré et al.. One approach which worked well in our experiments, was to assume that the dynamics are given by deterministic functions plus Gaussian noise with diagonal convariance matrices. This makes estimation easier in high dimensions since sampling in each dimension is independent of others, and the total samples needed is a constant factor of the number of dimensions. Gaussian distributions also have closed-form expressions for Wasserstein distance, given in terms of mean and covariance. We thus arrive at WR 2 L's optimisation problem allowing for best policies under worst-case yet bounded transition models: Wasserstein Robust Reinforcement Learning Objective: Our solution alternates between updates of θ and φ, keeping one fixed when updating the other. Fixing dynamics parameters φ, policy parameters θ can be updated by solving, which is the formulation of a standard RL problem. Consequently, one can easily adapt any policy search method for updating policies under fixed dynamical models. As described later in Section 4, we make use of proximal policy optimisation . When updating φ given a set of fixed policy parameters θ, the Wasserstein constraints must be respected. Unfortunately, even with the simplification introduced in Section 3.1 the constraint is still difficult to compute. To alleviate this problem, we propose to approximate the constraint in by its Taylor expansion up to second order. That is, defining W (φ):= E (s,a) W 2 2 (P φ (·|s, a), P 0 (·|s, a)), the above can be approximated around φ 0 by a second-order Taylor as: Recognising that W (φ 0) = 0 (the distance between the same probability densities), and ∇ φ W (φ 0) = 0 since φ 0 minimises W (φ), we can simplify the Hessian approximation by writing:. Substituting our approximation back in the original problem in Equation 8, we reach the following optimisation problem for determining model parameter given fixed policies: where is the Hessian of the expected squared 2-Wasserstein distance evaluated at φ 0. Optimisation problems with quadratic constraints can be efficiently solved using interior-point methods. To do so, one typically approximates the loss with a first-order expansion and determines a closed-form solution. Consider a pair of parameters θ [k] and φ [j] (which will correspond to parameters of the j'th inner loop of the k'th outer loop in the algorithm we present). To find φ [j+1], we solve: It is easy to show that a minimiser to the above equation can derived in a closed-form as: with g [k,j] denoting the gradient 3 evaluated at θ [k] and φ [j], i.e., g Generic Algorithm: Having described the two main steps needed for updating policies and models, we now summarise these findings in the pseudo-code in Algorithm 1. As the Hessian 4 of the Wasserstein distance is evaluated based on reference dynamics and any policy π, we pass it, along with and φ 0 as inputs. Then Algorithms 1 operates in a descent-ascent fashion in two main phases. In the first, lines 5 to 10 in Algorithm 1, dynamics parameters are updated using the closed-form solution in Equation 10, while ensuring that learning rates abide by a step size condition (we used the Wolfe conditions , though it can be some other method). With this, the second phase (line 11) utilises any state-of-the-art reinforcement learning method to adapt policy parameters generating θ [k+1]. Regarding the termination condition for the inner loop, we leave this as a decision for the user. It could be, for example, a large finite time-out, or the norm of the gradient g [k,j] being below a threshold, or whichever happens first., and j ← 0 Phase I: Update model parameter while fixing the policy: while termination condition not met do 7: Compute descent direction for the model parameters as given by Equation 10: Update candidate solution, while satisfying step size conditions (see discussion below) on the learning rate α: end while 10: Perform model update setting 11: Phase II: Update policy given new model parameters: Use any standard reinforcement learning algorithm for ascending in the gradient direction, e.g.,, with β [k] a learning rate. 13: end for Consider a simulator (or differential equation solver) S φ for which the dynamics are parameterised by a real vector φ, and for which we can execute steps of a trajectory (i.e., the simulator takes as input an action a and gives back a successor state and reward). For generating novel physics-grounded transitions, one can simply alter φ and execute the instruction in S φ from some a state s ∈ S, while applying an action a ∈ A. Not only does this ensure valid (under mechanics) transitions, but also promises scalability as specification parameters typically reside in lower dimensional spaces compared to the number of tuneable weights when using deep networks as transition models. Gradient Estimation: Recalling the update rule in Phase I of Algorithm 1, we realise the need for, estimating the gradient of the loss function with respect to the vector specifying the dynamics of the environment, i.e., g at each iteration of the inner-loop j. Handling simulators as black-box models, we estimate the gradients by sampling from a Gaussian distribution with mean 0 and σ 2 I co-variance matrix. Our choice for such estimates is not arbitrary but rather theoretically grounded as one can easily prove the following proposition: Proposition 1 (Zero-Order Gradient Estimate). For a fixed θ and φ, the gradient can be computed as: Hessian Estimation: Having derived a zero-order gradient estimator, we now generalise these notions to a form allowing us to estimate the Hessian. It is also worth reminding the reader that such a Hessian estimator needs to be performed one time only before executing the instructions in Algorithm 1 (i.e., H 0 is passed as an input). Precisely, we prove the following proposition: Proposition 2 (Zero-Order Hessian Estimate). The hessian of the Wasserstein distance around φ 0 can be estimated based on function evaluations. Recalling that, and defining W (s,a) (φ):= W 2 2 (P φ (·|s, a), P 0 (·|s, a)), we prove: Proofs of these propositions are given in Appendix A. They allow for a procedure where gradient and Hessian estimates can be simply based on simulator value evaluations while perturbing φ and φ 0. It is important to note that in order to apply the above, we are required to be able to evaluate An empirical estimate of the p-Wasserstein distance between two measures µ and ν can be performed by computing the p-Wasserstein distance between the empirical distributions evaluated at sampled data. That is, one can approximation µ by µ n = 1 n n i=1 δ xi where x i are identically and independently distributed according to µ. Approximating ν n similarly, we then realise that We evaluate WR 2 L on a variety of continuous control benchmarks from the MuJoCo environment. Dynamics in our benchmarks were parameterised by variables defining physical behaviour, e.g., density of the robot's torso, friction of the ground, and so on. We consider both low and high dimensional dynamics and demonstrate that our algorithm outperforms state-of-the-art from both standard and robust reinforcement learning. We are chiefly interested in policy generalisation across environments with varying dynamics, which we measure using average test returns on novel systems. The comparison against standard reinforcement learning algorithms allows us to understand whether lack of robustness is a critical challenge for sequential decision making, while comparisons against robust algorithms test if we outperform state-of-the-art that considered a similar setting to ours. From standard algorithms, we compare against proximal policy optimisation (PPO) , and trust region policy optimisation (TRPO) (b); an algorithm based on natural actor-crtic . From robust algorithms, we demonstrate how WR 2 L favours against robust adversarial reinforcement learning (RARL) , and action-perturbed Markov decision processes (PR-MDP) proposed in . Due to space constraints, the are presented in Appendix B.2. It is worth noting that we attempted to include deep deterministic policy gradients (DDPG) in our comparisons. Results including DDPG were, however, omitted as it failed to show any significant robustness performance even on relatively simple systems, such as the inverted pendulum; see reported in Appendix B.3. During initial trials, we also performed experiments parameterising models using deep neural networks. Results demonstrated that these models, though minimising training data error, fail to provide valid physics-grounded dynamics. For instance, we arrived at inverted pendula models that vary pole angles without exerting any angular speeds. This problem became even more apparent in high-dimensional systems, e.g., Hopper, Walker, etc due to the increased number of possible minima. As such, presented in this section make use of our zero-order method that can be regarded as a scalable alternative for robust solutions. We evaluate our method both in low and high-dimensional MuJuCo tasks . We consider a variety of systems including CartPole, Hopper, and Walker2D; all of which require direct joint-torque control. Keeping with the generality of our method, we utilise these frameworks as-is with no additional alterations, that is, we use the exact setting of these benchmarks as that shipped with OpenAI gym without any reward shaping, state-space augmentation, feature extraction, or any other modifications of-that-sort. Details are given in section B. Due to space constraints, for one-dimensional parameter variations are given in Appendix B.2, where it can be seen that WR 2 L outperforms both robust and non-robust algorithms when onedimensional simulator variations are considered. Figure 1 shows for dynamics variations along two dimensions. Here again, our methods demonstrates considerable robustness. The fourth column, "PPO mean", refers to experiments where PPO is trained on a dynamics sampled uniformly at random from the Wasserstein constraint set. It displayes more robustness than when trained on just the reference dynamics, however, as can be seen from Fig. 2, our method performs noticably better in high dimensions, which is the main strength of our algorithm. Results with High-Dimensional Model Variation: Though above demonstrate robustness, an argument against a min-max objective can be made especially when only considering lowdimensional changes in the simulator. Namely, one can argue the need for such an objective as opposed to simply sampling a set of systems and determining policies performing-well on average similar to the approach proposed in . A counter-argument to the above is that a gradient-based optimisation scheme is more efficient than a sampling-based one when high-dimensional changes are considered. In other words, a sampling procedure is hardly applicable when more than a few parameters are altered, while WR 2 L can remain suitable. To assess these claims, we conducted two additional experiments on the Hopper and HalfCheetah benchmarks. In the first, we trained robustly while changing friction and torso densities, and tested on 1,000 systems generated by varying all 11 dimensions of the Hopper dynamics, and 21 dimensions of the HalfCheetah system. The histogram Figures 2(b) and (f) demonstrate that the empirical densities of the average test returns are mostly centered around 3,000 for the Hopper, and around 4,500 for the Cheetah, which improves that of PPO trained on reference (Figures 2(a) and (e)) with return masses mostly accumulated at around 1,000 in the case of the Hopper and almost equally distributed when considering HalfCheetah. Such improvements, however, can be an artifact of the careful choice of the low-dimensional degrees of freedom allowed to be modified during Phase I of Algorithm 1. To get further insights, Figures 2(c) and (g) demonstrate the effectiveness of our method trained and tested while allowing to tune all 11 dimensional parameters of the Hopper sim- ulator, and the 21 dimensions of the HalfCheetah. Indeed, our are in accordance with these of the previous experiment depicting that most of the test returns' mass remains around 3,000 for the Hopper, and improves to accumulate around 4,500 for the HalfCheetah. Interestingly, however, our algorithm is now capable of acquiring higher returns on all systems 6 since it is allowed to alter all parameters defining the simulator. As such, we conclude that WR 2 L outperforms others when high-dimensional simulator variations are considered. In Figures 2(d) and (h), we see the for PPO trained with dynamics sampled uar from the Wasserstein constraint set. We see that although in the two-dimensional variation case this training method worked well (see Figures 1(d), (h), (l)), it does not scale well to high dimensions, and our method does better. Previous work on robust MDPs, e.g., , whilst valuable in its own right, is not sufficient for the RL setting due to the need in the latter case to give efficient solutions for large state and action spaces, and the fact that the dynamics are not known a priori. Closer to our own work, approaches the robustness problem by training on an ensemble of dynamics in order to be deployed on a target environment. The algorithm introduced, Ensemble Policy Optimisation (EPOpt), alternates between two phases: (i) given a distribution over dynamics for which simulators (or models) are available (the source domain), train a policy that performs well for the whole distribution; (ii) gather data from the deployed environment (target domain) to adapt the distribution. The objective is not max-min, but a softer variation defined by conditional value-at-risk (CVaR). The algorithm samples a set of dynamics {φ k} from a distribution over dynamics P ψ, and for each dynamics φ k, it samples a trajectory using the current policy parameter θ i. It then selects the worst performing -fraction of the trajectories to use to update the policy parameter. Clearly this process bears some resemblance to our algorithm, but there is a crucial difference: our algorithm takes descent steps in the φ space. The difference if important when the dynamics parameters sit in a high-dimensional space, since in that case, optimisationfrom-sampling could demand a considerable number of samples. In any case, our experiments demonstrate our algorithm performs well even in these high dimensions. We note that we were were unable to find the code for this paper, and did not attempt to implement it ourselves. The CVaR criterion is also adopted in , in which, rather than sampling trajectories and finding a quantile in terms of performance, two policies are trained simultaneously: a "protagonist" which aims to optimise performance, and an adversary which aims to disrupt the protagonist. The protagonist and adversary train alternatively, with one being fixed whilst the other adapts. We made comparisons against this algorithm in our experiments. More recently, studies robustness with respect to action perturbations. There are two forms of perturbation addressed: (i) Probabilistic Action Robust MDP (PR-MDP), and (ii) Noisy Action Robust MDP (NR-MDP). In PR-MDP, when an action is taken by an agent, with probability α, a different, possibly adversarial action is taken instead. In NR-MDP, when an action is taken, a perturbation is added to the action itself. and , the algorithm is suitable for applying deep neural networks, and the paper reports experiments on InvertedPendulum, Hopper, Walker2d and Humanoid. We tested against PR-MDP in some of our experiments, and found it to be lacking in robustness (see Section 5). a non-stationary Markov Decision Process model is considered, where the dynamics can change from one time step to another. The constraint is based on Wasserstein distance, specifically, the Wasserstein distance between dynamics at time t and t is bounded by L|t − t |, i.e., is L-Lipschitz with respect to time, for some constant L. They approach the problem by treating nature as an adversary and implement a Minimax algorithm. The basis of their algorithm is that due to the fact that the dynamics changes slowly (due to the Lipschitz constraint), a planning algorithm can project into the future the scope of possible future dynamics and plan for the worst. The ing algorithm, known as Risk Averse Tree Search, isas the name implies -a tree search algorithm. It operates on a sequence "snapshots" of the evolving MDP, which are instances of the MDP at points in time. The algorithm is tested on small grid world, and does not appear to be readily extendible to the continuous state and action scenarios our algorithm addresses. To summarise, our paper uses the Wasserstein distance for quantifying variations in possible dynamics, in common with , but is suited to applying deep neural networks for continuous state and action spaces. Our algorithm does not require a full dynamics available to it, merely a parameterisable dynamics. It competes well with the above papers, and operates well for high dimensional problems, as evidenced by the experiments. In this paper, we proposed a robust reinforcement learning algorithm capable of outperforming others in terms of test returns on unseen dynamics. The algorithm makes use of Wasserstein constraints for policies generalising across varying domains, and considers a zero-order method for scalable solutions. Empirically, we demonstrated superior performance against state-of-the-art from both standard and robust reinforcement learning on low and high-dimensional MuJuCo environments. In future work, we aim to consider robustness in terms of other components of MDPs, e.g., state representations, reward functions, and others. Furthermore, we will implement WR 2 L on real hardware, considering sim-to-real experiments. -Sub-Case III when indices are all distinct: We have Diagonal Elements Conclusion: Using the above we conclude that • Off-Diagonal Elements (i.e., when i = j): The above analysis is now repeated for computing the expectation of the off-diagonal elements of matrix B. Similarly, this can also be split into three sub-cases depending on indices: -Sub-Case III when indices are all distinct: We have Off-Diagonal Elements Conclusion: Using the above and due to the symmetric properties of H, we conclude that Finally, analysing c, one can realise that Substituting the above back in the original approximation in Equation 11, and using the linearity of the expectation we can easily achieve the statement of the proposition. For clarity, we summarise variables parameterising dynamics in Table 1, and detail specifics next. CartPole: The goal of this classic control benchmark is to balance a pole by driving a cart along a rail. The state space is composed of the position x and velocityẋ of the cart, as well as the angle θ and angular velocities of the poleθ. We consider two termination conditions in our experiments: 1) pole deviates from the upright position beyond a pre-specified threshold, or 2) cart deviates from its zeroth initial position beyond a certain threshold. To conduct robustness experiments, we parameterise the dynamics of the CartPole by the pole length l p, and test by varying l p ∈ [0.3, 3]. Hopper: In this benchmark, the agent is required to control a hopper robot to move forward without falling. The state of the hopper is represented by positions, {x, y, z}, and linear velocities, {ẋ,ẏ,ż}, of the torso in global coordinate, as well as angles, {θ i} 2 i=0, and angular speeds, {θ i} 2 i=0, of the three joints. During training, we exploit an early-stopping scheme if "unhealthy" states of the robot were visited. Parameters characterising dynamics included densities {ρ i} 3 i=0 of the four links, armature {a i} 2 i=0 and damping {ζ i} 2 i=0 of three joints, and the friction coefficient µ g. To test for robustness, we varied both frictions and torso densities leading to significant variations in dynamics. We further conducted additional experiments while varying all 11 dimensional specification parameters. Walker2D: This benchmark is similar to Hopper except that the controlled system is a biped robot with seven bodies and six joints. Dimensions for its dynamics are extended accordingly as reported in Table 1. Here, we again varied the torso density for performing robustness experiments in the range ρ 0 ∈. Halfcheetah: This benchmark is similar to the above except that the controlled system is a twodimensional slice of a three-dimensional cheetah robot. Parameters specifying the simulator consist of 21 dimensions, with 7 representing densities. In our two-dimensional experiments we varied the torso-density and floor friction, while in high-dimensional ones, we allowed the algorithm to control all 21 variables. Our experiments included training and a testing phases. During the training phase we applied Algorithm 1 for determining robust policies while updating transition model parameters according to the min-max formulation. Training was performed independently for each of the algorithms on the relevant benchmarks while ensuring best operating conditions using hyper-parameter values reported elsewhere (; ;). For all benchmarks, policies were represented using parametrised Gaussian distributions with their means given by a neural network and standard derivations by a group of free parameters. The neural network consisted of two hidden layers with 64 units and hyperbolic tangent activations in each of the layers. The final layer exploited linear activation so as to output a real number. Following the actor-critic framework, we also trained a standalone critic network having the same structure as that of the policy. For each policy update, we rolled-out in the current worst-case dynamics to collect a number of transitions. The number associated to these transitions was application-dependent and varied between benchmarks in the range of 5,000 to 10,000. The policy was then optimised (i.e., Phase II of Algorithm 1) using proximal policy optimization with a generalised advantage estimation. To solve the minimisation problem in the inner loop of Algorithm 1, we sampled a number of dynamics from a diagonal Gaussian distribution that is centered at the current worst-case dynamics model. The number of sampled dynamics and the variance of the sampled distributions depended on both the benchmark itself, and well as the dimensions of the dynamics. Gradients needed for model updates were estimated using the in Propositions 7 and 8. Finally, we terminated training when the policy entropy dropped below an application-dependent threshold. When testing, we evaluated policies on unseen dynamics that exhibited simulator variations as described earlier. We measured performance using returns averaged over 20 episodes with a maximum length of 1,000 time steps on testing environments. We note that we used non-discounted mean episode rewards to compute such averages. Figure 3 shows the robustness of policies on various taks. For a fair comparison, we trained two standard policy gradient methods (TRPO (b) and PPO ), and two robust RL algorithms (RARL , PR-MDP ) with the reference dynamics preset by our algorithm. The range of evaluation parameters was intentionally designed to include dynamics outside of the -Wasserstein ball. Clearly, WR 2 L outperforms all baselines in this benchmark. As mentioned in the experiments section of the main paper, we refrained from presenting involving deep deterministic policy gradients (DDPG) due to its lack in robustness even on simple systems, such as the CartPole. In Section 3.2 we presented a closed form solution to the following optimisation problem: which took the form of:

Represent the TLDR for finding the corresponding scientific article

An RL algorithm that learns to be robust to changes in dynamics

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce two approaches for efficient and scalable inference in stochastic simulators for which the density cannot be evaluated directly due to, for example, rejection sampling loops.


Question: Represent the scientific article to find a short TLDR

Question: Partially observable Markov decision processes (POMDPs) are a natural model for scenarios where one has to deal with incomplete knowledge and random events. Applications include, but are not limited to, robotics and motion planning. However, many relevant properties of POMDPs are either undecidable or very expensive to compute in terms of both runtime and memory consumption. In our work, we develop a game-based abstraction method that is able to deliver safe bounds and tight approximations for important sub-classes of such properties. We discuss the theoretical implications and showcase the applicability of our on a broad spectrum of benchmarks. In offline motion planning, we aim to find a strategy for an agent that ensures certain desired behavior, even in the presence of dynamical obstacles and uncertainties BID0. If random elementslike uncertainty in the outcome of an action or in the movement of dynamic obstacles -need to be taken into account, the natural model for such scenarios are Markov decision processes (MDPs). MDPs are non-deterministic models which allow the agent to perform actions under full knowledge of the current state of the agent its surrounding environment. In many applications, though, full knowledge cannot be assumed, and we have to deal with partial observability BID1. For such scenarios, MDPs are generalized to partially observable MDPs (POMDPs). In a POMDP, the agent does not know the exact state of the environment, but only an observation that can be shared between multiple states. Additional information about the likelihood of being in a certain state can be gained by tracking the observations over time. This likelihood is called the belief state. Using an update function mapping a belief state and an action as well as the newly obtained observation to a new belief state, one can construct a (typically infinite) MDP, commonly known as the belief MDP.While model checking and strategy synthesis for MDPs are, in general, well-manageable problems, POMDPs are much harder to handle and, due to the potentially infinite belief space, many problems are actually undecidable BID2. Our aim is to apply abstraction and abstraction refinement techniques to POMDPs in order to get good and safe approximative for different types of properties. As a case study, we work with a scenario featuring a controllable agent. Within a certain area, the agent needs to traverse a room while avoiding both static obstacles and randomly moving opponents. The area is modeled as a grid, the static obstacles as grid cells that may not be entered. Our assumption for this scenario is that the agent always knows its own position, but the positions of an opponent is only known if its distance from the agent is below a given threshold and if the opponent is not hidden behind a static obstacle. We assume that the opponents move probabilistically. This directly leads to a POMDP model for our case study. For simplification purposes, we only deal with one opponent, although our approach supports an arbitrary number of opponents. We assume the observation function of our POMDPs to be deterministic, but more general POMDPs can easily be simplified to this case. The goal is to find a strategy which maximizes the probability to navigate through the grid from an initial to a target location without collision. For a grid size of n × n cells and one opponent, the number of states in the POMDP is in O(n 4), i. e., the state space grows rapidly with increasing grid size. In order to handle non-trivial grids, we propose an approach using game-based abstraction BID3.Intuitively, we lump together all states that induce the same observation; for each position of the agent, we can distinguish between all states in which the opponent's position is known, but states in which the position is unknown are merged into one far away state BID4. In order to get a safe approximation We show that any strategy computed with our abstraction that guarantees a certain level of safety can be mapped to a strategy for the original POMDP guarantiing at least the same level of safety. In particular, we establish a simulation relation between paths in the probabilistic game and paths in the POMDP. Intuitively, each path in the POMDP can be reproduced in the probabilistic game if the second player resolves the nondeterminism in a certain way. Game-based model checking assumes the non-determinism to be resolved in the worst way possible, so it will provide a lower bound on the level of safety achievable in the actual POMDP. For full proof see BID4. We analyzed the game-based models using the PRISM-games model checker and compared the obtained with the stateof-the-art POMDP model checker PRISM-pomdp BID5, showing that we can handle grids that are considerably larger than what PRISM-pomdp can handle, while still getting schedulers that induce values which are close to optimal. TAB0 shows a few of our experiments for verifying a reach-avoid property on a grid without obstacles. The colums show the probability (computed by the respective method) to reach a goal state without a collision. As one can see, the abstraction approach is faster by orders of magnitude than solving the POMDP directly, and the game model also is much smaller for large grids while still getting very good approximations for the actual probabilities. The strategies induce even better values when they are mapped back to the original POMDP. While being provably sound, our approach is still targeting an undecidable problem and as such not complete in the sense that in general no strategy with maximum probability for success can be deduced. In particular for cases with few paths to the goal location, the gap between the obtained bounds and the actual maximum can become large. For those cases, we define a scheme to refine the abstraction by encoding one or several steps of history into the current state, which leads to larger games and accordingly longer computation times, but also to better . TAB0 showcases an implementation of this one-step history refinement. We use a benchmark representing a long, narrow tunnel, in which the agent has to pass the opponent once, but, due to the abstraction, can actually run into the it repeatedly if the abstraction-player has the opponent re-appear in front of the agent. With longer tunnels, the probability to safely arrive in a goal state diminishes. Adding a refinement which remembers the last known position of the opponent and thus restricting the non-deterministic movement keeps the probability constant for arbitrary length. We developed a game-based abstraction technique to synthesize strategies for a class of POMDPs. This class encompasses typical grid-based motion planning problems under restricted observability of the environment. For these scenarios, we efficiently compute strategies that allow the agent to maneuver the grid in order to reach a given goal state while at the same time avoiding collisions with faster moving obstacles. Experiments show that our approach can handle state spaces up to three orders of magnitude larger than general-purpose state-of-the-art POMDP solvers in less time, while at the same time using fewer states to represent the same grid sizes.

Represent the short summary for finding the corresponding scientific article (long)

This paper provides a game-based abstraction scheme to compute provably sound policies for POMDPs.

Represent the TLDR for finding the corresponding scientific article

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper we approach two relevant deep learning topics: i) tackling of graph structured input data and ii) a better understanding and analysis of deep networks and related learning algorithms. With this in mind we focus on the topological classification of reachability in a particular subset of planar graphs (Mazes). Doing so, we are able to model the topology of data while staying in Euclidean space, thus allowing its processing with standard CNN architectures. We suggest a suitable architecture for this problem and show that it can express a perfect solution to the classification task. The shape of the cost function around this solution is also derived and, remarkably, does not depend on the size of the maze in the large maze limit. Responsible for this behavior are rare events in the dataset which strongly regulate the shape of the cost function near this global minimum. We further identify an obstacle to learning in the form of poorly performing local minima in which the network chooses to ignore some of the inputs. We further support our claims with training experiments and numerical analysis of the cost function on networks with up to $128$ layers. Deep convolutional networks have achieved great success in the last years by presenting human and super-human performance on many machine learning problems such as image classification, speech recognition and natural language processing ). Importantly, the data in these common tasks presents particular statistical properties and it normally rests on regular lattices (e.g. images) in Euclidean space BID3 ). Recently, more attention has been given to other highly relevant problems in which the input data belongs to non-Euclidean spaces. Such kind of data may present a graph structure when it represents, for instance, social networks, knowledge bases, brain activity, protein-interaction, 3D shapes and human body poses. Although some works found in the literature propose methods and network architectures specifically tailored to tackle graph-like input data BID3; BID4; BID15; BID22; BID23 b) ), in comparison with other topics in the field this one is still not vastly investigated. Another recent focus of interest of the machine learning community is in the detailed analysis of the functioning of deep networks and related algorithms BID8; BID12 ). The minimization of high dimensional non-convex loss function by means of stochastic gradient descent techniques is theoretically unlikely, however the successful practical achievements suggest the contrary. The hypothesis that very deep neural nets do not suffer from local minima BID9 ) is not completely proven BID36 ). The already classical adversarial examples BID27 ), as well as new doubts about supposedly well understood questions, such as generalization BID43 ), bring even more relevance to a better understanding of the methods. In the present work we aim to advance simultaneously in the two directions described above. To accomplish this goal we focus on the topological classification of graphs BID29; BID30 ). However, we restrict our attention to a particular subset of planar graphs constrained by a regular lattice. The reason for that is threefold: i) doing so we still touch upon the issue of real world graph structured data, such as the 2D pose of a human body BID1; BID16 ) or road networks BID25; BID39 ); ii) we maintain the data in Euclidean space, allowing its processing with standard CNN architectures; iii) this particular class of graphs has various non-trivial statistical properties derived from percolation theory and conformal field theories BID5; BID20; BID34 ), allowing us to analytically compute various properties of a deep CNN proposed by the authors to tackle the problem. Specifically, we introduce Maze-testing, a specialized version of the reachability problem in graphs BID42 ). In Maze-testing, random mazes, defined as L by L binary images, are classified as solvable or unsolvable according to the existence of a path between given starting and ending points in the maze (vertices in the planar graph). Other recent works approach maze problems without framing them as graphs BID37; BID28; BID33 ). However, to do so with mazes (and maps) is a common practice in graph theory BID2; BID32 ) and in applied areas, such as robotics BID11; BID7 ). Our Mazetesting problem enjoys a high degree of analytical tractability, thereby allowing us to gain important theoretical insights regarding the learning process. We propose a deep network to tackle the problem that consists of O(L 2) layers, alternating convolutional, sigmoid, and skip operations, followed at the end by a logistic regression function. We prove that such a network can express an exact solution to this problem which we call the optimal-BFS (breadth-first search) minimum. We derive the shape of the cost function around this minimum. Quite surprisingly, we find that gradients around the minimum do not scale with L. This peculiar effect is attributed to rare events in the data. In addition, we shed light on a type of sub-optimal local minima in the cost function which we dub "neglect minima". Such minima occur when the network discards some important features of the data samples, and instead develops a sub-optimal strategy based on the remaining features. Minima similar in nature to the above optimal-BFS and neglect minima are shown to occur in numerical training and dominate the training dynamics. Despite the fact the Maze-testing is a toy problem, we believe that its fundamental properties can be observed in real problems, as is frequently the case in natural phenomena BID31 ), making the presented analytical analysis of broader relevance. Additionally important, our framework also relates to neural network architectures with augmented memory, such as Neural Turing Machines BID13 ) and memory networks BID40; BID35 ). The hot-spot images FIG9, used to track the state of our graph search algorithm, may be seen as an external memory. Therefore, to observe how activations spread from the starting to the ending point in the hot-spot images, and to analyze errors and the landscape of the cost function (Sec. 5), is analogous to analyze how errors occur in the memory of the aforementioned architectures. This connection gets even stronger when such memory architectures are employed over graph structured data, to perform task such as natural language reasoning and graph search; BID17; BID14 ). In these cases, it can be considered that their memories in fact encode graphs, as it happens in our framework. Thus, the present analysis may eventually help towards a better understanding of the cost functions of memory architectures, potentially leading to improvements of their weight initialization and optimization algorithms thereby facilitating training BID26 ).The paper is organized as follows: Sec. 2 describes in detail the Maze-testing problem. In Sec. 3 we suggest an appropriate architecture for the problem. In Sec. 4 we describe an optimal set of weights for the proposed architecture and prove that it solves the problem exactly. In Sec. 5 we report on training experiments and describe the observed training phenomena. In Sec. 6 we provide an analytical understanding of the observed training phenomena. Finally, we conclude with a discussion and an outlook. Let us introduce the Maze-testing classification problem. Mazes are constructed as a random two dimensional, L × L, black and white array of cells (I) where the probability (ρ) of having a black cell is given by ρ c = 0.59274, while the other cells are white. An additional image (H 0), called the initial hot-spot image, is provided. It defines the starting point by being zero (Off) everywhere except on a 2 × 2 square of cells having the value 1 (On) chosen at a random position (see FIG9 . A Maze-testing sample (i.e. a maze and a hot-spot image) is labelled Solvable if the ending point, defined as a 2 × 2 square at the center of the maze, is reachable from the starting point (defined by the hot-spot image) by moving horizontally or vertically along black cells. The sample is labelled Unsolvable otherwise. A maze-testing sample consists of a maze (I) and an initial hot-spot image (H0). The proposed architecture processes H0 by generating a series of hot-spot images (Hi>0) which are of the same dimension as H0 however their pixels are not binary but rather take on values between 0 (Off, pale-orange) and 1 (On, red). This architecture can represent an optimal solution, wherein the red region in H0 spreads on the black cluster in I to which it belongs. Once the spreading has exhausted, the Solvable/Unsolvable label is determined by the values of Hn at center (ending point) of the maze. In the above example, the maze in question is Unsolvable, therefore the On cells do not reach the ending point at the center of Hn. A maze in a Maze-testing sample has various non-trivial statistical properties which can be derived analytically based on from percolation theory and conformal field theory BID5; BID20; BID34 ). Throughout this work we directly employ such statistical properties, however we refer the reader to the aforementioned references for further details and mathematical derivations. At the particular value chosen for ρ, the problem is at the percolation-threshold which marks the phase transition between the two different connectivity properties of the maze: below ρ c the chance of having a solvable maze decays exponentially with r (the geometrical distance between the ending and starting points). Above ρ c it tends to a constant at large r. Exactly at ρ c the chance of having a solvable maze decays as a power law (1/r η, η = 5/24). We note in passing that although Mazetesting can be defined for any ρ, only the choice ρ = ρ c leads to a computational problem whose typical complexity increases with L.Maze-testing datasets can be produced very easily by generating random arrays and then analyzing their connectivity properties using breadth-first search (BFS), whose worse case complexity is O(L 2). Notably, as the system size grows larger, the chance of producing solvable mazes decay as 1/L η, and so, for very large L, the labels will be biased towards unsolvable. There are several ways to de-bias the dataset. One is to select an unbiased subset of it. Alternatively, one can gradually increase the size of the starting-point to a starting-square whose length scales as L η. Unless stated otherwise, we simply leave the dataset biased but define a normalized test error (err), which is proportional to the average mislabeling rate of the dataset divided by the average probability of being solvable. Here we introduce an image classification architecture to tackle the Maze-testing problem. We frame maze samples as a subclass of planar graphs, defined as regular lattices in the Euclidean space, which can be handle by regular CNNs. Our architecture can express an exact solution to the problem and, at least for small Mazes (L ≤ 16), it can find quite good solutions during training. Although applicable to general cases, graph oriented architectures find it difficult to handle large sparse graphs due to regularization issues BID15; BID22 ), whereas we show that our architecture can perform reasonably well in the planar subclass. Our network, shown in FIG9, is a deep feedforward network with skip layers, followed by a logistic regression module. The deep part of the network consists of n alternating convolutional and sigmoid layers. Each such layer (i) receives two L × L images, one corresponding to the original maze (I) and the other is the output of the previous layer (H i−1). It performs the operation DISPLAYFORM0, where * denotes a 2D convolution, the K convolutional kernel is 1×1, the K hot kernel is 3×3, b is a bias, and σ(x) = (1+e −x) −1 is the sigmoid function. The logistic regression layer consists of two perceptrons (j = 0, 1), acting on DISPLAYFORM1 where H n is the rasterized/flattened version of H n, W j is a 2 × L 2 matrix, and b reg is a vector of dimension 2. The logistic regression module outputs the label Solvable if p 1 ≥ p 0 and Unsolvable otherwise. The cost function we used during training was the standard negative log-likelihood. As we next show, the architecture above can provide an exact solution to the problem by effectively forming a cellular automaton executing a breadth-first search (BFS). A choice of parameters which achieves this is λ ≥ λ c = 9.727±0.001, DISPLAYFORM0 T, where q center is the index of H n which corresponds to the center of the maze. Let us explain how the above neural network processes the image (see also FIG9). Initially H 0 is On only at the starting-point. Passing through the first convolutional-sigmoid layer it outputs H 1 which will be On (i.e. have values close to one) on all black cells which neighbor the On cells as well as on the original starting point. Thus On regions spread on the black cluster which contains the original starting-point, while white clusters and black clusters which do not contain the starting-point remain Off (close to zero in H i). The final logistic regression layer simply checks whether one of the 2 × 2 cells at the center of the maze are On and outputs the labels accordingly. To formalize the above we start by defining two activation thresholds, v l and v h, and refer to activations which are below v l as being Off and to those above v h as being On. The quantity v l is defined as the smallest of the three real solutions of the equation v l = σ(5v l − 0.5λ). Notably we previously chose λ > λ c as this is the critical value above which three real solutions to v l (rather than one) exist. For v h we choose 0.9.Next, we go case by case and show that the action of the convolutional-sigmoid layer switches activations between Off and On just as a BFS would. This amounts to bounding the expression σ(K hot * H i−1 + K * I + b) for all possibly 3 × 3 sub-arrays of H i−1 and 1 × 1 sub-arrays of I. There are thus 2 10 possibilities to be examined. FIG1 shows the desired action of the layer on three important cases (A-C). Each case depicts the maze shape around some arbitrary point x, the hot-spot image around x before the action of the layer (H i−1), and the desired action of the layer (H i). Case A. Having a white cell at x implies I[x] = 0 and therefore the argument of the above sigmoid is smaller than −0.5λ c this regardless of H i−1 at and around x. Thus H i [x] < v l and so it is Off. As the 9 activations of H i−1 played no role, case A covers in fact 2 9 different cases. Case B. Consider a black cell at x, with H i−1 in its vicinity all being Off (vicinity here refers to x and its 4 neighbors). Here the argument is smaller or equal to 5v l − 0.5λ c, and so the activation remains Off as desired. Case B covers 2 4 cases as the values of H i−1 on the 4 corners were irrelevant. Case C. Consider a black cell at x with one or more On activations of H i−1 in its vicinity. Here the argument is larger than v h λ c − 0.5λ c = 0.4λ c. The sigmoid is then larger than 0.97 implying it is On. Case C covers 2 4 (2 5 − 1) different cases. Since 2 9 + 2 4 + 2 4 (2 5 − 1) = 2 10 we exhausted all possible cases. Lastly it can be easily verified that given an On (Off) activation at the center of the full maze the logistic regression layer will output the label Solvable (Unsolvable).Let us now determine the required depth for this specific architecture. The previous analysis tells us that at depth d unsolvable mazes would always be labelled correctly however solvable mazes would be label correctly only if the shortest-path between the starting-point and the center is d or less. The worse case scenario thus occurs when the center of the maze and the starting-point are connected by a one dimensional curve twisting its way along O(L 2) sites. Therefore, for perfect performance the network depth would have to scale as the number of sites namely n = O(L 2). A tighter but probabilistic bound on the minimal depth can be established by borrowing various from percolation theory. It is known, from BID44, that the typical length of the shortest path (l) for critical percolation scales as r dmin, where r is the geometrical distance and d min = 1.1. Moreover, it is known that the probability distribution P (l|r) has a tail which falls as l −2 for l ≈> r dmin BID10 ). Consequently, the chance that at distance r the shortest path is longer than r dmin r a, where a is some small positive number, decays to zero and so, d should scale as L with a power slightly larger than d min (say n = L 1.2). We have performed several training experiments with our architecture on L = 16 and L = 32 mazes with depth n = 16 and n = 32 respectively, datasets of sizes M = 1000, M = 10000, and M = 50000. Unless stated otherwise we used a batch size of 20 and a learning rate of 0.02. In the following, we split the experiments into two different groups corresponding to the related phenomena observed during training, which will the analyzed in detail in the next section. Optimal-BFS like minima. For L = 16, M = 10000 mazes and a positive random initialization for K hot and K in the network found a solution with ≈ 9% normalized test error performance in 3 out of the 7 different initializations (baseline test error was 50%). In all three successful cases the minima was a variant of the Optimal-BFS minima which we refer to as the checkerboard-BFS minima. It is similar to the optimal-BFS but spreads the On activations from the starting-point using a checkerboard pattern rather than a uniform one, as shown in FIG2. The fact that it reaches ≈ 9% test error rather than zero is attributed to this checkerboard behavior which can occasionally miss out the exit point from the maze. Neglect minima. Again for L = 16 but allowing for negative entries in K and K hot test error following 14 attempts and 500 epochs did not improve below 44%. Analyzing the weights of the network, the 6% improvement over the baseline error (50%) came solely from identifying the inverse correlation between many white cells near the center of the maze and the chance of being solvable. Notably, this heuristic approach completely neglects information regarding the starting-point of the maze. For L = 32 mazes, despite trying several random initialization strategies including positive entries, dataset sizes, and learning rates, the network always settled into such a partial neglect minimum. In an unsuccessful attempt to push the weights away from such partial neglect behavior, we performed further training experiments with a biased dataset in which the maze itself was uncorrelated with the label. More accurately, marginalizing over the starting-point there is an equal chance for both labels given any particular maze. To achieve this, a maze shape was chosen randomly and then many random locations were tried-out for the starting-point using that same maze. From these, we picked 5 that ed in a Solvable label and 5 that ed in an Unsolvable label. Maze shapes which were always Unsolvable were discarded. Both the L = 16 and L = 32 mazes trained on this biased dataset performed poorly and yielded 50% test error. Interestingly they improved their cost function by settling into weights in which b ≈ −10 is large compared to [K hot] ij <≈ 1 while W and b were close to zero (order of 0.01). We have verified that such weights imply that activations in the last layer have a negligible dependence on the starting-point and a weak dependence on the maze shape. We thus refer to this minimum as a "total neglect minimum". Here we seek an analytical understanding of the aforementioned training phenomena through the analysis of the cost function around solutions similar or equal to those the network settled into during training. Specifically we shall first study the cost function landscape around the optimal-BFS minimum. As would become clearer at the end of that analysis, the optimal BFS shares many similarities with the checkerboard-BFS minimum obtained during training and one thus expects a similar cost function landscape around both of these. The second phenomena analyzed below is the total neglect minimum obtained during training on the biased dataset. The total neglect minimum can be thought of as an extreme version of the partial neglect minima found for L = 32 in the original dataset. Our analysis of the cost function near the optimal-BFS minimum will be based on two separate models capturing the short and long scale behavior of the network near this miminum. In the first model we approximate the network by linearizing its action around weak activations. This model would enable us to identify the density of what we call "bugs" in the network. In the second model we discretize the activation levels of the neural network into binary variables and study how the ing cellular automaton behaves when such bugs are introduced. FIG3 shows a numerical example of the dynamics we wish to analyze. Up to layer 19 (H19) the On activations spread according to BFS however at H20 a very faint localized unwanted On activation begins to develop (a bug) and quickly saturates (H23). Past this point BFS dynamics continues normally but spreads both the original and the unwanted On activations. While not shown explicitly, On activations still appear only on black maze cells. Notably the bug developed in rather large black region as can be deduced from the large red region in its origin. See also a short movie showing the occurrence of this bug at https://youtu.be/2I436BVAVdM and more bugs at https://youtu.be/kh-AfOo4TkU.At https://youtu.be/t-_TDkt3ER4 a similar behavior is shown for the checkerboard-BFS. Unlike an algorithm, a neural network is an analog entity and so a-priori there are no sharp distinctions between a functioning and a dis-functioning neural network. An algorithm can be debugged and the bug can be identified as happening at a particular time step. However it is unclear if one can generally pin-point a layer and a region within where a deep neural network clearly malfunctioned. Interestingly we show that in our toy problem such pin-pointing of errors can be done in a sharp fashion by identifying fast and local processes which cause an unwanted switching been Off and On activations in H i (see FIG3). We call these events bugs, as they are local, harmful, and have a sharp meaning in the algorithmic context. Below we obtain asymptotic expressions for the chance of generating such bugs as the network weights are perturbed away from the optimal-BFS minimum. The main of this subsection, derived below, is that the density of bugs (or chance of bug per cell) scales as DISPLAYFORM0 for (λ − λ c) <≈ 0 and zero for λ − λ c >= 0 where C ≈ 1.7. Following the analysis below, we expect the same dependence to hold for generic small perturbations only with different C and λ c. We have tested this claim numerically on several other types of perturbations (including ones that break the π/2 rotational symmetry of the weights) and found that it holds. To derive Eq., we first recall the analysis in Sec. 4, initially as it is decreased λ has no effect but to shift v l (the Off activation threshold) to a higher value. However, at the critical value (λ = λ c, v l = v l,c) the solution corresponding to v l vanishes (becomes complex) and the correspondence with the BFS algorithm no longer holds in general. This must not mean that all Off activations are no longer stable. Indeed, recall that in Sec. 4 the argument that a black Off cell in the vicinity of Off cells remains Off FIG1, Case B) assumed a worse case scenario in which all the cells in its vicinity where both Off, black, and had the maximal Off activation allowed (v l). However, if some cells in its vicinity are white, their Off activations levels are mainly determined by the absence of the large K term in the sigmoid argument and orders of magnitude smaller than v l. We come to the that black Off cells in the vicinity of many white cells are less prone to be spontaneously turned On than black Off cells which are part of a large cluster of black cells (see also the bug in FIG3). In fact using the same arguments one can show that infinitesimally below λ c only uniform black mazes will cause the network to malfunction. To further quantify this, consider a maze of size l × l where the hot-spot image is initially all zero and thus Off. Intuitively this hot-spot image should be thought of as a sub-area of a larger maze located far away from the starting-point. In this case a functioning network must leave all activation levels below v l. To assess the chance of bugs we thus study the probability that the output of the final convolutional-sigmoid layer will have one or more On cells. To this end, we find it useful to linearize the system around low activation yielding (see the Appendix for a complete derivation) DISPLAYFORM1 where r b denotes black cells (I(r b) = 1), the sum is over the nearest neighboring black cells to r b, DISPLAYFORM2 For a given maze (I), Eq., defines a linear Hermitian operator (L I) with random off-diagonal matrix elements dictated by I via the restriction of the off-diagonal terms to black cells. Stability of Off activations is ensured if this linear operator is contracting or equivalently if all its eigenvalues are smaller than 1 in magnitude. Hermitian operators with local noisy entries have been studied extensively in physics, in the context of disordered systems and Anderson localization BID19 ). Let us describe the main relevant . For almost all I's the spectrum of L consists of localized eigenfunctions (φ m). Any such function is centered around a random site (x m) and decays exponentially away from that site with a decay length of χ which in our case would be a several cells long. Thus given φ m with an eigenvalue |E m | > 1, t repeated actions of the convolutional-sigmoid layer will make ψ n [x] in a χ vicinity of x m grow in size as e Emt. Thus (|E m | − 1) −1 gives the characteristic time it takes these localized eigenvalue to grow into an unwanted localized region with an On activation which we define as a bug. Our original question of determining the chance of bugs now translates into a linear algebra task: finding, Nλ, the number of eigenvalues in L I which are larger than 1 in magnitude, averaged over I, for a given λ. Sinceλ simply scales all eigenvalues one finds that Nλ is the number of eigenvalues larger thanλ −1 in L I withλ = 1. Analyzing this latter operator, it is easy to show that the maximal eigenvalues occurs when φ n (r) has a uniform pattern on a large uniform region where the I is black. Indeed if I contains a black uniform true box of dimension l u × l u, the maximal eigenvalue is easily shown to be E lu = 5 − 2π 2 /(l u) 2. However the chance that such a uniform region exists goes as (l/l u) 2 e log(ρc)l 2 u and so P (∆E) ∝ l 2 e log(ρc)2π 2 (∆E), where ∆E = 5 − E. This reasoning is rigorous as far as lower bounds on Nλ are concerned, however it turns out to capture the functional behavior of P (∆E) near ∆E = 0 accurately BID18 ) which is given by ∆E), where the unknown constant C captures the dependence on various microscopic details. In the Appendix we find numerically that C ≈ 0.7. Following this we find DISPLAYFORM3 DISPLAYFORM4 The range of integration is chosen to includes all eigenvalues which, following a multiplication byλ, would be larger than 1.To conclude we found the number of isolated unwanted On activations which develop on l × l Off regions. Dividing this number by l 2 we obtain the density of bugs (ρ bug) near λ ≈ λ c. The last technical step is thus to express ρ bug in terms of λ. Focusing on the small ρ bug region or ∆E → 0 +, we find that ∆E = 0 occurs when dσ dx (σ −1 (η ∞ (λ))) = 1/(5λ),λ = 1/5, and λ = λ c = 9.72.Expanding around λ = λ c we find DISPLAYFORM5 2 ). Approximating the integral over P (x) and taking the leading scale dependence, we arrive at Eq. FORMULA3 with C = C 10λc 49−λc. In this subsection we wish to understand the large scale effect of ρ bug namely, its effect on the test error and the cost function. Our key here are that DISPLAYFORM0 DISPLAYFORM1 despite its appearance it can be verified that the above right hand side is smaller than L −5/48 within its domain of applicability. To derive Eqs. FORMULA9 and FORMULA10, we note that as a bug is created in a large maze, it quickly switches On the cells within the black "room" in which it was created. From this region it spreads according to BFS and turns On the entire cluster connected to the buggy room (see FIG3). To asses the effect this bug has on performance first note that solvable mazes would be labelled Solvable regardless of bugs however unsolvable mazes might appear solvable if a bug occurs on a cell which is connected to the center of the maze. Assuming we have an unsolvable maze, we thus ask what is the chance of it being classified as solvable. Given a particular unsolvable maze instance (I), the chance of classifying it as solvable is given by p err (I) = 1 DISPLAYFORM2 where s counts the number of sites in the cluster connected to the central site (central cluster). The probability distribution of s for percolation is known and given by p(s) = Bs 1−τ, τ = 187/91 BID6 ), with B being an order of one constant which depends on the underlying lattice. Since clusters have a fractional dimension, the maximal cluster size is L d f. Consequently, p err (I) averaged over all I instances is given by DISPLAYFORM3 ds, which can be easily expressed in terms of Gamma functions (Γ(x), Γ(a, x)) (see BID0). In the limit of ρ bug <≈ L −d f, where its derivatives with respect to ρ bug are maximal, it simplifies to DISPLAYFORM4 whereas for ρ bug > L −d f, its behavior changes to p err = (−BΓ(2 − τ))ρ DISPLAYFORM5 bug. Notably once ρ bug becomes of order one, several of the approximation we took break down. Let us relate p err to the test error (err). In Sec. the cost function was defined as the mislabeling chance over the average chance of being solvable (p solvable). Following the above discussion the mislabelling chance is p err p solvable and consequently err = p err. Combining Eqs. 1 and 5 we obtain our key , Eqs.As a side, one should appreciate a potential training obstacle that has been avoided related to the fact that err ∝ ρ 5/91 big. Considering L → ∞, if ρ bug was simply proportional to (λ c − λ), err will have a sharp singularity near zero. For instance, as one reduces err by a factor of 1/e, the gradients increase by e 86/5 ≈ 3E + 7. These effects are in accordance with ones intuition that a few bugs in a long algorithm will typically have a devastating effect on performance. Interestingly however, the essential singularity in ρ bug (λ), derived in the previous section, completely flattens the gradients near λ c.Thus the essentially singularity which comes directly from rare events in the dataset strongly regulates the test error and in a related way the cost function. However it also has a negative side-effect concerning the robustness of generalization. Given a finite dataset the rarity of events is bounded and so having λ < λ c may still provide perfect performance. However when encountering a larger dataset some samples with rarer events (i.e. larger black region) would appear and the network will fail sharply on these (i.e. the wrong prediction would get a high probability). Further implications of this dependence on rare events on training and generalization errors will be studied in future work. To provide an explanation for this phenomena let us divide the activations of the upper layer to its starting-point dependent and independent parts. Let H n denote the activations at the top layer. We expand them as a sum of two functions DISPLAYFORM0 where the function A and B are normalized such that their variance on the data (α and β, respectively) is 1. Notably near the reported total neglect minima we found that α β ≈ e −10. Also note that for the biased dataset the maze itself is uncorrelated with the labels and thus β can be thought of as noise. Clearly any solution to the Maze testing problem requires the starting-point dependent part (α) to become larger than the independent part (β). We argue however that in the process of increasing α the activations will have to go through an intermediate "noisy" region. In this noisy region α grows in magnitude however much less than β and in particular obeys α < β 2. As shown in the Appendix the negative log-likelihood, a commonly used cost function, is proportional to β 2 − α for α, β 1. Thus it penalizes random false predictions and, within a region obeying α < β 2 it has a minimum (global with respect to that region) when α = β = 0. The later being the definition of a total neglect minima. Establishing the above α β 2 conjecture analytically requires several pathological cases to be examined and is left for future work. In this work we provide an argument for its typical correctness along with supporting numerics in the Appendix. A deep convolution network with a finite kernel has a notion of distance and locality. For many parameters ranges it exhibits a typical correlation length (χ). That is a scale beyond which two activations are statistically independent. Clearly to solve the current problem χ has to grow to an order of L such that information from the input reaches the output. However as χ gradually grows, relevant and irrelevant information is being mixed and propagated onto the final layer. While β depends on information which is locally accessible at each layer (i.e. the maze shape), α requires information to travel from the first layer to the last. Consequently α and β are expected to scale differently, as e −L/χ and e −1/χ resp. (for χ << L). Given this one finds that α β 2 as claimed. Further numerical support of this conjecture is shown in the Appendix where an upper bound on the ratio α/β 2 is studied on 100 different paths leading from the total neglect miminum found during training to the checkerboard-BFS minimum. In all cases there is a large region around the total neglect minimum in which α β 2. Despite their black-box reputation, in this work we were able to shed some light on how a particular deep CNN architecture learns to classify topological properties of graph structured data. Instead of focusing our attention on general graphs, which would correspond to data in non-Euclidean spaces, we restricted ourselves to planar graphs over regular lattices, which are still capable of modelling real world problems while being suitable to CNN architectures. We described a toy problem of this type (Maze-testing) and showed that a simple CNN architecture can express an exact solution to this problem. Our main contribution was an asymptotic analysis of the cost function landscape near two types of minima which the network typically settles into: BFS type minima which effectively executes a breadth-first search algorithm and poorly performing minima in which important features of the input are neglected. Quite surprisingly, we found that near the BFS type minima gradients do not scale with L, the maze size. This implies that global optimization approaches can find such minima in an average time that does not increase with L. Such very moderate gradients are the of an essential singularity in the cost function around the exact solution. This singularity in turn arises from rare statistical events in the data which act as early precursors to failure of the neural network thereby preventing a sharp and abrupt increase in the cost function. In addition we identified an obstacle to learning whose severity scales with L which we called neglect minima. These are poorly performing minima in which the network neglects some important features relevant for predicting the label. We conjectured that these occur since the gradual incorporation of these important features in the prediction requires some period in the training process in which predictions become more noisy. A "wall of noise" then keeps the network in a poorly performing state. It would be interesting to study how well the and lessons learned here generalize to other tasks which require very deep architectures. These include the importance of rare-events, the essential singularities in the cost function, the localized nature of malfunctions (bugs), and neglect minima stabilized by walls of noise. These conjectures potentially could be tested analytically, using other toy models as well as on real world problems, such as basic graph algorithms (e.g. shortest-path) BID14 ); textual reasoning on the bAbI dataset ), which can be modelled as a graph; and primitive operations in "memory" architectures (e.g. copy and sorting) BID13 ). More specifically the importance of rare-events can be analyzed by studying the statistics of errors on the dataset as it is perturbed away from a numerically obtained minimum. Technically one should test whether the perturbation induces an typical small deviation of the prediction on most samples in the dataset or rather a strong deviation on just a few samples. Bugs can be similarly identified by comparing the activations of the network on the numerically obtained minimum and on some small perturbation to that minimum while again looking at typical versus extreme deviations. Such an analysis can potentially lead to safer and more robust designs were the network fails typically and mildly rather than rarely and strongly. Turning to partial neglect minima these can be identified provided one has some prior knowledge on the relevant features in the dataset. The correlations or mutual information between these features and the activations at the final layer can then be studied to detect any sign of neglect. If problems involving neglect are discovered it may be beneficial to add extra terms to the cost function which encourage more mutual information between these neglected features and the labels thereby overcoming the noise barrier and pushing the training dynamics away from such neglect minimum. We have implemented the architecture described in the main text using Theano BID38 ) and tested how cost changes as a function of δ = λ c − λ (λ c = 9.727..) for mazes of sizes L = 24, 36 and depth (number of layers) 128. These depths are enough to keep the error rate negligible at δ = 0. A slight change made compared to Maze-testing as described in the main text, is that the hot-spot was fixed at a distance L/2 for all mazes. The size of the datasets was between 1E + 5 and 1E + 6. We numerically obtained the normalized performance (cost L (δ)) as a function of L and δ. As it follows from Eq. in the main text the curve, log(L −2+5/24 cost L (δ)), for the L = 24 and L = 36 should collapse on each other for ρ bug < L −d f. FIG4 ) of the main-test depicts three such curves, two for L = 36, to give an impression of statistical error, and one for L = 24 curve (green), along with the fit to the theory (dashed line). The fit, which involves two parameters (the proportionally constant in Eq. of the main text and C) captures well the behavior over three orders of magnitude. As our are only asymptotic, both in the sense of large L and λ → λ c, minor discrepancies are expected. To prepare the action of sigmoid-convolutional for linearization we find it useful to introduce the following variables on locations (r b, r w) with black (b) and white (w) cells DISPLAYFORM0 a(r w) = e −5.5λ.homogeneous. Consequently destabilization occurs exactly atλ = 1/5 and is not blurred by the inhomogeneous terms. Recall that λ c is defined as the value at which the two lower solutions of. It is now easy to verify that even within the linear approximation destabilization occurs exactly at λ c. The source of this agreement is the fact that d vanishes for a uniform black maze. The qualitative lesson here is thus the following: The eigenvectors of S with large s, are associated with large black regions in the maze. It is only on the boundaries of such regions that d is non-zero. Consequently near λ ≈ λ c the d term projected on the largest eigenvalues can, to a good accuracy, be ignored and stability analysis can be carried on the homogeneous equation ψ = S ψ where s n < 1 means stability and s n > 1 implies a bug. Consider an abstract classification tasks where data point x ∈ X are classified into two categories l ∈ {0, 1} using a deterministic function f: X → {0, 1} and further assume for simplicity that the chance of f (x) = a is equal to f (x) = b. Phrased as a conditional probability distribution P f (l|x) is given by P f (f (a)|x) = 1 while P f (!f (a)|x) = 0. Next we wish to compare the following family of DISPLAYFORM0 where g|X → {0, 1} is a random function, uncorrelated with f (x), outputting the labels {0, 1} with equal probability. Notably at α = 1/2, β = 0 it yields P f while at α, β = 0 it is simply the maximum entropy distribution. Let us measure the log-likelihood of P α,β under P f for α, β 1 L(α, β) = We thus find that β reduces the log-likelihood in what can be viewed as a penalty to false confidence or noise. Assuming, as argued in the main text, that α is constrained to be smaller than β 2 near β ≈ 0, it is preferable to take both α and β to zero and reach the maximal entropy distribution. We note by passing that the same arguments could be easily generalized to f (x), g(x) taking real values leading again to an O(α) − O(β 2) dependence in the cost function. Let us relate the above notations to the ones in the main text. Clearly x = ({I}, H 0 ) and {0, 1} = {U nsolvable, Solvable}. Next we recall that in the main text α and β multiplied the vectors function representing the H 0 -depended and H 0 -independent parts of H n. The probability estimated by the logistic regression module was given by P (Solvable|x) = e K Solvable · Hn e − K Solvable · Hn + e − K U nsolvable · Hn P (U nsolvable|x) = e K U nsolvable · Hn e − K Solvable · Hn + e − K U nsolvable · Hn which yields, to leading order in α and β P α,β (l|x) = 1/2 + α(2l + 1) DISPLAYFORM1 where K − = (K Solvable − K U nsolvable)/2 and (2l + 1) understood as the taking the values ±1.Consequently (2f − 1) and (2g − 1) are naturally identified with K Solvable · A/N A and K Solvable · B/N B respectively with N A and N B being normalization constants ensuring a variance of 1. While (α, β) = (N A α, N B β). Recall also that by construction of the dataset, the g we thus obtain is uncorrelated with f. 2 CONJECTUREHere we provide numerical evidence showing that α β 2 in a large region around the total neglect minima found during the training of our architecture on the biased dataset (i.e. the one where marginalizing over the starting-point yields a 50/50 chance of being solvable regardless of the maze shape).For a given set of K h ot, K and b parameters we fix the maze shape and study the variance of the top layer activations given O different starting points. We pick the maximal of these and then average this maximal variance over O different mazes. This yields our estimate of α. In fact it is an upper bound on α as this averaged-max-variance may reflect wrong prediction provided that they depend on H 0.We then obtain an estimate of β by again calculating the average-max-variance of the top layer however now with H 0 = 0 for all maze shapes. Next we chose a 100 random paths parametrized by γ leading from the total neglect minima (γ = 0) for the total neglect through a random point at γ = 15, and then to the checkerboard-BFS minima at γ = 30. The random point was placed within a hyper-cube of length 4 having the total neglect minima at its center. The path was a simple quadratic interpolation between the three point. The graph below shows the statistics of α/β 2 on these 100 different paths. Notably no path even had α > e −30 β 2 within the hyper-cube. We have tried three different other lengths for the hyper cube (12 and 1) and arrived at the same . The natural logarithm of an upper bound to α/β 2 as a function of a paramterization (γ) of a path leading from the numerically obtained total neglect minima to the checkerboard BFS minima through a random point. The three different curves show the max,mean, and median based on a 100 different paths. Notably no path violated the α β 2 constrain in the vicinity of the total neglect minima.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

A toy dataset based on critical percolation in a planar graph provides an analytical window to the training dynamics of deep neural networks

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.