Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: State-of-the-art in imitation learning are currently held by adversarial methods that iteratively estimate the divergence between student and expert policies and then minimize this divergence to bring the imitation policy closer to expert behavior. Analogous techniques for imitation learning from observations alone (without expert action labels), however, have not enjoyed the same ubiquitous successes. Recent work in adversarial methods for generative models has shown that the measure used to judge the discrepancy between real and synthetic samples is an algorithmic design choice, and that different choices can in significant differences in model performance. Choices including Wasserstein distance and various $f$-divergences have already been explored in the adversarial networks literature, while more recently the latter class has been investigated for imitation learning. Unfortunately, we find that in practice this existing imitation-learning framework for using $f$-divergences suffers from numerical instabilities stemming from the combination of function approximation and policy-gradient reinforcement learning. In this work, we alleviate these challenges and offer a reparameterization of adversarial imitation learning as $f$-divergence minimization before further extending the framework to handle the problem of imitation from observations only. Empirically, we demonstrate that our design choices for coupling imitation learning and $f$-divergences are critical to recovering successful imitation policies. Moreover, we find that with the appropriate choice of $f$-divergence, we can obtain imitation-from-observation algorithms that outperform baseline approaches and more closely match expert performance in continous-control tasks with low-dimensional observation spaces. With high-dimensional observations, we still observe a significant gap with and without action labels, offering an interesting avenue for future work. Imitation Learning (IL) refers to a paradigm of reinforcement learning in which the learning agent has access to an optimal, reward-maximizing expert for the underlying environment. In most work, this access is provided through a dataset of trajectories where each observed state is annotated with the action prescribed by the expert policy. This is often an extremely powerful learning paradigm in contrast to standard reinforcement learning, since not all tasks of interest admit easily-specified reward functions. Additionally, not all environments are amenable to the prolonged and potentially unsafe exploration needed for reward-maximizing agents to arrive at satisfactory policies . While the traditional formulation of the IL problem assumes access to optimal expert action labels, the provision of such information can often be laborious (in the case of a real, human expert) or incur significant financial cost (such as using elaborate instrumentation to record expert actions). Additionally, this restrictive assumption removes a vast number of rich, observation-only data sources from consideration . To bypass these challenges, recent work (; a; b; ; has explored what is perhaps a more natural problem formulation in which an agent must recover an imitation policy from a dataset containing only expert observation sequences. While this Imitation Learning from Observations (ILfO) setting carries tremendous potential, such as enabling an agent to learn complex tasks from watching freely available videos on the Internet, it also is fraught with significant additional challenges. In this paper, we show how to incorporate recent advances in generative-adversarial training of deep neural networks to tackle imitation-learning problems and advance the state-of-the-art in ILfO. With these considerations in mind, the overarching goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels. Figure 1: Evaluating the original f -VIM framework (and its ILfO counterpart, f -VIMO) in the Ant (S ∈ R 111) and Hopper (S ∈ R 11) domains with the Total Variation distance. f -VIM/VIMO-sigmoid denotes our instantiation of the frameworks, detailed in Sections 4.2 and 4.3. Note that, in both plots, the lines for TV-VIM and TV-VIMO overlap. The rich literature on Generative Adversarial Networks has expanded in recent years to include alternative formulations of the underlying objective that yield qualitatively different solutions to the saddle-point optimization problem;;. Of notable interest are the findings of who present Variational Divergence Minimization (VDM), a generalization of the generative-adversarial approach to arbitrary choices of distance measures between probability distributions drawn from the class of f -divergences (; . Applying VDM with varying choices of fdivergence, encounter learned synthetic distributions that can exhibit differences from one another while producing equally realistic samples. Translating this idea for imitation is complicated by the fact that the optimization of the generator occurs via policy-gradient reinforcement learning . Existing work in combining adversarial IL and f -divergences, despite being well-motivated, fails to account for this difference; the end (shown partially in Figure 1, where TV-VIM is the method of, and discussed further in later sections) are imitation-learning algorithms that scale poorly to environments with higher-dimensional observations. In this work, we assess the effect of the VDM principle and consideration of alternative fdivergences in the contexts of IL and ILfO. We begin by reparameterizing the framework of for the standard IL problem. Our version transparently exposes the choices practitioners must make when designing adversarial imitation algorithms for arbitrary choices of f -divergence. We then offer a single instantiation of our framework that, in practice, allows stable training of good policies across multiple choices of f -divergence. An example is illustrated in Figure 1 where our methods (TV-VIM-sigmoid and TV-VIMO-sigmoid) in significantly superior policies. We go on to extend our framework to encapsulate the ILfO setting and examine the efficacy of the ing new algorithms across a range of continuous-control tasks in the MuJoCo domain. Our empirical validate our framework as a viable unification of adversarial imitation methods under the VDM principle. With the assistance of recent advances in stabilizing regularization for adversarial training , improvements in performance can be attained under an appropriate choice of f -divergence. However, there is still a significant performance gap between the recovered imitation policies and expert behavior for tasks with high dimensional observations, leaving open directions for future work in developing improved ILfO algorithms. The algorithms presented in this work fall in with inverse reinforcement learning (IRL) (Ng et al.; ; ; ; ;) approaches to IL. Early successes in this regime tend to rely on hand-engineered feature rep-resentations for success (; ;). Only in recent years, with the aid of deep neural networks, has there been a surge in the number of approaches that are capable of scaling to the raw, high-dimensional observations found in real-world control problems (; ; ; ; ;). Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive experts (; ; ;); at the heart of these methods is the Generative Adversarial Imitation Learning (GAIL) approach which produces high-fidelity imitation policies and achieves state-of-the-art across numerous continuous-control benchmarks by leveraging the expressive power of Generative Adversarial Networks (GANs) for modeling complex distributions over a high-dimensional support. From an IRL perspective, GAIL can be viewed as iteratively optimizing a parameterized reward function (discriminator) that, when used to optimize an imitation policy (generator) via policy-gradient reinforcement learning , allows the agent to shift its own behavior closer to that of the expert. From the perspective of GANs, this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert. While a large body of prior work exists for IL, recent work has drawn attention to the more challenging problem of imitation learning from observation (; ; ; ; a; b; ; . To more closely resemble observational learning in humans and leverage the wealth of publiclyavailable, observation-only data sources, the ILfO problem considers learning from expert demonstration data where no expert action labels are provided. Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via reinforcement learning (; ;). In contrast, Torabi et al. (2018a) introduce Behavioral Cloning from Observation (BCO) which leverages state-action trajectories collected under a random policy to train an inverse dynamics model for inferring the action responsible for a transition between two input states (assuming the two represent a state and next-state pair). With this inverse model in hand, the observation-only demonstration data can be converted into the more traditional dataset of state-action pairs over which standard BC can be applied. Recognizing the previously discussed limitations of BC approaches, Torabi et al. (2018b) introduce the natural GAIL counterpart for ILfO, Generative Adversarial Imitation from Observation (GAIFO); GAIFO is identical to GAIL except the distributions under consideration in the adversarial game are over state transitions (state and next-state pairs), as opposed to stateaction pairs requiring expert action labels. While Torabi et al. (2018b) offer empirical for continuous-control tasks with low-dimensional features as well as raw image observations, GAIFO falls short of expert performance in both settings leaving an open challenge for scalable ILfO algorithms that achieve expert performance across a wide spectrum of tasks. A central question of this work is to explore how alternative formulations of the GAN objective that underlies GAIFO might yield superior ILfO algorithms. For a more in-depth survey of ILfO approaches, we refer readers to. We refer readers to the Appendix for a broader overview of prior work. We begin by formulating the problems of imitation learning and imitation learning from observation respectively before taking a closer look at f -divergences and connecting them to imitation learning. We operate within the Markov Decision Process (MDP) formalism defined as a five-tuple M = S, A, R, T, γ where S denotes a (potentially infinite) set of states, A denotes a (potentially infinite) set of actions, R: S × A × S → R is a reward function, T: S × A → ∆(S) is a transition function, and γ ∈ is a discount factor. At each timestep, the agent observes the current state of the world, s t ∈ S, and randomly samples an action according to its stochastic policy π: S → ∆(A). The environment then transitions to a new state according to the transition function T and produces a reward signal according to the reward function R that is communicative of the agent's progress through the overall task. Unlike, the traditional reinforcement learning paradigm, the decision-making problem presented in IL lacks a concrete reward function; in lieu of R, a learner is provided with a dataset of expert demonstrations D = {τ 1, τ 2, . . . τ N} where each τ i = (s i1, a i1, s i2, a i2, . . .) represents the sequence of states and corresponding actions taken by an expert policy, π *. Naturally, the goal of an IL algorithm is to synthesize a policy π using D, along with access to the MDP M, whose behavior matches that of π *. While the previous section outlines several possible avenues for using D to arrive at a satisfactory imitation policy, our work focuses on adversarial methods that build around GAIL . Following from the widespread success of GANs , GAIL offers a highly-performant approach to IL wherein, at each iteration of the algorithm, transitions sampled from the current imitation policy are first used to update a discriminator, D ω (s, a), that acts a binary classifier to distinguish between state-action pairs sampled according to the distributions induced by the expert and student. Subsequently, treating the imitation policy as a generator, policy-gradient reinforcement learning is used to shift the current policy towards expert behavior, issuing higher rewards for those generated state-action pairs that are regarded as belonging to the expert according to D ω (s, a). More formally, this minimax optimization follows as where ρ π * (s, a) and ρ π (s, a) denote the undiscounted stationary distributions over state-action pairs for the expert and imitation policies respectively. Here represents the unconstrained output of a discriminator neural network with parameters ω and σ(v) = (1 + e −x) −1 denotes the sigmoid activation function. Since the imitation policy only exerts control over the latter term in the above objective, the per-timestep reward function maximized by reinforcement learning is given as r(s, a, s) = − log(1 − D ω (s, a)). In practice, an entropy regularization term is often added to the objective when optimizing the imitation policy so as to avoid premature convergence to a suboptimal solution (; ;). In order to accommodate various observation-only data sources and remove the burden of requiring expert action labels, the imitation from observation setting adjusts the expert demonstration dataset D such that each trajectory τ i = (s i1, s i2, . . .) consists only of expert observation sequences. Retaining the goal of recovering an imitation policy that closely resembles expert behavior, Torabi et al. (2018b) introduce GAIFO as the natural extension of GAIL for matching the state transition distribution of the expert policy. Note that an objective for matching the stationary distribution over expert state transitions enables the provision of per-timestep feedback while simultaneously avoid the issues of temporal alignment that arise when trying to match trajectories directly. The ing algorithm iteratively finds a solution to the following minimax optimization: where ρ π * (s, s) and ρ π (s, s) now denote the analogous stationary distributions over successive state pairs while D ω (s, s) = σ(V ω (s, s)) represents binary classifier over state pairs. Similar to GAIL, the imitation policy is optimized via policy-gradient reinforcement learning with per-timestep rewards computed according to r(s, a, s) = − log(1 − D ω (s, s)) and using entropy regularization as needed. In this section, we begin with an overview of f -divergences, their connection to GANs, and their impact on IL through the f -VIM framework ) (Section 4.1). We then present an alternative view of the framework that transparently exposes the fundamental choice practictioners must make in order to circumvent practical issues that arise when applying f -VIM to high-dimensional tasks (Section 4.2). We conclude by presenting our approach for ILfO as f -divergence minimization Table of various f -divergences studied in this work as well as the specific choices of activation function g f given by and utilized in. Also shown are the convex conjugates, inverse convex conjugates, and their respective domains. (Section 4.3) followed by a brief discussion of a regularization technique used to stabilize discriminator training in our experiments (Section 4.4). The GAIL and GAIFO approaches engage in an adversarial game where the discriminator estimates the divergence between state-action or state transition distributions according to the JensenShannon divergence . In this work, our focus is on a more general class of divergences, that includes the Jensen-Shannon divergence, known as Ali-Silvey distances or fdivergences (; . For two distributions P and Q with support over a domain X and corresponding continuous densities p and q, we have the f -divergence between them according to: where f : R + → R is a convex, lower-semicontinuous function such that f = 0. As illustrated in Table 1, different choices of function f yield well-known divergences between probability distributions. In order to accommodate the tractable estimation of f -divergences when only provided samples from P and Q, offer an approach for variational estimation of f -divergences. Central to their procedure is the use of the convex conjugate function or Fenchel conjugate (Hiriart-Urruty & Lemaréchal, 2004), f *, which exists for all convex, lower-semicontinuous functions f and is defined as the following supremum: Using the duality of the convex conjugate (f where T is an arbitrary class of functions T : X → dom f * . extend the use of this variational lower bound for GANs that utilize arbitrary f -divergences, or f -GANs. Specifically, the two distributions of interest are the real data distribution P and a synthetic distribution represented by a generative model Q θ with parameters θ. The variational function is also parameterized as T ω acting as the discriminator. This gives rise to the VDM principle which defines the f -GAN objective represent the variational function as X → R represents the unconstrained discriminator network while g f : R → dom f * is an activation function chosen in accordance with the f -divergence being optimized. Table 1 includes the "somewhat arbitrary" but effective choices for g f suggested by and we refer readers to their excellent work for more details and properties of f -divergences and f -GANs. Recently, have formalized the generalization from GAN to f -GAN for the traditional IL problem. They offer the f -Variational Imitation (f -VIM) framework for the specific case of estimating and then minimizing the divergence between state-action distributions induced by expert and imitation policies: min where V ω: S × A → R denotes the discriminator network that will supply per-timestep rewards during the outer policy optimization which itself is carried out over policy parameters θ via policygradient reinforcement learning . In particular, the per-timestep rewards provided to the agent are given according to r(s, a, While do an excellent job of motivating the use of f -divergences for IL (by formalizing the relationship between divergences over trajectory distributions vs. state-action distributions) and connecting f -VIM to existing imitation-learning algorithms, their experiments focus on smaller problems to study the mode-seeking/mode-covering aspects of different f -divergences and the implications of such behavior depending on the multimodality of the expert trajectory distribution. Meanwhile, in the course of attempting to apply f -VIM to large-scale imitation problems, we empirically observe numerical instabilities stemming from function approximation, demanding a reformulation of the framework. In their presentation of the f -VIM framework, retain the choices for activation function g f introduced by for f -GANs. Recall that these choices of g f play a critical role in defining the reward function optimized by the imitation policy on each iteration of f -VIM, r(s, a, s) = f * (g f (V ω (s, a))). It is well known in the reinforcement-learning literature that the nature of the rewards provided to an agent have strong implications on learning success and efficiency . While the activation choices made for f -GANs are suitable given that both optimization problems are carried out by backpropagation, we assert that special care must be taken when specifying these activations (and implicitly, the reward function) for imitation-learning algorithms. A combination of convex conjugate and activation function could induce a reward function that engenders numerical instability or a simply challenging reward landscape, depending on the underlying policy-gradient algorithm utilized . Empirically, we found that the particular activation choices for the KL and reverse KL divergences shown in Table 1 (linear and exponential, respectively) produced imitation-learning algorithms that, in all of our evaluation environments, failed to complete execution due to numerical instabilities caused by exploding policy gradients. In the case of the Total Variation distance, the corresponding f -GAN activation for the variational function is a tanh, requiring a learning agent to traverse a reward interval of [−1, 1] by crossing an intermediate region with reward signals centered around 0. To refactor the f -VIM framework so that it more clearly exposes the choice of reward function to practictioners and shifts the issues of reward scale away from the imitation policy, we propose uniformly applying an activation function g f (v) = f * −1 (r(v)) where f * −1 (t) denotes the inverse of the convex conjugate (see Table 1). Here r is effectively a free parameter that can be set according to one of the many heuristics used throughout the field of deep reinforcement learning for maintaining a reasonable reward scale so long as it obeys the domain of the inverse conjugate dom f * −1. In selecting g f accordingly, the reparameterized saddlepoint optimization for f -VIM becomes min where the per-timestep rewards used during policy optimization are given by r(s, a, s) = r(V ω (s, a) ). In applying this choice, we shift the undesirable scale of the latter term in VDM towards the discriminator, expecting it to be indifferent since training is done by backpropagation. As one potential instantiation, we consider r(u) = σ(u) where σ(·) denotes the sigmoid function leading to bounded rewards in the interval that conveniently adhere to dom f * −1 for almost all of the f -divergences examined in this work 1. In Section 5, we evaluate imitation-learning algorithms with this choice against those using f -VIM with the original f -GAN activations; we find that, without regard for the scale of rewards and the underlying reinforcement-learning problem being solved, the f -GAN activation choices either produce degenerate solutions or completely fail to produce an imitation policy altogether. Applying the variational lower bound of and the corresponding f -GAN extension, we can now present our Variational Imitation from Observation (f -VIMO) extension for a general family of ILfO algorithms that leverage the VDM principle in the underlying saddle-point optimization. Since optimization of the generator will continue to be carried out by policy-gradient reinforcement learning, we adhere to our reparameterization of the f -VIM framework and present the f -VIMO objective as: with the per-timestep rewards given according to r(s, a, s) = r(V ω (s, s)). We present the full approach as Algorithm 1. Just as in Section 4.2, we again call attention to Line 5 where the discriminator outputs (acting as individual reward signals) scale the policy gradient, unlike the more conventional discriminator optimization of Line 4 by backpropagation; this key difference is the primary motivator for our specific reparameterization of the f -VIM framework. Just as in the previous section, we take r(u) = σ(u) as a particularly convenient choice of activation given its agreement to the inverse conjugate domains dom f * −1 for many choices of f -divergence and we employ this instantiation throughout all of our experiments. We leave the examination of alternative choices for r to future work. Update θ i to θ i+1 via a policy-gradient update with rewards given by r(V ω (s, s)): 6: end for The refactored version of f -VIM presented in Section 4.2 is fundamentally addressing instability issues that may occur on the generator side of adversarial training; in our experiments, we also examine the utility of regularizing the discriminator side of the optimization for improved stability. Following from a line of work examining the underlying mathematical properties of GAN optimization 2018; ), we opt for the simple gradient-based regularization of which (for f -VIMO) augments the discriminator loss with the following regularization term: where ψ is a hyperparameter controlling the strength of the regularization. The form of this specific penalty follows from the analysis of; intuitively, its purpose is to disincentivize the 1 For Total Variation distance, we use r(u) = Table 1). discriminator from producing a non-zero gradient that shifts away from the Nash equilibrium of the minimax optimization when presented with a generator that perfectly matches the true data distribution. While originally developed for traditional GANs and shown to empirically exhibit stronger convergence properties over Wasserstein GANs, this effect is still desirable for the adversarial IL setting where the reward function (discriminator) used for optimizing the imitation policy should stop changing once the expert state-transition distribution has been matched. In practice, we compare f -VIM and f -VIMO both with and without the use of this regularization term and find that R(ω) can improve the stability and convergence of f -VIMO across almost all domains. We examine four instantiations of the f -VIM and f -VIMO frameworks (as presented in Sections 4.2 and 4.3) corresponding to imitation algorithms with the following choices of f -divergence: GAN, Kullback-Leibler, reverse KL, and Total Variation. We conduct our evaluation across four MuJoCo environments of varying difficulty: Ant, Hopper, HalfCheetah, and Walker (see the Appendix for more details on individual environments). The core questions we seek to answer through our empirical are as follows: 1. What are the implications of the choice of activation for the variational function in f -VIM on imitation policy performance? 2. Do f -divergences act as a meaningful axis of variation for IL and ILfO algorithms? 3. What is the impact of discriminator regularization on the stability and convergence properties of f -VIM/f -VIMO? 4. How does the impact of different f -divergences vary with the amount of expert demonstration data provided? To answer the first three questions above, we report the average total reward achieved by the imitation policy throughout the course of learning with rewards as defined by the corresponding OpenAI Gym environment . Shading in all plots denote 95% confidence intervals computed over 10 random trials with 10 random seeds. Expert demonstration datasets of 50 trajectories were collected from agents trained via Proximal Policy Optimization (PPO) ; 20 expert demonstrations were randomly subsampled at the start of learning and held fixed for the duration of the algorithm. We also utilize PPO as the underlying reinforcement-learning algorithm for training the imitation policy with a clipping parameter of 0.2, advantage normalization, entropy regularization coefficient 1e −3, and the Adam optimizer . Just as in we use a discount factor of γ = 0.995 and apply Generalized Advantage Estimation with parameter λ = 0.97. We run both f -VIM and f -VIMO for a total of 500 iterations, collecting 50000 environment samples per iteration. The policy and discriminator architectures are identically two separate multi-layer perceptrons each with two hidden layers of 100 units separated by tanh nonlinearities. A grid search was used for determining the initial learning rate, number of PPO epochs, and number of epochs used for discriminator training (please see the Appendix for more details) and we report for the best hyperparameter settings. To address our final question, we take the best hyperparameter settings recovered when given 20 expert demonstrations and re-run all algorithms with {1, 5, 10, 15} expert demonstrations that are randomly sampled at the start of each random trial and held fixed for the duration of the algorithm. We then record the average return of the final imitation policy for each level of expert demonstration. To highlight the importance of carefully selecting the variational function activation g f and validate our modifications to the f -VIM framework, we present in Figure 2 comparing to the original f -VIM framework of and its natural ILfO counterpart. Activation functions for the original methods are chosen according to the choices outlined in;. In our experiments using the KL and reverse KL divergences, we found that none of the trials reached completion due to numerical instabilities caused by exploding policy gradients. Consequently, we only present for the Total Variation distance. We observe that under the original f -GAN activation selection, we fail to produce meaningful imitation policies with learning stagnating after 100 iterations or less. As previously mentioned, we suspect that this stems from the use of tanh with TV leading to a dissipating reward signal. We present in Figure 3 to assess the utility of varying the choice of divergence in f -VIM and f -VIMO across each domain. In considering the impact of f -divergence choice, we find that most of the domains must be examined in isolation to observe a particular subset of f -divergences that stand out. In the IL setting, we find that varying the choice of f -divergence can yield different learning curves but, ultimately, produce near-optimal (if not optimal) imitation policies across all domains. In contrast, we find meaningful choices of f -divergence in the ILfO setting including {KL, TV} for Hopper, RKL for HalfCheetah, and {GAN, TV} for Walker. We note that the use of discriminator regularization per is crucial to achieving these performance gains, whereas the regularization generally fails to help performance in the IL setting. This finding is supportive of the logical intuition that ILfO poses a fundamentally more-challenging problem than standard IL. As a negative , we find that the Ant domain (the most difficult environment with S ⊂ R 111 and A ⊂ R 8) still poses a challenge for ILfO algorithms across the board. More specifically, we observe that discriminator regularization hurts learning in both the IL and ILfO settings. While the choice of RKL does manage to produce a marginal improvement over GAIFO, the gap between existing stateof-the-art and expert performance remains unchanged. It is an open challenge for future work to either identify the techniques needed to achieve optimal imitation policies from observations only or characterize a fundamental performance gap when faced with sufficiently large observation spaces. In Figure 4, we vary the total number of expert demonstrations available during learning and observe that certain choices of f -divergences can be more robust in the face of less expert data, both in the IL and ILfO settings. We find that KL-VIM and TV-VIM are slightly more performant than GAIL when only provided with a single expert demonstration. Notably, in each domain we see that certain choices of divergence for f -VIMO do a better job of residing close to their f -VIM counterparts suggesting that future improvements may come from examining f -divergences in the small-data regime. This idea is further exemplified when accounting for collected while using discriminator regularization . We refer readers to the Appendix for the associated learning curves. Our work leaves many open directions for future work to close the performance gap between student and expert policies in the ILfO setting. While we found the sigmoid function to be a suitable instantiation of our framework, exploring alternative choices of variational function activations could prove useful in synthesizing performant ILfO algorithms. Alternative choices of f -divergences could lead to more substantial improvements than the choices we examine in this paper. Moreover, while this work has a direct focus on f -divergences, Integral Probability Metrics (IPMs) (Müller, 1997;) represent a distinct but well-established family of divergences between probability distributions. The success of Total Variation distance in our experiments, which doubles as both a fdivergence and IPM , is suggestive of future work building IPM-based ILfO algorithms. In this work, we present a general framework for imitation learning and imitation learning from observations under arbitrary choices of f -divergence. We empirically validate a single instantiation of our framework across multiple f -divergences, demonstrating that we overcome the shortcomings of prior work and offer a wide class of IL and ILfO algorithms capable of scaling to larger problems. (; ;), where an agent must leverage demonstration data (typically provided as trajectories, each consisting of expert state-action pairs) to produce an imitation policy that correctly captures the demonstrated behavior. Within the context of LfD, a finer distinction can be made between behavioral cloning (BC) and inverse reinforcement learning (IRL) (Ng et al.; ; ; ; ;) approaches; BC approaches view the demonstration data as a standard dataset of input-output pairs and apply traditional supervisedlearning techniques to recover an imitation policy. Alternatively, IRL-based methods synthesize an estimate of the reward function used to train the expert policy before subsequently applying a reinforcement-learning algorithm to recover the corresponding imitation policy. Although not a focus of this work, we also acknowledge the myriad of approaches that operate at the intersection of IL and reinforcement learning or augment reinforcement learning with IL (; ; ; ; ;). While BC approaches have been successful in some settings (; ;), they are also susceptible to failures stemming from covariate shift where minute errors in the actions of the imitation policy compound and force the agent into regions of the state space not captured in the original demonstration data. While some preventative measures for covariate shift do exist (b), a more principled solution can be found in methods like DAgger and its descendants (; ;) that remedy covariate shift by querying an expert to provide on-policy action labels. It is worth noting, however, that these approaches are only feasible in settings that admit such online interaction with an expert and, even then, failure modes leading to poor imitation policies do exist (a). The algorithms presented in this work fall in with IRL-based approaches to IL. Early successes in this regime tend to rely on hand-engineered feature representations for success (; ;). Only in recent years, with the aid of deep neural networks, has there been a surge in the number of approaches that are capable of scaling to the raw, highdimensional observations found in real-world control problems (; ; ; ; ;). Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive experts (; ; ;); at the heart of these methods is the Generative Adversarial Imitation Learning (GAIL) approach which produces high-fidelity imitation policies and achieves state-of-the-art across numerous continuous-control benchmarks by leveraging the expressive power of Generative Adversarial Networks (GANs) for modeling complex distributions over a high-dimensional support. From an IRL perspective, GAIL can be viewed as iteratively optimizing a parameterized reward function (discriminator) that, when used to optimize an imitation policy (generator) via policy-gradient reinforcement learning , allows the agent to shift its own behavior closer to that of the expert. From the perspective of GANs, this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert. While a large body of prior work exists for IL, numerous recent works have drawn attention to the more challenging problem of imitation learning from observation (; ; ; ; a; b; ; . In an effort to more closely resemble observational learning in humans and leverage the wealth of publicly-available, observation-only data sources, the ILfO problem considers learning from expert demonstration data where no expert action labels are provided. Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via reinforcement learning (; ; choice and the multimodality of the expert trajectory distribution, we provide an empirical evaluation of their f -VIM framework across a range of continous control tasks in the Mujoco domain . Empirically, we find that some of the design choices f -VIM inherits from the original f -GAN work are problematic when coupled with adversarial IL and training of the generator by policy-gradient reinforcement learning, instead of via direct backpropagation as in traditional GANs. Consequently, we refactor their framework to expose this point and provide one practical instantiation that works well empirically. We then go on to extend the f -VIM framework to the IFO problem (f -VIMO) and evaluate the ing algorithms empirically against the state-of-the-art, GAIFO. Here we provide details of the MuJoCo environments used in our experiments as well as the details of the hyperparameter search conducted for all algorithms (IL and ILfO) presented. All environments have continuous observation and action spaces of varying dimensionality (as shown below). All algorithms evaluated in each environment were trained for a total of 500 iterations, collecting 50, 000 environment transitions per iteration. C.3 f -DIVERGENCE VARIATIONAL BOUND SWAP Throughout this paper, we advocate for the use of the following variational lower bound to the f -divergence for both f -VIM and f -VIMO: In particular, we value the above form as it clearly exposes the choice of reward function for the imitation policy as a free parameter that, in practice, has strong implications for the stability and convergence of adversarial IL/ILfO algorithms. Alternatively, one may consider appealing to the original lower bound of , used in f -GANs unmodified, but swapping the positions of the two distributions: Consequently, the term in this lower bound pertaining to the imitation policy is now similar to that of the bound in Equation 11; namely, an almost arbitrary activation function, g f, applied to the output of the variational function (discriminator) V ω. The difference being that the codomain of g f must obey the domain of the convex conjugate, f *, while the codomain of r must respect the domain of the inverse convex conjugate, f * −1. We evaluate these two choices empirically below for the specific choice of the KL-divergence in the Ant and Hopper domains (the two most difficult domains of our evaluation). We find that the original unswapped bound in Equation 11 used throughout this paper outperforms the variants with the distributions swapper, for both the IL and ILfO settings. Crucially, we find that the KL-VIM in the Ant domain no longer achieves expert performance while optimizing the swapped bound.

Represent the TLDR for finding the corresponding scientific article

The overall goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels, through the use of f-divergences.

Represent the short summary for finding the corresponding scientific article (long)

We identify and formalize the memorization problem in meta-learning and solve this problem with novel meta-regularization method, which greatly expand the domain that meta-learning can be  applicable to and effective on.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Momentary fluctuations in attention (perceptual accuracy) correlate with neural activity fluctuations in primate visual areas. Yet, the link between such momentary neural fluctuations and attention state remains to be shown in the human brain. We investigate this link using a real-time cognitive brain machine interface (cBMI) based on steady state visually evoked potentials (SSVEPs): occipital EEG potentials evoked by rhythmically flashing stimuli. Tracking momentary fluctuations in SSVEP power, in real-time, we presented stimuli time-locked to when this power reached (predetermined) high or low thresholds. We observed a significant increase in discrimination accuracy (d') when stimuli were triggered during high (versus low) SSVEP power epochs, at the location cued for attention. Our indicate a direct link between attention’s effects on perceptual accuracy and and neural gain in EEG-SSVEP power, in the human brain. To identify efficient acquisition software, we compared the round-trip (closed-loop) delay for four 34 acquisition software: ActiView, Lab Streaming Layer, OpenVibe and Fieldtrip (Fig. 1B). We measured 35 round-trip delay by varying the EEG+event packet size, across different sampling frequencies, fit a 36 line to the data, and estimated the intercept, which is a measure of the overhead. We observed that Fieldtrip produced the least overhead of 10.98 ± 0.50 ms. EEG data recording: Scalp EEG recordings were performed with 41 occipital electrodes out of 39 the total 128 electrodes. The data was streamed in real-time using the Fieldtrip buffer at 128 Hz. EEG 40 data was also stored at 4096 Hz for offline analyses. Spectral analysis was performed using Chronux 41 2.12 toolbox EEGLAB 13.6.5b functions were used to generate the topographical plots. Dense-array Studies). Finally, the EEG data was re-referenced to the average reference. higher, across the population, for AI-high trials as compared to AI-low trials (p < 0.01). We did not

represent the short summary for finding the corresponding scientific article (long).

With a cognitive brain-machine interface, we show a direct link between attentional effects on perceptual accuracy and neural gain in EEG-SSVEP power, in the human brain.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We extended single-trial space-by-time tensor decomposition based on non-negative matrix factorization to efficiently discount pre-stimulus baseline activity that improves decoding performance on data with non-negligible baselines.


Question: Represent the article for finding a single-sentence TLDR

Question: Recent studies have demonstrated the vulnerability of deep convolutional neural networks against adversarial examples. Inspired by the observation that the intrinsic dimension of image data is much smaller than its pixel space dimension and the vulnerability of neural networks grows with the input dimension, we propose to embed high-dimensional input images into a low-dimensional space to perform classification. However, arbitrarily projecting the input images to a low-dimensional space without regularization will not improve the robustness of deep neural networks. We propose a new framework, Embedding Regularized Classifier (ER-Classifier), which improves the adversarial robustness of the classifier through embedding regularization. Experimental on several benchmark datasets show that, our proposed framework achieves state-of-the-art performance against strong adversarial attack methods. Deep neural networks (DNNs) have been widely used for tackling numerous machine learning problems that were once believed to be challenging. With their remarkable ability of fitting training data, DNNs have achieved revolutionary successes in many fields such as computer vision, natural language progressing, and robotics. However, they were shown to be vulnerable to adversarial examples that are generated by adding carefully crafted perturbations to original images. The adversarial perturbations can arbitrarily change the network's prediction but often too small to affect human recognition . This phenomenon brings out security concerns for practical applications of deep learning. Two main types of attack settings have been considered in recent research (Goodfellow et al.; a; ;): black-box and white-box settings. In the black-box setting, the attacker can provide any inputs and receive the corresponding predictions. However, the attacker cannot get access to the gradients or model parameters under this setting; whereas in the white-box setting, the attacker is allowed to analytically compute the model's gradients, and have full access to the model architecture and weights. In this paper, we focus on defending against the white-box attack which is the harder task. Recent work presented both theoretical arguments and an empirical one-to-one relationship between input dimension and adversarial vulnerability, showing that the vulnerability of neural networks grows with the input dimension. Therefore, reducing the data dimension may help improve the robustness of neural networks. Furthermore, a consensus in the highdimensional data analysis community is that, a method working well on the high-dimensional data is because the data is not really of high-dimension . These high-dimensional data, such as images, are actually embedded in a low dimensional space. Hence, carefully reducing the input dimension may improve the robustness of the model without sacrificing performance. Inspired by the observation that the intrinsic dimension of image data is actually much smaller than its pixel space dimension and the vulnerability of a model grows with its input dimension , we propose a defense framework that embeds input images into a low-dimensional space using a deep encoder and performs classification based on the latent embedding with a classifier network. However, an arbitrary projection does not guarantee improving the robustness of the model, because there are a lot of mapping functions including non-robust ones from the raw input space to the low-dimensional space capable of minimizing the classification loss. To constrain the mapping function, we employ distribution regularization in the embedding space leveraging optimal transport theory. We call our new classification framework Embedding Regularized Classifier (ER-Classifier). To be more specific, we introduce a discriminator in the latent space which tries to separate the generated code vectors from the encoder network and the ideal code vectors sampled from a prior distribution, i.e., a standard Gaussian distribution. Employing a similar powerful competitive mechanism as demonstrated by Generative Adversarial Networks , the discriminator enforces the embedding space of the model to follow the prior distribution. In our ER-Classifier framework, the encoder and discriminator structures together project the input data to a low-dimensional space with a nice shape, then the classifier performs prediction based on the lowdimensional embedding. Based on the optimal transport theory, the proposed ER-Classifier minimizes the discrepancy between the distribution of the true label and the distribution of the framework output, thus only retaining important features for classification in the embedding space. With a small embedding dimension, the effect of the adversarial perturbation is largely diminished through the projection process. We compare ER-Classifier with other state-of-the-art defense methods on MNIST, CIFAR10, STL10 and Tiny Imagenet. Experimental demonstrate that our proposed ER-Classifier outperforms other methods by a large margin. To sum up, this paper makes the following three main contributions: • A novel unified end-to-end robust deep neural network framework against adversarial attacks is proposed, where the input image is first projected to a low-dimensional space and then classified. • An objective is induced to minimize the optimal transport cost between the true class distribution and the framework output distribution, guiding the encoder and discriminator to project the input image to a low-dimensional space without losing important features for classification. • Extensive experiments demonstrate the robustness of our proposed ER-Classifier framework under the white-box attacks, and show that ER-Classifier outperforms other state-ofthe-art approaches on several benchmark image datasets. As far as we know, our approach is the first that applies optimal transport theory, i.e., a Wasserstein distance regularization, to a bottleneck embedding layer of a deep neural network in a purely supervised learning setting without considering any reconstruction loss, although optimal transport theory or a discriminator loss has been applied to generative models in an unsupervised learning setting ; Our method is also the first that establishes the connection between a Wasserstein distance regularization and the robustness of deep neural networks for defending against adversarial examples. In this section, we summarize related work into three categories: attack methods, defense mechanisms and optimal transport theory. We first discuss different white-box attack methods, followed by a description of different defense mechanisms against, and finally optimal transport theory. Under the white-box setting, attackers have all information about the targeted neural network, including network structure and gradients. Most white-box attacks generate adversarial examples based on the gradient of loss function with respect to the input. An algorithm called fast gradient sign method (FGSM) was proposed in (Goodfellow et al.) which generates adversarial examples based on the sign of gradient. Many other white-box attack methods have been proposed recently (; ; ; b), and among them C&W and PGD attacks have been widely used to test the robustness of machine learning models. C&W attack: The adversarial attack method proposed by Carlini and Wagner (b) is one of the strongest white-box attack methods. They formulate the adversarial example generating process as an optimization problem. The proposed objective function aims at increasing the probability of the target class and minimizing the distance between the adversarial example and the original input image. Therefore, C&W attack can be viewed as a gradient-descent based adversarial attack. PGD attack: The projected gradient descent attack is proposed by , which finds adversarial examples in an -ball of the image. The PGD attack updates in the direction that decreases the probability of the original class most, then projects the back to the -ball of the input. An advantage of PGD attack over C&W attack is that it allows direct control of distortion level by changing, while for C&W attack, one can only do so indirectly via hyper-parameter tuning. Both C&W attack and PGD attack have been frequently used to benchmark the defense algorithms due to their effectiveness . In this paper, we mainly use l ∞ -PGD untargeted attack to evaluate the effectiveness of the defense method under white-box setting. Instead of crafting different adversarial perturbation for different input image, an algorithm was proposed by to construct a universal perturbation that causes natural images to be misclassified. However, since this universal perturbation is image-agnostic, it is usually larger than the image-specific perturbation generated by PGD and C&W. Many works have been done to improve the robustness of deep neural networks. To defend against adversarial examples, defenses that aim to increase model robustness fall into three main categories: i) augmenting the training data with adversarial examples to enhance the existing classifiers (; ; Goodfellow et al.); ii) leveraging model-specific strategies to enforce model properties such as smoothness ; and, iii) trying to remove adversarial perturbations from the inputs (; ;). We select three representative methods that are effective under white-box setting. Adversarial training: Augmenting the training data with adversarial examples can increase the robustness of the deep neural network. Madry et al. recently introduced a minmax formulation against adversarial attacks. The proposed model is not only trained on the original dataset but also adversarial example in the -ball of each input image. Random Self-Ensemble: Another effective defense method under white-box setting is RSE . The authors proposed a "noise layer", which fuses output of each layer with Gaussian noise. They empirically show that the noise layer can help improve the robustness of deep neural networks. The noise layer is applied in both training and testing phases, so the prediction accuracy will not be largely affected. Defense-GAN: Defense-GAN leverages the expressive capability of GANs to defend deep neural networks against adversarial examples. It is trained to project input images onto the range of the GAN's generator to remove the effect of the adversarial perturbation. Another defense method that uses the generative model to filter out noise is MagNet proposed by . However, the differences between ER-Classifier and the two methods are obvious. ER-Classifier focuses on reducing the dimension, and performing classification based on the low-dimensional embedding, while Defense-GAN and MagNet mainly apply the generative model to filter out the adversarial noise, and both Defense-GAN and MagNet perform classification on the original dimension space. showed that Defense-GAN is more robust than MagNet, so we only compare with Defense-GAN in the experiments. Other Related Methods: regularizes the latent space with Gaussian Mixture Model and applies KL-divergence to do the optimization. However, our method employs a simple but nice-shaped Gaussian prior for Wasserstein distance minimization to constrain the global shape of the latent embeddings, while permitting high freedom for the shapes of individual class distributions of latent embeddings. We want the classifier to decide the optimal class-specific distributions of latent embeddings. shares a similar idea to adversarial learning, but it employs virtual labels generated by a current classifier to identify search directions that can smooth the output label distribution of the classifier and is best suitable for semi-supervised learning. Please note that both methods in are designed for improving generalization performance but not for defending against adversarial examples. A recent paper shows that adversarial examples are purely human phenomenon and models tend to learn features that are not robust yet generalize well. We show that our Wasserstein distance regularization helps to identify robust features, which will be discussed later. Notations In this paper, we use l ∞ and l 2 distortion metrics to measure similarity. We report l ∞ distance in the normalized space, so that a distortion of 0.031 corresponds to 8/256, and l 2 distance as the total root-mean-square distortion normalized by the total number of pixels. We use calligraphic letters for sets (i.e., X), capital letters for random variables (i.e., X), and lower case letters for their values (i.e., x). The probability distributions are denoted with capital letters (i.e., P X) and corresponding densities with lower case letters (i.e., p X). We propose a novel defense framework, ER-Classifier, which aims at projecting the image data to a low-dimensional space to remove noise and stabilize the classification model by minimizing the optimal transport cost between the true label distribution P Y and the distribution of the ER-Classifier output (P C). An overview of the framework is shown in Figure 1. The encoder and discriminator structures together help diminish the effect of the adversarial perturbation by projecting input data to a space of lower dimension, then the classifier part performs classification based on the lowdimensional embedding. Mathematically, input images X ∈ X = R d are projected to a low-dimensional embedding vector Z ∈ Z = R k through the encoder Q φ. The discriminator D γ discriminates between the generated codeZ ∼ Q φ (Z|X) and the ideal code Z ∼ P Z. The classifier C τ performs classification based on the generated codeZ, producing output U ∈ U = R m, where m is the number of classes. The label of X is denoted as Y ∈ U. The Kantorovich's distance induced by the optimal transport problem is given by is the set of all joint distributions of (Y, U) with marginals P Y and P C, and c(y, u): U × U → R + is any measurable cost function. W c (P Y, P C) measures the divergence between probability distributions P Y and P C. When the probability measures are on a metric space, the p-th root of W c is called the p-Wasserstein distance. To minimize the Wasserstein distance between the distribution of the true label (P Y) and the distribution of the ER-Classifier output (P C), we can prove that it is sufficient to find a conditional distribution Q(Z|X) such that its marginal distribution Q Z is identical to a prior distribution P Z. The theorem and the proof are deferred to the Appendix. In this paper, we apply standard Gaussian as our prior distribution P Z, but other priors may be used for different cases. The final objective of ER-Classifier is: where Q can be a deterministic encoder as focused by this paper due to its simplicity or stochastic encoder as the one in a standard Variational Autoencoder, λ > 0 is a hyper-parameter and D is an arbitrary divergence between Q Z and P Z. To estimate the divergences between Q Z and P Z, we apply a GAN-based framework, fitting a discriminator to minimize the 1-Wasserstein distance between Q Z and P Z: We have also tried the Jensen-Shannon divergence, but as expected, Wasserstein distance provides more stable training and better . When training the framework, the weight clipping method proposed in Wasserstein GAN is applied to help stabilize the training of discriminator D γ. The training algorithm is summarized in Algorithm 1. At training stage, the encoder Q φ first maps the input x to a low-dimensional space, ing in generated code (z). Another ideal code (z) is sampled from the prior distribution, and the discriminator D γ discriminates between the ideal code (positive data) and the generated code (negative data). The classifier (C τ) predicts the image label based on the generated code (z). Sample {(x 1, y 1),..., (x n, y n)} from the training set 5: Sample {z 1, ..., z n} from the prior P Z 6: Update D γ by ascending the following objective by 1-step Adam: Update Q φ and C τ by descending the following objective by 1-step Adam: Update Q φ by ascending the following objective by 1-step Adam: 10: end while At inference time, only the encoder Q φ and the classifier C τ are used. The input image x is first mapped to a low-dimensional space by the encoder (z = Q φ (x)), then the latent codez is fed into the classifier to obtain the predicted label. The main goal of ER-Classifier is leveraging input space dimension reduction to remove adversarial perturbations. Therefore, other defense methods can also benefit from this property. Our framework is trained with min-max robust optimization . There are two Wasserstein distances (W-distances) in our framework. One is the W-distance between the aggregated latent embedding distribution Q(z) and the prior distribution P Z, and the other one is the W-distance between the true label distribution P Y and the distribution of the ER-Classifier output (P C). In Algorithm 1, we are minimizing the first one. The theorem in the Appendix shows that minimizing the first W-distance in combination with minimizing a standard cross-entropy loss as done in Algorithm 1 is equivalent to minimizing the second W-distance, which guarantees that the training process is not distracted from the main goal of the framework, classification. That is to say, Algorithm 1 will in a classifier with the following property: the global output distribution of the classifier will match the global ground-truth label distribution in the data no matter whether the encoder Q φ is deterministic or stochastic (the second W-distance is automatically minimized). It's hard to analyze the importance of the theorem in the Appendix if we just look at a deterministic encoder. Let's convert this deterministic encoder to a stochastic encoder that outputs a Gaussian z with a fixed variance and the mean being the same as its corresponding deterministic version. The theory tells us that, by minimizing the first W-distance over all sampled z's from this stochastic encoder and the standard cross-entropy loss, we will automatically minimize the second W-distance and preserve the global label frequency in the dataset, even though these z's are only -close to the deterministic encoding features of training data. Moreover, we find that minimizing the W-distance helps the encoder identify some robust features instead of non-robust features , because our proposed regularization constrains the -ball around each Q φ (Z|X) to contribute to preserving the global label distribution in the data, even with X integrated out. From this perspective, we can view our proposed framework as "a supervised variant" of Generative Adversarial Network or Wasserstein Autoencoder in which the Generator or Decoder is replaced by a Classifier that generates labels from low-dimensional latent embeddings preserving global label frequency in the training dataset. Replacing W-distance with KL divergence loses all these nice properties. In our framework, we use a simple but nice-shaped Gaussian prior P Z for W-distance minimization to constrain the global shape of the latent embeddings, while permitting high freedom for the shapes of individual class distributions of latent embeddings. We want the classifier to decide the optimal class-specific distributions of latent embeddings. In addition, it is interesting to explore how to set -ball to make sure the stochastic encoder to best align the latent embedding z to human-perceived robust features, which will be left as future work. In this section, we compare the performance of our proposed algorithm (ER-Classifier) with other state-of-the-art defense methods on several benchmark datasets: • MNIST : handwritten digit dataset, which consists of 60, 000 training images and 10, 000 testing images. Theses are 28 × 28 black and white images in ten different classes. classes, and each class has 500 training images, 50 testing images, making it a challenging benchmark for defense task. The resolution of the images is 64 × 64. Various defense methods have been proposed to improve the robustness of deep neural networks. Here we compare our algorithm with state-of-the-art methods that are robust in white-box setting. Madry's adversarial training (Madry's Adv) has been recognized as one of the most successful defense method in white-box setting, as shown in . Random Self-Ensemble (RSE) method introduced by adds stochastic components in the neural network, achieving similar performance to Madry's adversarial training algorithm. Another method we would like to compare with is Defense-GAN . It first trains a generative adversarial network to model the distribution of the training data. At inference time, it finds a close output to the input image and feed that output into the classifier. This process "projects" input images onto the range of GAN's generator, which helps remove the effect of adversarial perturbations. In , the author demonstrated the performance of Defense-GAN on MNIST and Fashion-MNIST, so we will compare our method with Defense-GAN on MNIST. Since the main goal of ER-Classifier is using dimension reduction to improve adversarial robustness, other defense methods can also benefit from this property. The proposed ER-Classifier is trained with min-max robust optimization . To demonstrate the dimension reduction ability of ER-Classifier, we include a variant ER-Classifier − which trains ER-Classifier without min-max robust optimization. In this section, we evaluate the defense methods against l ∞ -PGD untargeted attack, which is one of the strongest white-box attack methods. Models are evaluated under different distortion level , Based on Figure 2 and − tends to perform better than other state-of-the-art defense methods on MNIST, CIFAR10 and Tiny Imagenet. This phenomenon is obvious on CIFAR10 and it even performs better than ER-Classifier when the attack strength is strong. The reason might be that without min-max robust optimization, it is easier to regularize the embedding space. Testing Accuracy Defense-GAN 55.0 ER-Classifier 99.1 We also compare Defense-GAN with our method ERClassifier on MNIST. Although Defense-GAN was shown to be partly broken by , both ER-Classifier and Defense-GAN share the similar idea of projecting the input to a learned manifold, and comparing to Defense-GAN is important to demonstrate the advantage of our novel Wassserstein distance regularization. Please note that Defense-GAN is not our major comparison baseline in this paper. Both ER-Classifier and Defense-GAN are evaluated against the l 2 -C&W untargeted attack, one of the strongest white-box attack proposed in (b). Defense-GAN is evaluated using the method proposed in , and the code is available on github 1. ER-Classifier is evaluated against l 2 -C&W untargeted attack with the same hyper-parameter values as those used in the evaluation of Defense-GAN. The under l 2 ≤ 0.005 threshold are shown in Table 2. Based on Table 2, ER-Classifier is much more robust than Defense-GAN un- der the l 2 ≤ 0.005 threshold. Since did not evaluate Defense-GAN on CIFAR10, STL10 and Tiny Imagenet, without details of GAN structure, we can not compare with Defense-GAN on these datasets. We evaluate Madry's adversarial training, ER-Classifier, and ER-Classifier −, against a recently proposed black-box attack method called Nattack 2 on CIFAR10. Nattack is only performed on the first 100 images of CIFAR10 since the attack process takes a long time. We report the accuracy = number of correctly classified / number of attacked images (exactly 100). The accuracy of Madry's adv, ER-Classifier, and ER-Classifier − is, respectively, 38%, 43%, and 32%. ER-Classifier still outperforms Madry's adv. ER-Classifier framework consists of three parts, and the classification task is done by the encoder Q φ and classifier C τ. Without the discriminator part, the encoder can also project the input images to a low-dimensional space. However, arbitrarily projecting the images to a low-dimensional space with only the encoder part cannot improve the robustness of the model. In contrast, sometimes it even decreases the robustness of the model. To show that arbitrarily projecting the input images to a low-dimensional space can not improve the robustness, we fit a framework with only the encoder and classifier part (E-CLA), where the encoder and classifier have the same structures as in ER-Classifier, and compare E-CLA with the ER-Classifier framework. For a fair comparison, both structures are trained without min-max robust optimization. The are shown in Figure 3. Based on Figure 3, we can observe that ER-Classifier is much more robust than just the encoder and classifier structure on MNIST, CIFAR10 and Tiny Imagenet. It is also more robust on STL10 but not that much. The reason might be that there are only 5, 000 training images in STL10 and the resolution is 96 × 96. Therefore, it is harder to learn a good embedding with limited amount of images. However, even when the number of training images is limited, ER-Classifier is still much more robust than the E-CLA structure. This observation demonstrates that regularization on the embedding space helps improve the adversarial robustness. Notice that the performance of E-CLA structure is similar to the performance of model without defense method on CIFAR10, STL10 and Tiny Imagenet, and worse on MNIST, which means the robustness of ER-Classifier does not come from the structure design. Variational auto-encoder can project the images to low-dimensional space and use Kullback-Leibler divergence loss to regularize the embedding distribution, which does not need discriminator structure. Therefore, we also tried VAE-CLA, which applies Variational auto-encoder structure to do the projection and regularization. The experimental in Figure 3 show that VAE-CLA does not perform as well as ER-Classifier. Based on the observation of the Kullback-Leibler loss and classification loss during the training process, it seems difficult for VAE-CLA to balance between the two tasks. The reason might be that Kullback-Leibler distances are not sensible cost functions when learning distributions supported by low dimensional manifolds . However, the selection of prior is important as it imposes different restrictions on the embedding space. Three different prior distributions are tested on MNIST and CIFAR10 datasets. They are standard Gaussian, Uniform(−3, 3) and Cauchy, where Cauchy has the same support as standard Gaussian but is heavy tailed and 99.7% of the standard Gaussian points lies within [−3, 3]. All the models are trained without min-max robust optimization, and the experimental are shown in Figure 4. Based on the , all three priors work well, but standard Gaussian performs best on both datasets. Ding et al. prove that adversarial robustness is sensitive to the input data distribution, and if the data is uniformly distributed in the input space, no algorithm can achieve good robustness. They also empirically show that cornered/concentrated data distributions tend to achieve better robustness. This helps explain why regularizing the embedding space can help improve robustness. Though the projection process reduces the input dimension, the embedding space is still large. Prior distribution helps push the embedding space to be more concentrated, reducing the valid perturbation space. Details of hyper-parameter selection, model structure and code are included in the supplementary part. Embedding space visualization can also be found in the supplementary material. In this paper, we propose a new defense framework, ER-Classifier, which projects the input images to a low-dimensional space to remove adversarial perturbation and stabilize the model through minimizing the discrepancy between the true label distribution and the framework output distribution. We empirically show that ER-Classifier is much more robust than other state-of-the-art defense methods on several benchmark datasets. Future work will include further exploration of the low-dimensional space to improve the robustness of deep neural network. Mathematically, input images X ∈ X = R d are projected to a low-dimensional embedding vector Z ∈ Z = R k through the encoder Q φ. The discriminator D γ discriminates between the generated codeZ ∼ Q φ (Z|X) and the ideal code Z ∼ P Z. The classifier C τ performs classification based on the generated codeZ, producing output U ∈ U = R m, where m is the number of classes. The label of X is denoted as Y ∈ U. The ER-Classifier framework embeds important classification features by minimizing the discrepancy between the distribution of the true label (P Y) and the distribution of the framework output (P C). In the framework, the classifier (P C (U |Z)) maps a latent code Z sampled from a fixed distribution in a latent space Z, to the output U ∈ U = R m. The density of ER-Classifier output is defined as follow: In this paper we apply standard Gaussian as our prior distribution P Z, but other priors may be used for different cases. Assume there is an oracle f: X → U assigning the image data (X ∈ X) its true label (Y ∈ U). We want to minimize the optimal transport cost between the distribution of the true label (P Y) and the distribution of the ER-Classifier output (P C). There are various ways to define the distance or divergence between the target distribution and the model distribution. In this paper, we turn to the optimal transport theory , which provides a much weaker topology than many others. In real applications, data is usually embedded in a space of a much lower dimension, such as a non-linear manifold. Kullback-Leibler divergence, Jensen-Shannon divergence and Total Variation distance are not sensible cost functions when learning distributions supported by lower dimensional manifolds . In contrast, the optimal transport cost is more sensible in this setting. Kantorovich's distance induced by the optimal transport problem is given by where P(Y ∼ P Y, U ∼ P C) is the set of all joint distributions of (Y, U) with marginals P Y and P C, and c(y, u): U × U → R + is any measurable cost function. W c (P Y, P C) measures the divergence between probability distributions P Y and P C. When the probability measures are on a metric space, the p-th root of W c is called the p-Wasserstein distance. To minimize the optimal transport cost between the distribution of the true label (P Y) and the distribution of the ER-Classifier output (P C), it is sufficient to find a conditional distribution Q(Z|X) such that its marginal distribution Q Z is identical to the prior distribution P Z. Theorem 1 For P C as defined above with a deterministic P C (U |Z) and any function C: where Γ ∈ P(Y ∼ P Y, U ∼ P C) is the set of all joint distributions of (Y, U) with marginals P Y and P C, and (y, u): U × U → R + is any measurable cost function. Q Z is the marginal distribution of Z when X ∼ P X and Z ∼ Q(Z|X). (The proof is presented later.) Therefore, optimizing over the objective on the r.h.s is equivalent to minimizing the discrepancy between the true label distribution (P Y) and the output distribution P C, thus the important classification features are embedded in the low-dimensional space. This is the core idea of the paper, summarizing the high-dimensional data in a space of much lower dimension without losing important features for classification. To implement the r.h.s objective, the constraint on Q Z can be relaxed by adding a penalty term. The final objective of ER-Classifier is: where Q is any nonparametric set of probabilistic encoders, λ > 0 is a hyper-parameter and D is an arbitrary divergence between Q Z and P Z. To estimate the divergences between Q Z and P Z, we apply a GAN-based framework, fitting a discriminator to minimize the 1-Wasserstein distance between Q Z and P Z: We have also tried the Jensen-Shannon divergence, but as expected, Wasserstein distance provides more stable training and better . When training the framework, the weight clipping method proposed in Wasserstein GAN is applied to help stabilize the training of discriminator D γ. The proof of Theorem 1 is adapted from the proof of Theorem 1 in . Consider certain sets of joint probability distributions of three random variables (X, U, Z) ∈ X × U × Z. X can be taken as the input images, U as the output of the framework, and Z as the latent codes. P C,Z (U, Z) represents a joint distribution of a variable pair (U, Z), where Z is first sampled from P Z and then U from P C (U |Z). P C defined in is the marginal distribution of U when (U, Z) ∼ P C,Z. The joint distributions Γ(X, U) or couplings between values of X and U can be written as Γ(X, U) = Γ(U |X)P X (X) due to the marginal constraint. Γ(U |X) can be decomposed into an encoding distribution Q(Z|X) and the generating distribution P C (U |Z), and Theorem 1 mainly shows how to factor it through Z. In the first part, we will show that if P C (U |Z) are Dirac measures, we have where P(X ∼ P X, U ∼ P C) denotes the set of all joint distributions of (X, U) with marginals P X, P C, and likewise for P(X ∼ P X, Z ∼ P Z). The set of all joint distributions of (X, U, Z) such that X ∼ P X, (U, Z) ∼ P C,Z, and (U ⊥ ⊥ X)|Z are denoted by P X,U,Z. P X,U and P X,Z denote the sets of marginals on (X, U) and (X, Z) induced by P X,U,Z. From the definition, it is clear that P X,U ⊆ P(P X, P C). Therefore, we have The identity is satisfied if P C (U |Z) are Dirac measures, such as U = C(Z). This is proved by the following Lemma in . Lemma 1 P X,U ⊆ P(P X, P C) with identity if P C (U |Z = z) are Dirac for all z ∈ Z. (see details in .) In the following part, we show that Based on the definition, P(P X, P C), P X,U,Z and P X,U depend on the choice of conditional distributions P C (U |Z), but P X,Z does not. It is also easy to check that P X,Z = P(X ∼ P X, Z ∼ P Z). The tower rule of expectation, and the conditional independence property of P X,U,Z implies Finally, since Y = f (X), it is easy to get Now, and are proved and the three together prove Theorem 1. Our proposed framework readily applies to non-deterministic case. If the classifier part is nondeterministic, Lemma 1 provides only the inclusion of sets P X,U ⊆ P(P X, P U), and we can get an upper bound on the Wasserstein distance between the ground-truth and predicted label distributions: where we assume the conditional distributions P C (U |Z = z) have mean values C(z) ∈ R d and marginal variances σ 2 1,..., σ 2 d ≥ 0 for all z ∈ Z, where C: Z → X, and (y, u) = y − u 2. The above upper bound is derived by: and In equation 11, the second term of the second last row becomes 0 since the optimization will drive f (X) − C(Z) to zero. One important hyper-parameter for the ER-Classifier is the dimension of the embedding space. If the dimension is too small, important features are "collapsed" onto the same dimension, and if the dimension is too large, the projection will not extract useful information, which in too much noise and instability. The maximum likelihood estimation of intrinsic dimension proposed in 3 is used to calculate the intrinsic dimension of each image dataset, serving as a guide for selecting the embedding dimension. The sample size used in calculating the intrinsic dimension is 1, 000, and changing the sample size does not influence the much. Based on the intrinsic dimension calculated by , we test several different values around the suggested intrinsic dimension and evaluate the models against l ∞ -PGD attack. All models are trained without min-max robust optimization, and the experimental are shown in Figure 5. The final embedding dimension is chosen based on robustness, number of parameters, and testing accuracy when there is no attack. The final embedding dimensions and suggested intrinsic dimensions are shown in Table 3 Table 3: Pixel space dimension, intrinsic dimension calculated by , and final embedding dimension used. Based on Figure 5, the embedding dimension close to the calculated intrinsic dimension usually offers better except on MNIST. One explanation may be that MNIST is a simple handwritten digit dataset, so performing classification on MNIST may not require that many dimensions. Epsilon is an important hyper-parameter for adversarial training. When doing Madry's adversarial training, we test the model robustness with different and choose the best one. The experiment are shown in Figure 6. Based on Figure 6, we use = 0.3, 0.03, 0.03 in Madry's adversarial training on MNIST, CIFAR10 and STL10 respectively. For Tiny Imagenet, we use = 0.01. To make a fair comparison, we use the same when training ER-Classifier. In this section, we compare the embedding learned by Encoder+Classifier structure (E-CLA) and the embedding learned by ER-Classifier on several datasets without min-max robust optimization. We first generate embedding of testing data using the encoder (z = Q φ (x)), then project the embedding points (z) to 2-D space by tSNE . Then we generate adversarial images (x adv) against E-CLA and ER-Classifier using l ∞ -PGD attack. The adversarial embedding is generated by feeding the adversarial images into the encoder (z adv = Q φ (x adv)). Finally, we project the adversarial embedding points (z adv) to 2-D space. The are shown in Figure 7. The plots in the first and second rows are embedding visualization plots for E-CLA, and the plots in the third and last rows are the embedding visualization plots for ER-Classifier. In adversarial embedding visualization plots, the misclassified point is marked as "down triangle", which means the PGD attack successfully changed the prediction, and the correctly classified point is marked as "point", which means the attack fails. Based on Figure 7, we can see that E-CLA can learn a good embedding on legitimate images of MNIST. Embedding points for different classes are separated on the 2D space, but under adversarial attack, some embedding points of different classes are mixed together. However, ER-Classifier can generate good separated embeddings on both legitimate and adversarial images. On CIFAR10, the E-CLA can not generate good separated embeddings on either legitimate images or adversarial images, while ER-Classifier can generate good separated embeddings for both. Code for reproduction will be made available online at github later. The pseudocode for training ER-Classifier is shown in Listing 1. MNIST, STL10 and TinyImagenet classifier structures used for baseline methods are shown in Fig

A general and easy-to-use framework that improves the adversarial robustness of deep classification models through embedding regularization.

Represent the TLDR for finding the corresponding scientific article.

We present a new method that combines transfer-based and scored black-box adversarial attack, improving the success rate and query efficiency of black-box adversarial attack across different network architectures.