Question: Represent the scientific article to find a short TLDR

Question: We focus on the problem of learning a single motor module that can flexibly express a range of behaviors for the control of high-dimensional physically simulated humanoids. To do this, we propose a motor architecture that has the general structure of an inverse model with a latent-variable bottleneck. We show that it is possible to train this model entirely offline to compress thousands of expert policies and learn a motor primitive embedding space. The trained neural probabilistic motor primitive system can perform one-shot imitation of whole-body humanoid behaviors, robustly mimicking unseen trajectories. Additionally, we demonstrate that it is also straightforward to train controllers to reuse the learned motor primitive space to solve tasks, and the ing movements are relatively naturalistic. To support the training of our model, we compare two approaches for offline policy cloning, including an experience efficient method which we call linear feedback policy cloning. We encourage readers to view a supplementary video (https://youtu.be/CaDEf-QcKwA) summarizing our . A broad challenge in machine learning for control and robotics is to produce policies capable of general, flexible, and adaptive behavior of complex, physical bodies. To build policies that can effectively control simulated humanoid bodies, researchers must simultaneously overcome foundational challenges related to high-dimensional control, body balance, and locomotion. Recent progress in deep reinforcement learning has raised hopes that such behaviors can be learned end-to-end with minimal manual intervention. Yet, even though significant progress has been made thanks to better algorithms, training regimes, and computational infrastructure, the ing behaviors still tend to exhibit significant idiosyncrasies (e.g. BID2 .One advantage of working with humanoids in this context is that motion capture data is widely available and can serve to help design controllers that produce apparently humanlike movement. Indeed, recent developments are now allowing for the production of highly specialized expert policies which robustly, albeit narrowly, reproduce single motion capture clips (e.g. BID18 ; BID30).A remaining challenge on the way to truly flexible and general purpose control is to be able to sequence and generalize individual movements or "skills" in a task-directed manner. Achieving this goal requires not just the ability to acquire individual skills in the first place, but also an architecture and associated training procedure that supports representation, recruitment, and composition of a large number of skills. This paper presents a step in this direction. Specifically, the setting we focus on will be one in which we have a large number of robust experts that perform single skills well and we wish to transfer these skills into a shared policy that can do what each expert does as well as the expert, while also generalizing to unseen behaviors within the distribution of skills. To this end we design a system that performs one-shot imitation as well as permits straightforward reuse (or transfer) of skills. We require our approach to scale to a very large number of individual skills while also keeping manual intervention and oversight to a minimum. Our primary contribution is the development of a neural network architecture that can represent and generate many motor behaviors, which we refer to as neural probabilistic motor primitives. This architecture is designed to perform one-shot imitation, while learning a dense embedding space of a large number of individual motor skills. Once trained, this module does not just reproduce individual behaviors in the training data, but can sequence and compose these behaviors in a controlled fashion as well as synthesize novel movements consistent with the training data distribution. Empirically, we also find that training controllers to reuse this learned motor primitive module for new tasks generates surprisingly human-like movement and the behavior generated seems to interpolate the space of behaviors well. In order to facilitate transfer and compression of expert skills at the scale of thousands of behaviors, we wish to avoid closed-loop RL training. We call the general, offline, functional transfer of policy content policy transfer or policy cloning and consider two approaches. The natural baseline approach involves the application of behavioral cloning to data gathered by executing experts many times, with noise, and logging intended expert actions, resembling the approach of BID16. This works well, as it ensures the student behaves like the expert not only along nominal expert rollouts but also at points arrived at by perturbing the expert. However, this approach may require many rollouts, which can be costly to obtain in many settings. As a more efficient alternative we therefore consider a second solution that operates by comprehensively transferring the functional properties of an expert to a student policy by matching the local noise-feedback properties along one or a small number of representative expert reference trajectories. We call this specific proposal linear feedback policy cloning (LFPC), and we demonstrate that it is competitive with behavioral cloning from many more rollouts in our setting. Recent efforts in RL for humanoid control build on a large body of research in robotics and animation. While contemporary for learning from scratch BID35 can be impressive the behaviors are not consistently human-like. Learning from motion capture (mocap) can provide strong constraints, especially for running BID29. Several recent approaches have demonstrated that it is possible to acquire specific behavioral skills, possibly jointly with external RL objectives BID30 BID17. At present, the policies produced tend to be restricted to single skills/behaviors and can require very large quantities of environment interactions, motivating us to seek methods which reuse existing single-skill expert policies. Knowledge transfer refers to the broad class of approaches which transfer the input-output functional mapping, to some extent or another, from a teacher (or expert) to a student BID12 BID36 BID8. Distillation connotes the transfer of function from one or more expert systems into a single student system often with the goal of compression or of combining multiple experts qualities BID12 BID28 BID33 BID40. Imitation learning is the control-specific term for the production of a student policy from either an expert policy or the behavioral demonstrations of an expert. One basic algorithm is behavioral cloning, which refers to supervised training of the policy from state-action pairs. In the most simple case it only requires examples from the expert. A broader setting is that in which more liberal queries to the expert are permitted; e.g. for the online-imitation setting as in DAGGER BID32. This setting is often satisfied e.g. if we wish to combine behavior from multiple experts. One-shot imitation is a concept which means that a trained system, at test time, can watch an example behavior and imitate it, as, for instance, in BID7. More similar to our work is the setting examined by, in which full-body humanoid movements were studied. Compared with this latter work, we will employ an architecture here that encourages imitation of motor details, rather than overall movement type, and we scale our approach to more expert demonstrations. The most similar work also demonstrates large-scale one-shot humanoid tracking and was contemporaneously published BID4; the approach they described involves direct tracking as well as failure recovery, but relative to our work the authors do not consider skill reuse. The notion of motor primitives is widespread in neuroscience, where there is evidence that lower dimensional control signals can selectively coordinate and blend behaviors produced by spinal circuits BID3, and that the cortex organizes the space of primitive motor behaviors BID9. In our setting, motor primitives refer to the reusable embedding space learned from many related behaviors and the associated context-modulable policy capable of generating sensory-feedback-stabilized motor behavior when executed in an environment. The particular architecture we consider is inspired by the formalization presented in BID41, which places a probabilistic latent bottleneck on the sensory-motor mapping. In the robotics literature, there is a rich line of research into various parameterizations of motion trajectories used for robot control. A class of these are referred to as "movement primitives" (e.g. BID34, including the "probabilistic movement primitives" of BID27 (see also e.g. BID26 . These approaches can be seen as specific implementation choices for a certain notion of motor primitive, which emphasize the parameterization and learning of movement trajectories from repeated demonstrations BID27 BID20, rather than learning the actuation/stabilization element, which is often handled by a prespecified PID controller. It has previously been recognized that linear-feedback policies can work well around optimal trajectories or limit cycles even for high DoF bodies. These can be obtained by sample-based optimization (e.g. BID6) or by differential dynamic programming BID24. For linear-quadratic-Gaussian control BID1 or differential dynamic programming BID19 BID13, we obtain feedback policies where the feedback terms are computed from the value function, amounting effectively to feedbackstabilized plans. Work by BID23 has shown that linear-feedback policies ing from trajectory optimization can be used to train neural networks. We employ a similar idea to transfer optimal behavior from an existing policy, observing that an optimal policy implicitly reflects the structure of the (local) value landscape and appropriately functions as a feedback controller. Figure 1: Examples of representative experts learned from motion capture. From top to bottom, these are "run and dodge", "cartwheel", "backflip", and "twist". See accompanying video. Note that these four behaviors will be used as representative examples for validation in single-skill transfer experiments. In this section, we will first briefly describe the expert policies used in this work (Sec. 2.1). We then describe the Neural Probabilistic Motor Primitive architecture and objective (Sec. 2.2). We then describe two approaches for training the module offline (Sec. 2.3). In order to study how to transfer and consolidate experts, we must be able to generate adequate quantities of expert data. For this work, we use expert policies trained to reproduce motion capture clips. The approach we use for producing experts is detailed more fully in BID22 and largely follows BID30. It yields time-indexed neural network policies that are robust to moderate amounts of action noise (see appendix A for additional details on the training procedure). Some examples of the ing single-skill time-indexed policies that are obtained from this procedure are depicted in FIG4. All our experts were trained in MuJoCo environments. Data We use the CMU Mocap database 1, which contains more than 2000 clips of varying lengths from more than 100 subjects. The motions in this dataset are quite varied, including many clips of walking, turning, running, jumping, dancing, various hand movements, and many more idiosyncratic behaviors. From this, we selected various clips of generic whole-body movements -any clips longer than 6 seconds were cut into smaller pieces yielding approximately 3000, roughly 2-6 second snippets. Just over half of these are generic locomotion such as walking, running, jumping and turning. The rest of the clips mostly contained diverse hand movements while standing. We trained one expert policy per selected snippet, yielding 2707 expert policies in our training set. Our goal is to obtain a motor primitive module that can flexibly and robustly deploy, sequence, and interpolate a diverse set of skills from a large database of reference trajectories without any manual alignment or other processing of the raw experts. This requires a representation that does not just reliably encode all behavioral modes but also allows effective indexing of behaviors for recall. To ensure plausible and reliable transitions it is further desirable that the encoding of similar behaviors should be close in some sense in the representation space. Compression of many expert skills via a latent variable inverse model We achieve this goal by training an autoregressive latent variable model of the state-conditional action sequence which, at training time, is conditioned on short look-ahead snippets of the nominal/reference trajectory (see FIG0). This architecture has the general structure of an inverse model, which produces actions based on the current state and a target. The architecture and training scheme are designed for the embedding space to reflect short-term motor behavior. As we demonstrate below, this allows for the selective execution of particular behavioral modes and also admits one-shot imitation via the trajectory encoder. We use a model with a latent variable z t at each time step, modelling the state conditional action distribution. The encoder and decoder are distributions q(z t |z t−1, x t) and π(a t |z t, s t) where s t is the state as in preceding sections and x t is concatenation of a small number of future states x t = [s t, ..., s t+K]. The encoder and decoder are MLPs with two and three layers, respectively. For architecture and experimental details see appendix B. The generative part of the model is given by: DISPLAYFORM0 Temporally nearby trajectory snippets should have a similar representation in the latent space. To implement this intuition, we choose an AR process as a weak prior: DISPLAYFORM1 where σ = √ 1 − α 2, ensuring that marginally z t ∼ N (0, I), and set α = 0.95 in experiments unless otherwise stated. In subsequent efforts, it may be interesting to investigate different values of α and learnable priors. In order to train this model, we consider the evidence lower bound (ELBO): DISPLAYFORM2 with a β parameter to tune the weight of the prior. For β = 1 this objective forms the well-known variational lower bound to log p(a 1:T |s 1:T). This objective can be optimized using supervised learning (i.e. behavioral cloning from noisy rollouts) offline. Note we chose not to condition the encoder on actions, since we are interested in one-shot imitation in settings where actions are unobserved. We experimented with different values of K and obtained similar performance. All the reported in this paper use K = 5. Our architecture effectively implements a conditional information bottleneck between the desired future trajectory x t and the action a t given the past latent state z t−1 (similar to Alemi et al. FORMULA0). As discussed above the auto-correlated prior encourages an encoding in which temporally nearby latent states from the same trajectory tend to be close in the latent space, and the information bottleneck more generally encourages a limited dependence on x t with z t forming a compressed representation of the future trajectory as required for the action choice. When transferring knowledge from an expert policy to a student we would like the student to replicate the expert's behavior in the full set of states plausibly visited by the expert. In our case, experts trained to reproduce single clips can be conceptualized as nonlinear feedback controllers around a nominal trajectory, and the manifold of states visited by experts can be thought of as a tube around that reference. We require the student to be able to operate successfully in and remain close to this tube even in the face of small perturbations. Formally, to ensure that the student retains expert robustness, we would like expert actions µ E (s) and student actions µ θ (s) to be close under a plausible (noisy) expert state distribution ρ E. A surrogate loss used in imitation learning as well as knowledge transfer is the quadratic loss between actions (or activations BID36). DISPLAYFORM0 Behavioral cloning can refer to optimization of this objective, where ρ E is replaced with an empirical distribution of a set of state-action pairs S. This works well if S adequately covers the state distribution later experienced by the student. Anticipating and generating an appropriate set of states on which to train the student typically requires many rollouts and can thus be expensive. Since we are aiming to compress the behavior of thousands of experts we desire a computationally efficient method. We investigate two schemes that allow us to record the experts' state-action mappings on a small-sample estimate of the experts' state distributions and to then train the student via supervised learning. Both schemes are convenient to implement in a regular supervised learning pipeline and require neither querying many experts simultaneously (which limits scalability when dealing with thousands of experts) nor execution of the student at training time. Behavioral cloning from noisy rollouts The first approach amounts to simply gathering a number of noisy trajectories from the expert (either under a stochastic policy or with noise injection) while logging the optimal/mean action of the expert instead of the noisy action actually executed. A version of this is equivalent to the DART algorithm of BID16. We then perform behavioral cloning from that data. Specifically, given an expert policy π E, let µ E (s) be the mean action of the expert in state s. To obtain noisy rollouts, we run π η E, the expert with moderate action noise (η) to obtain a set of data {s DISPLAYFORM1 And we optimize the policy according to Eqn. 4, with the expectation over s ∼ ρ E being approximated by a sum over the set of state and expert-actions collected. While we expect this approach can work well, we do not expect it to be particularly efficient insofar as the expert may need to be executed for many rollouts. Linear-feedback policy cloning (LFPC) The second approach, which we refer to as linearfeedback policy cloning (LFPC), logs the action-state Jacobian as well as the expert action along a single nominal trajectory. The Jacobian can be used to construct a linear feedback controller which gives target actions in nearby perturbed states during training (described below). This approach is not intended to outperform behavioral cloning, as this should not be possible for arbitrary quantities of expert rollout data. Instead the motivation for LFPC is to do as well as behavioral cloning while using considerably fewer expert rollouts. As pointed out above, experts trained to reproduce single clips robustly can be thought of as nonlinear feedback controllers around this nominal trajectory. The nominal trajectory refers to the sequence of nominal state-action pairs {s t, a t} 1...T obtained by executing µ E (s) recursively from an initial point s 0. Since expert behavior in our setting is well characterized by single nominal trajectories, we expect we can capture the relevant behavior of the expert by a linearization around the nominal trajectory 3.Let δs be a small perturbation of the state and let J = dµ E (s) ds | s=s be the Jacobian. Then DISPLAYFORM2 This linearization induces a linear-feedback-stabilized policy that at each time-step has a nominal action a t, but also expects to be in state s t, and correspondingly adjusts the nominal action with a linear correction based on discrepancy between the nominal and actual state at time t: DISPLAYFORM3 We empirically validated that a linear feedback policy about the nominal trajectory of the expert can approximate the expert behavior reasonably well for clips we examine (see Fig. 3).Above we presented the expert as a feedback controller operating in a tube around some nominal trajectory with states s 1,..., s T, actions a 1,..., a T, and Jacobians J 1,..., J T. We approximate ρ E with the distribution of states introduced by state perturbations around this nominal trajectory: DISPLAYFORM4 However, this objective still requires expert evaluations at the perturbed states. Using the linearization described above we can replace the expert action µ E (s + δs) with the Jacobian-based linearfeedback policy µ F B (s + δs), which is available offline. This yields the LFPC objective: DISPLAYFORM5 One potentially important choice is the perturbation distribution ∆(s). Ideally, we would like ∆(s) to be the state-dependent distribution induced by physically plausible transitions, but estimating this distribution may require potentially expensive rollouts which we are trying to avoid. A cheaper object to estimate is the stationary transition noise distribution induced by noisy actions, which can be efficiently approximated from a small number of trajectories. Empirically, we found the objective 8 to be relatively robust to some variations in ∆, and we use a fixed marginal distribution for all clips. Objective 8 bears interesting similarities to approaches such as denoising autoencoders BID43, where networks can learn to ignore local noise perturbations on inputs sampled from a high-dimensional noise distribution. Further, BID23 successfully distill feedback policies obtained from a planner. One question left open by this latter work is that of how much data might be required. Empirically we show in the experiments below that the augmented objective 8 can produce the desired robustness even from a very limited set of states. Figure 3: Comparisons of trajectory rollouts for 4 reference behaviors for the nominal trajectory and at varying noise levels. Note that the score is determined by similarity to motion-capture reference and the expert may be slightly suboptimal so slight improvements on the expert may arise by chance. There are multiple, relevant perspectives on LFPC. From one perspective, LFPC amounts to a data augmentation method. From another vantage, the approach attempts to match the mean action as well as the Jacobian at the set of relevant behavioral states, here sampled along the nominal trajectory. In settings where expert behavior is more diverse or multimodal, LFPC should be applied to states which representatively cover relevant behavioral modes or perhaps are expanded backwards from goal states (roughly similar to the procedure used to expand LQR-trees by Tedrake 2009). Explicit Jacobian matching has been proposed elsewhere, for example in. See appendix C for further disambiguation relative to other approaches. To train our Neural Probabilistic Motor Primitive architecture using LFPC we can adapt the objective in Eqn. 3 as follows: DISPLAYFORM6 log π(a t + J t δs t |s t + δs t, z t) + β log p z (z t |z t−1) − log q(z t |z t−1, x t + δx t), where δs t are i.i.d. perturbations drawn from suitable perturbation distribution ∆ and δx t is the concatenation of [δs t, δs t+1, ..., δs t+K]. To ground our in a simple setting, we begin with transfer of a single-skill, time-indexed policy from one network to another. We compare the performance of various time-indexed policies for each of the experts depicted in FIG4 Fig. 3 ). For additional validation, see appendix D.Figure 4: Performance relative to expert policies for trained neural probabilistic motor primitive models. Performance of model variations are compared on training and testing data. We compare models trained using cloning with 100 trajectories per expert for different levels of regularization, using a smaller latent space of dimension 20 rather than 60 in all other experiments, as well as LFPC. Having validated that single skills can be transferred, we next consider how well we can compress behaviors of the 2707 experts in our training set into the neural probabilistic motor primitive architecture. Assessing the models using the action-reconstruction loss is not very intuitive since it does not capture model behavior in the environment. Instead we report a more relevant measure based on expert imitation. Here we encode an expert trajectory into a sequence of latent variables and then execute the policy in the environment conditioned on this sequence. Note that this approach is openloop with respect to the latents while being closed-loop with respect to state. We can then compare the performance of the trained system against experts on training and held-out clips according to the tracking reward used to train the experts originally. To account for different expert reward scales we report performance relative to the expert policy. Importantly, that this approach works is itself a partial validation of the premise of this work, insofar as open-loop execution of action sequences usually trivially fails with minor perturbations. The trained neural probabilistic motor primitive system can execute behaviors conditioned on an open-loop noisy latent variable trajectory, implying that the decoder has learned to stabilize the body during latent-conditioned behavior. There are a few key takeaways from the comparisons we have run (see Fig. 4). Most saliently cloning based on 100 trajectories from each expert with a medium regularization value (β = 0.1) works best. LFPC with comparable parameters works less well here, but has qualitatively fairly similar performance. Our ablations show that regularization and a large latent space are important for good . We also set the autoregressive parameter α = 0 (.95 in other runs), making the latent variables i.i.d.. This hurts performance, validating our choice of prior. We have no expectation that trajectories well outside the training distribution are likely to be either representable by the encoder or executable by the decoder. Nevertheless, when one-shot imitation of a trajectory fails, a natural question is whether the decoder is incapable of expressing the desired actions, or the encoder fails to encode the trajectory in such a way that the decoder will produce it. Figure 5: These panels consist of visualizations of the PCA latent space with comparisons in this space between one-shot latent-variable sequences and optimized latent variable sequences for various behaviors: A. Run B. Backwards walking C. Jumping. Running executes well based on the one-shot trajectory so serves as a reference for which optimization is not noticeably different. Walking backwards and jumping one-shot imitations fail, but are noticeably improved by optimization. We propose an analysis to distinguish this for borderline cases. For held out trajectories that yield unsatisfying performance on one-shot imitation, we can simply optimize directly: DISPLAYFORM0 where µ θ is the decoder mean. Empirically we see that this optimization meaningfully improves the executed behavior, and we visualize the shift in a three-dimensional space given by the first three principal components in Fig. 5.We exhibit three examples where we visualize the original latent trajectory as well as the optimized latent trajectory. Performance is significantly improved (see supplementary video), showing the latent space can represent behaviors for which one-shot imitation fails. However execution remains imperfect suggesting that while much of the fault may lie with the encoder, the decoder still may be slightly undertrained on these relatively rare behavior categories. Quantitatively, among a larger set of clips with less than 50% relative expert performance for one-shot imitation we found that optimization as described above improved median relative expert performance from 43% to 78%.Other exploratory probes of the module suggest that it is possible in certain cases to obtain seamless transitioning between behaviors by concatenating latent-variable trajectories and running the policy conditioned on this sequence (e.g. in order to perform a sequence of turns). See additional supplementary video. Reuse of motor primitive module Finally, we experimented with reuse of the decoder as a motor primitive module. We treat the latent space as a new custom action space and train a new high-level (HL) policy to operate in this space. At each time-step the high-level policy outputs a latent-variable (a) Median return value across 10 seeds for the goto-target task vs learner steps. Compared to a very weakly regularized module (β = 0.001), more regularized motor primitives modules both trained faster and achieved higher final performance.(b) Our model is able to track the target speed accurately. Shown here are target speed and actual speed in the egocentric forward direction for three episodes. The reward function is a Gaussian centered at the target speed. The shaded region corresponds to ± one standard deviation. z t. The actual action is then given by the motor primitive module p(a t |s t, z t). For training we used SVG BID10 with the Retrace off-policy correction for learning the Q-function BID25.A natural locomotion task that can challenge the motor module is a task which requires abrupt, frequently redirected movement with sharp turns and changes of speed. To implement this we provide the higher-level controller with a target that is constant until the humanoid is near it for a few timesteps at which point it randomly moves to another nearby location. While no single task will comprehensively probe the module, performing well in this task demands a wide range of quick locomotion behavior. With only a sparse task reward, the HL-controller can learn to control the body through the learned primitive space, and it produces rather humanlike task-directed movement. We observed that more regularized motor primitive modules had more stable initial behavior when connected to the untrained high-level controller (i.e. were less likely to fall at the beginning of training). Compared to a very weakly regularized module (β = 0.001), more regularized motor primitives modules both trained faster and achieved higher final performance (see FIG2). We also investigated a go-to-target task with bumpy terrain that is unobserved by the agent. The fact that our model can learn to solve this task demonstrates its robustness to unseen perturbations for which the motor primitive module was not explicitly trained. In another experiment we investigated a task in which the agent has to move at a random, changing target speed. This requires transitions between qualitatively different locomotion behavior such as walking, jogging, and running (see FIG2).See an extended video of these experiments. In a final reuse experiment, we consider an obstacle course requiring the agent to jump across gaps (as in BID22). We were able to solve this challenging task with a high-level controller that operated using egocentric visual inputs (see the main supplementary video).We emphasize a few points about these to impact their importance: Using a pretrained neural probabilistic motor primitives module, new controllers can be trained effectively from scratch on sparse reward tasks, the ing movements are visually rather humanlike without additional constraints implying that the learned embedding space is well structured, and the module enables fairly comprehensive and smooth coverage for the purposes of physics-based control. In this paper we have described approaches for transfer and compression of control policies. We have exhibited a motor primitive module that learns to represent and execute motor behaviors for control of a simulated humanoid body. Using either a variant of behavioral cloning or linear feedback policy cloning we can train the neural probabilistic motor primitive sytem to perform robust one-shotimitation, and with the latter we can use relatively restricted data consisting of only single rollouts from each expert. While LFPC did not work quite as well in the full-scale model as cloning from noisy rollouts, we consider it remarkable that it is possible in our setting to transfer expert behavior using a single rollout. We believe LFPC holds promise insofar as it may be useful in settings where rollouts are costly to obtain (e.g. adapted to real-world robotic applications), and there is room for further improvement as we did not carefully tune certain parameters, most saliently the marginal noise distribution ∆.The ing neural probabilistic motor primitive module is interpretable and reusable. We are optimistic that this kind of architecture could serve as a basis for further continual learning of motor skills. This work has been restricted to motor behaviors which do not involve interactions with objects and where a full set a of behaviors are available in advance. Meaningful extensions of this work may attempt to greatly enrich the space of behaviors or demonstrate how to perform continual learning and reuse of new skills. The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217.We used the reparametrization trick BID15 BID31 to train the model and used stochastic gradient descent with ADAM BID14 with a learning rate of 0.0001. In the case of models trained on 100 trajectories per expert we used minibatches of 512 subsequences of length 30.For LFPC we sampled 32 subsequences of length 30 and produced 5 perturbed state sequences per subsequence. In preliminary experiments the length of the subsequences did not have a major impact on model performance. Firstly, we note that the emphasis of the proposal in this work is to match the responsivity of the expert policy in a neighborhood around each state. This is distinct from activation matching or KL matching where the emphasis is on matching the action/activation distribution for a particular state BID33 BID40. Secondly, we emphasize that the kind of robust knowledge transfer we discuss here is distinct from that which is seen to be important in other settings. For example BID36 provide a line of reasoning that involves training a student system to match the exact activations of a teacher in the presence of perturbations on the student inputs. This logic is sound in the setting of large-scale vision systems. However in the context of control policies, this would look like: DISPLAYFORM0 This essentially means that the student policy is learning to "blindly" reproduce the action of the expert exactly, despite input perturbations. While this is well motivated if the noise is thought to be orthogonal to the proper functioning of the system, this is a very bad idea for control, where you need to pay close attention to small input perturbations. Technically, this amounts to setting the local feedback to zero, and behaving in a sort of open-loop-like fashion. Locomotion behavior is, at least in the simplest case roughly a limit cycle. In an additional experiment to test LFPC we gathered three gait cycles of running behavior and performed LFPC. Note that here the student policy need not be time-indexed even when the demonstrations were time-indexed. This restricted case shows striking generalization in the presence of noise (see The limited reference data originating from a time-indexed policy has been projected into the same space (green). Observe that the rollouts are considerably noisier and consistently deviate from the reference trajectory, nevertheless the cloned-policy trajectories return to the limit cycle.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Neural Probabilistic Motor Primitives compress motion capture tracking policies into one flexible model capable of one-shot imitation and reuse as a low-level controller.

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.


Question: represent the article for finding a single-sentence tldr!

Question: Data augmentation is a useful technique to enlarge the size of the training set and prevent overfitting for different machine learning tasks when training data is scarce. However, current data augmentation techniques rely heavily on human design and domain knowledge, and existing automated approaches are yet to fully exploit the latent features in the training dataset. In this paper we propose \textit{Parallel Adaptive GAN Data Augmentation}(PAGANDA), where the training set adaptively enriches itself with sample images automatically constructed from Generative Adversarial Networks (GANs) trained in parallel. We demonstrate by experiments that our data augmentation strategy, with little model-specific considerations, can be easily adapted to cross-domain deep learning/machine learning tasks such as image classification and image inpainting, while significantly improving model performance in both tasks. Our source code and experimental details are available at \url{https://github.com/miaojiang1987/k-folder-data-augmentation-gan/}. Deep learning and machine learning models produce highly successful when given sufficient training data. However, when training data is scarce, overfitting will occur and the ing model will generalize poorly. Data augmentation(DA) ameliorates such issues by enlarging the original data set and making more effective use of the information in existing data. Much prior work has centered on data augmentation strategies based on human design, including heuristic data augmentation strategies such as crop, mirror, rotation and distortion BID15 BID21 Proceedings of the 1 st. Copyright 2019 by the author(s). et al., 2003), interpolating through labeled data points in feature spaces BID5, and adversarial data augmentation strategies based on BID22 BID8. These methods have greatly aided many deep learning tasks across several domains such as classification BID15, image segmentation BID24 and image reconstruction/inpainting BID0.Despite their success, these DA methods generally require domain-specific expert knowledge, manual operations and extensive amount of tuning depending on actual contexts BID3 BID6. In particular, the need to directly operate on existing data with domain knowledge prevents many previous data augmentation strategies from being applicable to more general settings. To circumvent the need for specific domain knowledge in data augmentation, more recent work BID1 utilizes generative adversarial networks(GANs) BID10 to produce images that better encode features in the latent space of training data. By alternatively optimizing the generator G and the discriminator D in the GAN, the GAN is able to produce images similar to the original data and effectively complement the training set. It has been shown in experiments BID1 that GAN-based methods have indeed significantly boosted the performance of classifiers under limited data through automatic augmentation, but applications into other tasks are yet to be explored. Furthermore, given the computational complexity of GANs, a natural way to reduce runtime is to consider parallelism BID13 BID7.In view of these considerations, we propose in this paper Parallel Adaptive Generative Adversarial Network Data Augmentation(PAGANDA), where the training set adaptively enriches itself with sample images automatically constructed from Generative Adversarial Networks (GANs) trained in parallel. Our contributions can be summarized as follows:• We propose a general adaptive black-box data augmentation strategy to diversify enhance training data, with no task-specific requirements.• We also include in our model a novel K-fold parallel framework, which helps make the most use of the existing data.• Experiments over various datasets and tasks demonstrate the effectiveness of our method in different context. Data Augmentation(DA) Previous work on data augmentation can be classified into several groups. Traditional Heuristic DA strategies such as crop, mirror, rotation and distortion BID15 BID21 have found their way in many deep classification tasks, but these method generally require domain-specific expert knowledge, manual operations and extensive amount of tuning depending on actual contexts BID3 BID6. Other DA methods used interpolation through labeled data points in feature spaces BID5, but their dependence on class labels makes them inapplicable for tasks with weak or no supervision. Adversarial Data Augmentation strategies BID22 BID8 choose from a select number of transformation operations to maximize the loss function of the end classification model involved in the task. While good motivations for our methods, these methods make strong assumptions over the types of augmentation and are difficult to generalize. BID18 BID4 transform the problem of choosing data augmentation strategies into a reinforcement learning policy search problems, but the choice of augmentation methods are still limited and the reinforcement learning algorithms have non-trivial computation overhead in addition to the main task. ML problems with limited data For classfication with limited samples, BID19 proposed a convolutional neural network(CNN) to classify environmental sounds with limited samples. Other algorithms have been proposed in BID9 BID25 ), yet many of them have assumptions/constraints that hurts their capacity for generalization. For unsupervised learning models, recent research on sample complexity reduction in GAN training seeks to reparametrize the input noise using variational inference BID12 BID16, but this method has severe mathematical limitation that prevents further generalization. BID23 adopts transfer learning techniques to train a new GAN for limited data from a pre-trained GAN network. While effective, this approach requires a pre-trained network in the first place and doesn't apply to the cases when data is scarce. Parallel/Distributed GANs BID13 BID7 proposed the first distributed multidiscriminator generative adversarial models, yet these models require large datasets to train and have great computational complexity. Moreover, these models are trained on In this section we describe the details of Parallel Adaptive Generative Adversarial Network Data Augmentation (PAGANDA). Our method consists of three interrelated components: generative data augmentation, parallel image generation with fold division, and adaptive weight adjustment. To ensure that make full use of the information contained in the existing images, the first part of our method involves generative data augmentation, which constructs varied images given the training set by repeatedly generating samples from and adding samples to the training set using a generative adversarial net. We start off with a limited training set, and consecutively run the generative adversarial net using the set. After running a fixed number t of regular training epochs, we proceed to the augmentation epoch where the augmentation is conducted. During the augmentation epoch, we extract a number of sample images from the generator G using standard procedures of sample image generation as described in BID17 BID11. For this batch of samples, we calculate the Inception Score(IS) as defined by BID20 to measure the authenticity of the images generated, which we denote as w. Here the Inception Score provides a metric of the power of generator to produce realistic images: the higher the value of w, the more power the corresponding generator G. This batch of images are then added back into the original training set for subsequent augmentation epochs. We alternate running t regular training epochs and the augmentation epoch for a fixed number of times or until convergence. FIG0 is a flow-chart of our procedure. Notice that our procedure is agnostic to the specific architecture of generative adversarial net used to augment the training data. Since GANs capture the information in the latent feature space of the images and translate such information into generated images, our method has the capacity to reveal the potential features that are possibly not visually evident in the original training images. Moreover, compared with many other data augmentation strategies which require one to pre-define the operations to be carried on the images, our method automatically enriches the training set and does not require human intervention. The second part of our method consists of a parallel data generation strategy, inspired by K-fold cross validation in machine learning BID2. Dividing the training data into K folds at the beginning, we run in parallel K independent generators DISPLAYFORM0. Each generator G i is trained on one of data groups, and each data group i consists of K − 1 folds of the training set, except for the i-th fold. After images are generated in each generator G i in the training epochs, the sample images produced by each generator during the augmentation epoch are fed back into the respective training data groups. To allow for maximal usage of each generated image, we insert the images in a way such that the images generated by one generator G i are sent to the training data groups corresponding to all other K − 1 generators except for that corresponding to G i. This is to insure that the different generators in parallel have access to as many varied data pieces as possible in subsequent steps of training, so as to prevent overfitting and bolster the robustness of our strategy. FIG1 demonstrates our algorithm. Furthermore, to determine which generators are the most effective in generating authentic images, we introduce adaptive generator weighting at each augmentation epoch. At the initial stage, all the generators are treated equally. Before the batch of sample images generated by one generator G i are sent to the data group corresponding to other K − 1 generators, we collect the inception scores {w i} K i=1 computed in section 3.1. Since higher inception scores imply better performance of the generator, we define the generator weight p i of a generator G i as DISPLAYFORM0 and use this weight to determine how many images should be sampled from generator G i to be sent to other data groups for subsequent training in the very next augmentation epoch. When the total number of samples to be collected from generators are fixed, this method enables generators with better realistic image generation power to contribute more to the future training data groups. More realistic training sets thus augmented, in turn, exert more positive influence on the images to be generated. Note that all three strategies introduced go hand in hand, with no need for model specific considerations. As demonstrated by our experiments Section 4, training different GANs in parallel from different folds of data substantially boosts the quality of the training set and that of the generated images. To illustrate the effectiveness of PAGANDA for multiple machine learning tasks, we have applied our data augmentation method to two tasks: image classification and image inpainting. For image classification we constructed our dataset from Imagenet and Cifar-10 by randomly drawing 5000 images from each dataset respectively and applied PAGANDA on these reduced datasets. The augmented datasets are then used to train an AlexNet CNN classifier, and the classification are compared with the obtained from an AlexNet trained on the corresponding original unaugmented datasets. For image inpainting, we constructed our datasets from Places dataset. We chose images from the Ocean subset from Places to obtain the reduced Places Dataset. To ensure the parallelism of the experiments, we trained our model in a multi-threaded environment to make simultaneously training. Under such a setting, all the data groups are trained at the same time, and each GAN model corresponding to each data group is trained in a separate thread. All of our experiments are conducted on a server with Tesla-V GPU (32GB RAM, 7.8 TeraFLOPS) and Intel Xeon Processor E5 (2.00 GHz). For our experiments on classification, we first augment the reduced Cifar-10 and reduced Imagenet datasets, and then train the CNN classifier with the augmented dataset. The classifier accuracies with and without augmentation are listed in TAB0 below. For the task of inpainting, we augment the reduced dataset constructed in the experiment. Without loss of generality, we train a WGAN-GP model for inpainting from the augmented dataset. We then select testing images that are not selected in the training set, and add to them gray masks covering the center part of these images. We then applied our trained WGAN-GP to generate patches that cover the masked portion of the inpainting image. Figure 4 lists a couple of generated images with and without augmentation. Visual comparisons demonstrate the effectiveness of our method. In sum, our paper shows that PAGANDA effectively improves the performances for different machine learning tasks with little task-specific considerations. Our strategy is not only simple to implement, but also demonstrates capability to generate onto different settings since it does not require specific information about the task being analyzed. As a further step, we are investigating the relationship between our proposed approach and other established methods. We hope to apply our idea to other generative models such as VAE BID14 and further optimize our strategy using recent theoretical advances, and wish to investigate the scenarios where the tasks involved are interrelated. Application wise, we are aiming to apply our parallel GAN model to multi-modal image synthesis/generation where training data is limited.

We present an automated adaptive data augmentation that works for multiple different tasks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep neural networks with millions of parameters may suffer from poor generalizations due to overfitting. To mitigate the issue, we propose a new regularization method that penalizes the predictive distribution between similar samples. In particular, we distill the predictive distribution between different samples of the same label and augmented samples of the same source during training. In other words, we regularize the dark knowledge (i.e., the knowledge on wrong predictions) of a single network, i.e., a self-knowledge distillation technique, to force it output more meaningful predictions. We demonstrate the effectiveness of the proposed method via experiments on various image classification tasks: it improves not only the generalization ability, but also the calibration accuracy of modern neural networks. Deep neural networks (DNNs) have achieved state-of-the-art performance on many machine learning applications, e.g., computer vision , natural language processing , and reinforcement learning . As the scale of training dataset increases, the size of DNNs (i.e., the number of parameters) also scales up to handle such a large dataset efficiently. However, networks with millions of parameters may incur overfitting and suffer from poor generalizations (; . To address the issue, many regularization strategies have been investigated in the literature: early stopping, L 1 /L 2 -regularization , dropout , batch normalization and data augmentation Regularizing the predictive or output distribution of DNNs can be effective because it contains the most succinct knowledge of the model. On this line, several strategies such as entropy maximization and angular-margin based methods have been proposed in the literature. They can be also influential to solve related problems, e.g., network calibration , detection of out-of-distribution samples and exploration of the agent in reinforcement learning . In this paper, we focus on developing a new output regularizer for deep models utilizing the concept of dark knowledge , i.e., the knowledge on wrong predictions made by DNN. Its importance has been first evidenced by the so-called knowledge distillation and investigated in many following works (; ; ;). While the related works use the knowledge distillation (KD; Hinton et al. 2015) to transfer the dark knowledge learned by a teacher network to a student network, we regularize the dark knowledge itself during training a single network, i.e., self-knowledge distillation. Specifically, we propose a new regularization technique, coined class-wise self-knowledge distillation (CS-KD) that matches or distills the predictive distribution of DNNs between different samples of the same label (class-wise regularization) and augmented samples of the same source (sample-wise regularization) as shown in Figure 1. One can expect that the proposed regularization method forces DNNs to produce similar wrong predictions if samples are of the same class, while the conventional cross-entropy loss does not consider such consistency on the wrong predictions. We demonstrate the effectiveness of our regularization method using deep convolutional neural networks, such as ResNet and DenseNet trained for image classification tasks on various datasets including CIFAR-100 , TinyImageNet 1, CUB-200-2011 , Stanford Dogs , and MIT67 datasets. We compare or combine our method with prior regularizers. In our experiments, the top-1 error rates of our method are consistently smaller than those of prior output regularization methods such as angular-margin based methods and entropy regularization . In particular, the gain tends to be larger in overall for the top-5 error rates and the expected calibration errors , which confirms that our method indeed makes predictive distributions more meaningful. Moreover, we investigate a variant of our method by combining it with other types of regularization method for boosting performance, such as the mixup regularization and the original KD method. We improve the top-1 error rate of mixup from 37.09% to 31.95% and that of KD from 39.32% to 35.36% under ResNet trained by the CUB-200-2011 dataset. Our method is very simple to use, and would enjoy a broader usage in the future. In this section, we introduce a new regularization technique, named class-wise self-knowledge distillation (CS-KD). Throughout this paper, we focus on fully-supervised or classification tasks, and denote x ∈ X as an input and y ∈ Y = {1, ..., C} as its ground-truth label. Suppose that a softmax classifier is used to model a posterior distribution, i.e., given the input x, the predictive distribution is as follows:, where f = [f i] denotes the logit-vector of DNN, parameterized by θ and T > 0 is the temperature scaling parameter. We first consider matching the predictive distributions on samples of the same class, which distills their dark knowledge into the model itself. To this end, we propose a class-wise regularization loss that enforces consistent predictive distributions in the same class. Formally, given input x and another randomly sampled input x having the same label y, it is defined as follows: where KL denotes the Kullback-Leibler (KL) divergence and θ is a fixed copy of the parameters θ. As suggested by , the gradient is not propagated through θ to avoid Algorithm 1 Class-wise self-knowledge distillation (CS-KD) Initialize parameters θ. while θ has not converged do for (x, y) in a sampled batch do g θ ← 0 Get another sample x randomly which has the same label y from the training set. Generate x aug, x aug using data augmentation methods. Compute gradient: Update parameters θ using gradients g θ. end while the model collapsing issue. Similar to the knowledge distillation method (KD) by , L cls matches two predictions. While the original KD matches predictions of a sample from two networks, we do predictions of different samples from a single network. Namely, our method performs self-knowledge distillation. In addition to enforcing the intra-class consistency of predictive distributions, we apply this idea to the single-sample scenario by augmenting the input data. For a given training sample x, the proposed sample-wise regularization loss L sam is defined as follows: where x aug is an augmented input that is modified by some data augmentation methods, e.g., resizing, rotating, random cropping (; ;, cutout , and auto-augmentation . In our experiments, we use standard augmentation methods for ImageNet (i.e., flipping and random sized cropping) because they make training more stable. In summary, the total training loss L tot is defined as a weighted sum of the two regularization terms with cross-entropy loss as follows: where λ cls and λ sam are balancing weights for each regularization, respectively. Note that the first term is the cross-entropy loss of softmax outputs with temperature T = 1. In other words, we not only train the true label, but also regularize the wrong labels. The full training procedure with the proposed loss L tot is summarized in Algorithm 1. Datasets. To demonstrate our method under general situations of data diversity, we consider various image classification tasks including conventional classification and fine-grained classification tasks. We use CIFAR-100 and TinyImageNet 2 datasets for conventional classification tasks, and CUB-200-2011 , Stanford Dogs , and MIT67 datasets for fine-grained classification tasks. Note that fine-grained image classification tasks have visually similar classes and consist of fewer training samples per class compared to conventional classification tasks. We sample 10% of the training dataset randomly as a validation set for CIFAR-100 and TinyImageNet and report the test accuracy based on the validation accuracy. For the fine-grained datasets, we report the best validation accuracy. Network architecture. We consider two state-of-the-art convolutional neural network architectures: ResNet and DenseNet . We use standard ResNet-18 with 64 filters and DenseNet-121 with growth rate of 32 for image size 224 × 224. For CIFAR-100 and TinyImageNet, we modify the first convolutional layer 3 with kernel size 3 × 3, strides 1 and padding 1, instead of the kernel size 7 × 7, strides 2 and padding 3, for image size 32 × 32. Evaluation metric. For evaluation, we measure the following metrics: • Top-1 / 5 error rate. Top-k error rate is the fraction of test samples for which the correct label is amongst the top-k confidences. We measured top-1 and top-5 error rates to evaluate the generalization performance of the models. • Expected Calibration Error (ECE). ECE approximates the difference in expectation between confidence and accuracy, by partitioning predictions into M equally-spaced bins and taking a weighted average of bins' difference of confidence and accuracy, i.e., ECE = • Recall at k (R@k). Recall at k is the percentage of test samples that have at least one example from the same class in k nearest neighbors on the feature space. To measure the distance between two samples, we use L 2 -distance between their average-pooled features in the penultimate layer. We compare the recall at 1 scores to evaluate intra-class variations of learned features. Hyper-parameters. All networks are trained from scratch and optimized by stochastic gradient descent (SGD) with momentum 0.9, weight decay 0.0001 and an initial learning rate of 0.1. The learning rate is divided by 10 after epochs 100 and 150 for all datasets and total epochs are 200. We set batch size as 128 for conventional, and 32 for fine-grained classification tasks. We use standard flips, random resized crops, 32 for conventional and 224 for fine-grained classification tasks, overall experiments. Furthermore, we set T = 4, λ cls = 1 for all experiments and λ sam = 1 for experiments on fine-grained classification tasks, and λ sam = 0 on conventional classification tasks. To compute expected calibration error (ECE), we set the number of bins M as 20. Baselines. We compare our method with prior regularization methods such as the state-of-the-art angular-margin based methods and entropy regularization . They also regularize predictive distributions as like ours. • AdaCos . 4 AdaCos dynamically scales the cosine similarities between training samples and corresponding class center vectors to maximize angular-margin. • Virtual-softmax . Virtual-softmax injects an additional virtual class to maximize angular-margin. • Maximum-entropy . Maximum-entropy is a typical entropy regularization, which maximizes the entropy of the predictive distribution. Note that AdaCos and Virtual-softmax regularize the predictive or output distribution of DNN to learn feature representation by reducing intra-class variations and enlarging inter-class margins. Comparison with output regularization methods. We measure the top-1 error rates of the proposed method (denoted by CS-KD) by comparing with Virtual-softmax, AdaCos, and Maximumentropy on various image classification tasks. Table 1 shows that CS-KD outperforms other baselines consistently. In particular, CS-KD improves the top-1 error rate of cross-entropy loss from 46.00% to 33.50% in the CUB-200-2011 dataset, while the top-1 error rates of other baselines are even worse than the cross-entropy loss (e.g., AdaCos in the CIFAR-100, Virtual-softmax in the MIT67, and Maximum-entropy in the TinyImageNet and the MIT67 under DenseNet). The imply that our method is more effective and stable than other baselines. Compatibility with other types of regularization methods. We investigate orthogonal usage with other types of regularization methods such as mixup and knowledge distillation (KD). Mixup utilizes convex combinations of input pairs and corresponding label pairs for training. We combine our method with mixup regularization by applying the class-wise regularization loss L cls to mixed inputs and mixed labels, instead of standard inputs and labels. Table 2 shows the effectiveness of our method combined with mixup regularization. Interestingly, this simple idea significantly improves the performances of fine-grained classification tasks. In particular, our method improves the top-1 error rate of mixup regularization from 37.09% to 31.95%, where the top-1 error rate of cross-entropy loss is 46.00% in the CUB-200-2011. KD regularizes predictive distributions of student network to learn the dark knowledge of a teacher network. We combine our method with KD to learn dark knowledge from the teacher and itself simultaneously. Table 3 shows that the top-1 error rate under using our method solely is close to that of KD, although ours do not use additional teacher networks. Besides, learning knowledge from a teacher network improves the top-1 error rate of our method from 39.32% to 35.36% in the CUB-200-2011 dataset. The show a wide applicability of our method, compatible to use with other regularization methods. One can expect that our method forces DNNs to produce meaningful predictions by reducing the intra-class variations. To verify this, we analyze feature embedding and various evaluation metrics, including the top-1, top-5 error, expected calibration error and R@1. In Figure 2, we visualize feature embedding of the penultimate layer from ResNet-18 trained with various regularization techniques by t-SNE in the CIFAR-100 dataset. One can note that intra-class variations are significantly decreased by our method (Figure 2f), while Virtualsoftmax (Figure 2b) and AdaCos (Figure 2c) only reduce the angular-margin. We also provide quantitative analysis on the feature embedding by measuring the R@1 values, which are related to intra-class variations. Note that the larger value of R@1 means the more reduced intra-class variations on the feature embedding . As shown in Table 4, R@1 values can be significantly improved when ResNet-18 is trained with our methods. In particular, R@1 of our method is 59.22% in the CUB-200-2011 dataset, while R@1 of Virtual-softmax and Adacos are 55.56% and 54.86%, respectively. Moreover, Table 4 shows the top-5 error rates of our method significantly outperform other regularization methods. Figure 3 and Table 4 show that our method enhances model calibration significantly, which also confirm that ours forces DNNs to produce more meaningful predictions. Table 4: Top-1 / 5 error, ECE, and Recall at 1 rates (%) of ResNet-18. The arrow on the right side of the evaluation metric indicates ascending or descending order of the value. We reported the mean and standard deviation over 3 runs with different random seed, and the best are indicated in bold. &; ) which show accuracy as a function of confidence, for ResNet-18 trianed on CIFAR-100 using (a) Cross-entropy, (b) Virtual-softmax, (c) AdaCos, and (d) Maximum-entropy. All methods are compared with our proposed method, CS-KD. Regularization techniques. Numerous techniques have been introduced to prevent overfitting of neural networks, including early stopping, weight decay, dropout , and batch normalization . Alternatively, regularization methods for the output distribution also have been explored: showed that label-smoothing, which is a mixture of the ground-truth and the uniform distribution, improves generalization of neural networks. proposed penalizing low entropy output distributions, which improves exploration in reinforcement learning and supervised learning. proposed a powerful data augmentation method called mixup, which works as a regularizer that can be utilized with smaller weight decay. We remark that our method enjoys orthogonal usage with the prior methods, i.e., our methods can be combined with prior methods to further improve the generalization performance. Knowledge distillation. Knowledge distillation is an effective learning method to transfer the knowledge from a powerful teacher model to a student. This pioneering work showed that one can use softmax with temperature scaling to match soft targets for transferring dark knowledge, which contains the information of non-target labels. There are numerous follow-up studies to distill knowledge in the aforementioned teacher-student framework. FitNets tried to learn features of a thin deep network using a shallow one with linear transform. introduced a transfer method that matches attention maps of the intermediate features, and tried to maximize the mutual information between intermediate layers of teacher and student for enhanced performance. proposed a loss function for matching Jacobian of the networks output instead of the feature itself. We remark that our method and knowledge distillation have a similar component, i.e., using a soft target distribution, but our method utilizes the soft target distribution from itself. We also remark that joint usage of our method and the prior knowledge distillation methods is effective. Margin-based softmax losses. There have been recent efforts toward boosting the recognition performances via enlarging inter-class margins and reducing intra-class variation. Several approaches utilized metric-based methods that measure similarities between features using Euclidean distances, such as triplet and contrastive loss . To make the model extract discriminative features, center loss and range loss were proposed to minimize distances between samples belong to the same class. COCO loss (b) and NormFace optimized cosine similarities, by utilizing reformulations of softmax loss and metric learning with feature normalization. applied ring loss for soft normalization which uses a convex norm constraint. More recently, angular-margin based losses were proposed for further improvement. Lsoftmax and A-softmax (a) combined angular margin constraints with softmax loss to encourage the model to generate more discriminative features. CosFace, AM-softmax and ArcFace introduced angular margins for a similar purpose, by reformulating softmax loss. Different from L-Softmax and A-Softmax, Virtual-softmax encourages a large margin among classes via injecting additional virtual negative class. In this paper, we discover a simple regularization method to enhance generalization performance of deep neural networks. We propose two regularization terms which penalizes the predictive distribution between different samples of the same label and augmented samples of the same source by minimizing the Kullback-Leibler divergence. We remark that our ideas regularize the dark knowledge (i.e., the knowledge on wrong predictions) itself and encourage the model to produce more meaningful predictions. Moreover, we demonstrate that our proposed method can be useful for the generalization and calibration of neural networks. We think that the proposed regularization techniques would enjoy a broader range of applications, e.g., deep reinforcement learning and detection of out-of-distribution samples .

Represent the short summary for finding the corresponding scientific article (long)

We propose a new regularization technique based on the knowledge distillation.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.