Question: Represent the article for finding a single-sentence TLDR

Question: A significant challenge for the practical application of reinforcement learning toreal world problems is the need to specify an oracle reward function that correctly defines a task. Inverse reinforcement learning (IRL) seeks to avoid this challenge by instead inferring a reward function from expert behavior. While appealing, it can be impractically expensive to collect datasets of demonstrations that cover the variation common in the real world (e.g. opening any type of door). Thus in practice, IRL must commonly be performed with only a limited set of demonstrations where it can be exceedingly difficult to unambiguously recover a reward function. In this work, we exploit the insight that demonstrations from other tasks can be used to constrain the set of possible reward functions by learning a "prior" that is specifically optimized for the ability to infer expressive reward functions from limited numbers of demonstrations. We demonstrate that our method can efficiently recover rewards from images for novel tasks and provide intuition as to how our approach is analogous to learning a prior. Reinforcement learning (RL) algorithms have the potential to automate a wide range of decisionmaking and control tasks across a variety of different domains, as demonstrated by successful recent applications ranging from robotic control BID19 BID23 to game playing BID28 BID44. A key assumption of the RL problem statement is the availability of a reward function that accurately describes the desired tasks. For many real world tasks, reward functions can be challenging to manually specify, while being crucial to good performance BID1. Most real world tasks are multifaceted and require reasoning over multiple factors in a task (e.g. an autonomous vehicle navigating a city at night), while simultaneously providing appropriate reward shaping to make the task feasible with tractable exploration BID32. These challenges are compounded by the inherent difficulty of specifying rewards for tasks with high-dimensional observation spaces such as images. Inverse reinforcement learning (IRL) is an approach that aims to address this problem by instead inferring the reward function from demonstrations of the task BID31. This has the appealing benefit of taking a data-driven approach to reward specification in place of hand engineering. In practice however, rewards functions are rarely learned as it can be prohibitively expensive to provide demonstrations that cover the variability common in real world tasks (e.g., collecting demonstrations of opening every type of door knob). In addition, while learning a complex function from high dimensional observations might make an expressive function approximator seem like a reasonable modelling assumption, in the "few-shot" domain it is notoriously difficult to unambiguously recover a good reward function with expressive function approximators. Prior solutions have thus instead relied on low-dimensional linear models with handcrafted features that effectively encode a strong prior on the relevant features of a task. This requires engineering a set of features by hand that work well for a specific problem. In this work, we propose an approach that instead explicitly learns expressive features that are robust even when learning with limited demonstrations. Our approach relies on the key observation that related tasks share common structure that we can leverage when learning new tasks. To illustrate, considering a robot navigating through a home. While the exact reward function we provide to the robot may differ depending on the task, there is a structure amid the space of useful behaviours, such as navigating to a series of landmarks, and there are certain behaviors we always want to encourage or discourage, such as avoiding obstacles or staying a reasonable distance from humans. This notion agrees with our understanding of why humans can easily infer the intents and goals (i.e., reward functions) of even abstract agents from just one or a few demonstrations BID4, as humans have access to strong priors about how other humans accomplish similar tasks accrued over many years. Similarly, our objective is to discover the common structure among different tasks, and encode the structure in a way that can be used to infer reward functions from a few demonstrations. Figure 1: A diagram of our meta-inverse RL approach. Our approach attempts to remedy over-fitting in few-shot IRL by learning a "prior" that constraints the set of possible reward functions to lie within a few steps of gradient descent. Standard IRL attempts to recover the reward function directly from the available demonstrations. The shortcoming of this approach is that there is little reason to expect generalization as it is analogous to training a density model with only a few examples. More specifically, in this work we assume access to a set of tasks, along with demonstrations of the desired behaviors for those tasks, which we refer to as the meta-training set. From these tasks, we then learn a reward function parameterization that enables effective few-shot learning when used to initialize IRL in a novel task. Our method is summarized in Fig. 1. Our key contribution is an algorithm that enables efficient learning of new reward functions by using meta-training to build a rich "prior" for goal inference. Using our proposed approach, we show that we can learn deep neural network reward functions from raw pixel observations with substantially better data efficiency than existing methods and standard baselines. Inverse reinforcement learning (IRL) BID31 is the problem of inferring an expert's reward function directly from demonstrations. Prior methods for performing IRL range from margin based approaches BID0 BID34 to probabilistic approaches BID33 BID55. Although it is possible to extend our approach to any other IRL method, in this work we base on work on the maximum entropy (MaxEnt) framework BID55. In addition to allowing for sub-optimality in the expert demonstrations, MaxEnt-IRL can be re-framed as a maximum likelihood estimation problem. 4).In part to combat the under-specified nature of IRL, prior work has often used low-dimensional linear parameterizations with handcrafted features BID0 BID55. In order to learn from high dimensional input, BID52 proposed applying fully convolutional networks BID41 to the MaxEnt IRL framework BID55 for several navigation tasks BID51 BID4. Other methods that have incorporated neural network rewards include guided cost learning (GCL) BID11, which uses importance sampling and regularization for scalability to high-dimensional spaces, and adversarial IRL BID13. Several other methods have also proposed imitation learning approaches based on adversarial frameworks that resemble IRL, but do not aim to directly recover a reward function BID16 BID15 BID21. In this work, instead of improving the ability to learn reward functions on a single task, we focus on the problem of effectively learning to use prior demonstration data from other IRL tasks, allowing us to learn new tasks from a limited number demonstrations even with expressive non-linear reward functions. Prior work has explored the problem of multi-task IRL, where the demonstrated behavior is assumed to have originated from multiple experts achieving different goals. Some of these approaches include those that aim to incorporate a shared prior over reward functions through extending the Bayesian IRL BID33 framework to the multi-task setting BID7 BID6. Other approaches have clustered demonstrations while simultaneously inferring reward functions for each cluster BID3 or introduced regularization between rewards to a common "shared reward" BID25. Our work is similar in that we also seek to encode prior information common to the tasks. However, a critical difference is that our method specifically aims to distill the meta-training tasks into a prior that can then be used to learn rewards for new tasks efficiently. The goal therefore is not to acquire good reward functions that explain the meta-training tasks, but rather to use them to learn efficiently on new tasks. Our approach builds on work on the broader problem of meta-learning BID40 BID5 BID30 BID48 and generative modelling BID37 BID36 BID29. Prior work has proposed a variety of solutions for learning to learn including memory based methods BID8 BID38 BID27, methods that learn an optimizer and/or initialization BID2 BID35 BID11 BID24, and methods that compare new datapoints in a learned metric space BID20 BID49 BID42 BID46. Our work is motivated by the goal of broadening the applicability of IRL, but in principle it is possible to adapt many of these meta-learning approaches for our problem statement. We leave it to future work to do a comprehensive investigation of different meta-learning approaches which could broaden the applicability of IRL. In this section, we introduce our notation and describe the IRL and meta-learning problems. The standard Markov decision process (MDP) is defined by the tuple (S, A, p s, r, γ) where S and A denote the set of possible states and actions respectively, r: S × A → R is the reward function, γ ∈ is the discount factor and p s: S × S × A → denotes the transition distribution over the next state s t+1, given the current state s t and current action a t. Typically, the goal of "forward" RL is to maximize the expected discounted return R(τ) = T t=1 γ t−1 r(s t, a t).In IRL, we instead assume that the reward function is unknown but that we instead have access to a set of expert demonstrations D = {τ 1, . . ., τ K}, where DISPLAYFORM0 The goal of IRL is to recover the unknown reward function r from the set of demonstrations. We build on the maximum entropy (MaxEnt) IRL framework by BID55, which models the probability of the trajectories as being distributed proportional to their exponentiated return DISPLAYFORM1 where Z is the partition function, Z = τ exp(R(τ))dτ. This distribution can be shown to be induced by the optimal policy in entropy regularized forward RL problem: DISPLAYFORM2 This formulation allows us to pose the reward learning problem as a maximum likelihood estimation (MLE) problem in an energy-based model r φ: DISPLAYFORM3 Learning in general energy-based models of this form is common in many applications such as structured prediction. However, in contrast to applications where learning can be supervised by millions of labels (e.g. semantic segmentation), the learning problem in Eq. 3 must typically be performed with a relatively small number of example demonstrations. In this work, we seek to address this issue in IRL by providing a way to integrate information from prior tasks to constrain the optimization in Eq. 3 in the regime of limited demonstrations. The goal of meta-learning algorithms is to optimize for the ability to learn efficiently on new tasks. Rather than attempt to generalize to new datapoints, meta-learning can be understood as attempting to generalize to new tasks. It is assumed in the meta-learning setting that there are two sets of tasks that we refer to as the meta-training set {T i ; i = 1..N} and meta-test set {T j ; j = 1..M}, which are both drawn from a distribution p(T). During meta-training time, the meta-learner attempts to learn the structure of the tasks in the meta-training set, such that when it is presented with a new test task, it can leverage this structure to learn efficiently from a limited number of examples. To illustrate this distinction, consider the case of few-shot learning setting. Let f θ denote the learner, and let a task be defined by learning from K training examples DISPLAYFORM0 One approach to meta-learning is to directly parameterize the meta-learner with an expressive model such as a recurrent or recursive neural network BID8 BID27 conditioned on the task training data and the inputs for the test task: DISPLAYFORM1. Such a model is optimized using log-likelihood across all tasks. In this approach to meta-learning, since neural networks are known to be universal function approximators BID43, any desired structure between tasks can be implicitly encoded. Rather than learn a single black-box function, another approach to meta-learning is to learn components of the learning procedure such as the initialization BID11 or the optimization algorithm BID35 BID2. In this work we extend the approach of model agnostic meta-learning (MAML) introduced by BID11, which learns an initialization that is adapted by gradient descent. Concretely, in the supervised learning case, given a loss function L(θ, X T, Y T) (e.g. cross-entropy), MAML performs the following optimization DISPLAYFORM2 where the optimization is over an initial set of parameters θ and the loss on the held out tasks X test T becomes the signal for learning the initial parameters for gradient descent on X tr T. This optimization is analogous to adding a constraint in a multi-task setting, which we show in later sections is analogous in our setting to learning a prior over reward functions. Our goal in meta-IRL is to learn how to learn reward functions across many tasks such that the model can infer the reward function for a new task using only one or a few expert demonstrations. Intuitively, we can view this problem as aiming to learn a prior over the intentions of human demonstrators, such that when given just one or a few demonstrations of a new task, we can combine the learned prior with the new data to effectively determine the human's reward function. Such a prior is helpful in inverse reinforcement learning settings, since the space of relevant reward functions is much smaller than the space of all possible rewards definable on the raw observations. During meta-training, we have a set of tasks {T i ; i = 1..N}. Each task T i has a set of demonstrations D T = {τ 1, . . ., τ K} from an expert policy which we partition into disjoint D tr T and D test T sets. The demonstrations for each meta-training task are assumed to be produced by the expert according to the maximum entropy model in Section 3.1. During meta-training, these tasks will be used to encodes common structure so that our model can quickly acquire rewards for new tasks from just a few demonstrations. After meta-training, our method is presented with a new task. During this meta-test phase, the algorithm must infer the parameters of the reward function r φ (s t, a t) for the new task from a few demonstrations. As is standard in meta-learning, we assume that the test task is from the same distribution of tasks seen during meta-training, a distribution that we denote as p(T). In order to meta-learn a reward function that can act as a prior for new tasks and new environments, we first formalize the notion of a good reward by defining a loss L T (θ) on the reward function r θ for a particular task T. We use the MaxEnt IRL loss L IRL discussed in Section 3, which, for a given D T, leads to the following gradient BID55: DISPLAYFORM0 DISPLAYFORM1 via chain rule 16:Compute updated parameters DISPLAYFORM2 Sample demos DISPLAYFORM3 via chain rule Compute meta-gradient 20: DISPLAYFORM4 Update initial parameters where µ τ are the state visitations under the optimal maximum entropy policy under r θ, and µ D are the mean state visitations under the demonstrated trajectories. If our end goal were to achieve a single reward function that works as well as possible across all tasks in {T i ; i = 1..N}, then we could simply follow the mean gradient across all tasks. However, our objective is different: instead of optimizing performance on the meta-training tasks, we aim to learn a reward function that can be quickly and efficiently adapted to new tasks at meta-test time. In doing so, we aim to encode prior information over the task distribution in this learned reward prior. We propose to implement such a learning algorithm by finding the parameters θ, such that starting from θ and taking a small number of gradient steps on a few demonstrations from given task leads to a reward function for which a set of test demonstrations have high likelihood, with respect to the MaxEnt IRL model. In particular, we would like to find a θ such that the parameters DISPLAYFORM5 lead to a reward function r φ T for task T, such that the IRL loss (corresponding to negative loglikelihood) for a disjoint set of test demonstrations, given by L DISPLAYFORM6 IRL, is minimized. The corresponding optimization problem for θ can therefore be written as follows: DISPLAYFORM7 Our method acquires this prior θ over rewards in the task distribution p(T) by optimizing this loss. This amounts to an extension of the MAML algorithm in Section 3.2 to the inverse reinforcement learning setting. This extension is quite challenging, because computing the MaxEnt IRL gradient requires repeatedly solving for the current maximum entropy policy and visitation frequencies, and the MAML objective requires computing derivatives through this gradient step. Next, we describe in detail how this is done. An overview of our method is also outlined in Alg. 1.Meta-training. The computation of the meta-gradient for the objective in Eq. 7 can be conceptually separated into two parts. First, we perform the update in Eq. 6 by computing the expected state visitations µ, which is the expected number of times an agent will visit each state. We denote this overall procedure as STATE-VISITATIONS-POLICY, and follow BID55 by first computing the maximum entropy optimal policy in Eq. 2 under the current r θ, and then approximating µ using dynamic programming. Next, we compute the state visitation distribution of the expert using a procedure which we denote as STATE-VISITATIONS-TRAJ. This can be done either empirically, by averaging the state visitation of the experts demonstrations, or by using STATE-VISITATIONS-POLICY if the true reward is available at meta-training time. This allows us to recover the IRL gradient according to Eq. 5, which we can then apply to compute φ T according to Eq. 6.Second, we need to differentiate through this update to compute the gradient of the meta-loss in Eq. 7. Note that the meta-loss itself is the IRL loss evaluated with a different set of test demonstrations. We follow the same procedure as above to evaluate the gradient of L IRL with respect to the post-update parameters φ T, and then apply the chain rule to compute the meta-gradient: DISPLAYFORM0 where on the second line we differentiate through the MaxEnt-IRL update. The derivation of this expression is somewhat more involved and provided in the supplementary Appendix D.Meta-testing. Once we have acquired the meta-trained parameters θ that encode a prior over p(T), we can leverage this prior to enable fast, few-shot IRL of novel tasks in {T j ; j = 1..M}. For each task, we first compute the state visitations from the available set of demonstrations for that task. Next, we use these state visitations to compute the gradient, which is the same as the inner loss gradient computation of the meta-training loop in Alg. 1. We apply this gradient to adapt the parameters θ to the new task. Even if the model was trained with only one to three inner gradient steps, we found in practice that it was beneficial to take substantially more gradient steps during meta-testing; performance continued to improve with up to 20 steps. The objective in Eq. 6 optimizes for parameters that enable that reward function to adapt and generalize efficiently on a wide range of tasks. Intuitively, constraining the space of reward functions to lie within a few steps of gradient descent can be interpreted as expressing a "locality" prior over reward function parameters. This intuition can be made more concrete with the following analysis. DISPLAYFORM0 Figure 2: Our approach can be understood as approximately learning a distribution over the demonstrations τ, in the factor graph p(τ) = 1 Z T t=1 Φr(φ T, st, at)Φdyn(st+1, st, at) (above) where we learn a prior over φT, which during meta-test is used for MAP inference over new expert demonstrations. By viewing IRL as maximum likelihood estimation, we can take the perspective of BID14 who showed that for a linear model, fast adaptation via a few steps of gradient descent in MAML is performing MAP inference over φ, under a Gaussian prior with the mean θ and a covariance that depends on the step size, number of steps and curvature of the loss. This is based on the connection between early stopping and regularization previously discussed in , which we refer the readers to for a more detailed discussion. The interpretation of MAML as imposing a Gaussian prior on the parameters is exact in the case of a likelihood that is quadratic in the parameters (such as the log-likelihood of a Gaussian in terms of its mean). For any non-quadratic likelihood, this is an approximation in a local neighborhood around θ (i.e. up to convex quadratic approximation). In the case of very complex parameterizations, such as deep function approximators, this is a coarse approximation and unlikely to be the mode of a posterior. However, we can still frame the effect of early stopping and initialization as serving as a prior in a similar way as prior work BID45 BID10 BID14. More importantly, this interpretation hints at future extensions to our approach that could benefit from employing more fully Bayesian approaches to reward and goal inference. Our evaluation seeks to answer two questions. First, we aim to test our core hypothesis that leveraging prior task experience enables reward learning for new tasks with just a few demonstrations. Second, we compare our method with alternative algorithms that make use of multi-task experience. We test our core hypothesis by comparing learning performance on a new task starting from the learned initialization produced by MandRIL, compared to starting from scratch with a random initialization. This comparison is meant to evaluate whether prior experience on other tasks can in fact make inverse RL more efficient. To our knowledge, there is no prior work that addresses the meta-inverse reinforcement learning problem introduced in this paper. Thus, to provide a point of comparison and calibrate the difficulty of the tasks, we adapt two alternative black-box meta-learning methods to the IRL setting. The comparisons to both of the black-box methods described below evaluate the importance of incorporating the IRL gradient into the meta-learning process, rather than learning the adaptation process entirely from scratch.• Demo conditional model: Our method implicitly conditions on the demonstrations through the gradient update. In principle, a conditional deep model with sufficient capacity could implicitly implement a similar learning rule. Thus, we consider a conditional model (often referred to as a "contextual model" BID12), which receives the demonstration as an additional input.• Recurrent meta-learner: We additionally compare to an RNN-based meta-learner BID38 BID9. Specifically, we implement a conditional model by feeding both images and sequences of states visited by the demonstrations to an LSTM.Our approach can be understood as explicitly optimizing for an effective parameter initialization for the IRL problem. In order to test the benefits of our proposed formulation, we also compare with finetuning an initialization obtained with the same set of prior tasks, but with supervised pretraining as follows:• Supervised pre-training: We compare to following the average gradient during meta-training, averaged across tasks, and fine-tuning at meta-test time (as discussed in Section 4). This comparison evaluates the benefits of optimizing explicitly for weights that perform well under fine-tuning. We compare to pre-training on a single task as well as all the meta-training tasks. Figure 3: An example task: When learning a task, the agent has access to the image (left) and demonstrations (red arrows). To evaluate the agent's learning (right), the agent is tested for its ability to recover the reward for the task when the objects have been rearranged. The reward structure we wish to capture can be illustrated by considering the initial state in blue. An policy acting optimally under a correctly inferred reward should interpret the other objects as obstacles, and prefer a path on the dirt in between them. Next, we describe our environment and evaluation. Spriteworld navigation domain. Since most prior IRL works (and multi-task IRL works) have studied settings where linear reward function approximators suffice (i.e. low-dimensional state spaces made up from hand-designed features), we design an experiment that is significantly more challenging-that requires learning rewards on raw pixels-while still exhibiting multi-task structure needed to test our core hypothesis. We consider a navigation problem where we aim to learn a convolutional neural network that directly maps image pixels to rewards. To do so, we introduce "SpriteWorld," which is a synthetically generated task, some examples of which are shown in Fig. 3. The task visuals are inspired by Starcraft and work applying learning algorithms to perform micromanagement (e.g. BID47), although we do not use the game engine. Tasks involve navigating to goal objects while exhibiting preference over terrain types (e.g. the agent prefers to traverse dirt tiles over traversing grass tiles). At meta-test time, we provide one or a few demonstrations in a single training environment and evaluate the reward learned using these demonstrations in a new, test environment that contains the same objects as the training environment, but arranged differently. Evaluating in a new test environment is critical to measure that the reward learned the correct visual cues, rather than simply memorizing the demonstration trajectory. held-out tasks test performance (top), and test performance on held-out tasks with novel sprites (bottom). All methods are capable are overfitting to the training environment (See Appendix A). However, in both test settings, MandRIL achieves comparable performance to the training environment, while the other methods overfit until they receive at least 10 demonstrations. The recurrent meta-learner has a value difference larger than 60 in both test settings. Shaded regions show 95% confidence intervals. The underlying MDP structure of SpriteWorld is a grid, where the states are each of the grid cells, and the actions enable the agent to move to any one of its 8-connected neighbors. We generate unique tasks from this domain as follows. First, we randomly choose a set of 3 sprites from a total of 100 sprites from the original game (creating a total of 161,700 unique tasks). We randomly place these three sprites within a randomly generated terrain tiling; we designate one of the sprites to be the goal of the navigation task, i.e. the object to which the agent must navigate. The other two objects are treated as obstacles for which the agent incurs a large negative reward for not avoiding. In each task, we optimize our model on a meta-training set and evaluate the ability of the reward function to the generalize to a rearrangement of the same objects. For example, suppose the goal of the task is to navigate to sprite A, while avoiding sprites B and C. Then, to generate an environment for evaluation, we resample the positions of the sprites, while the underlying task remains the same (i.e., navigate to A). This requires the model to make use of the visual patterns in the scene to generalize effectively, rather than simply memorizing positions of sprites. We evaluate on novel combinations of units seen in meta-training, as well as the ability to generalize to new unseen units. We provide further details on our setup in Appendices A and B.We measure performance using the expected value difference, which measures the sub-optimality of a policy learned under the learned reward; this is a standard performance metric used in prior IRL work BID22 BID52. The metric is computed by taking the difference between the value of the optimal policy under the learned reward and the value of the optimal policy under the true reward. Evaluation protocol. We evaluate on held-out tasks that were unseen during meta-training. We consider two settings: tasks involving new combinations and placements of sprites, with individual sprites that were present during meta-training, and tasks with combinations of entirely new sprites which we refer to as "out of domain objects." For each task, we generate one environment (set of sprite positions) along with one or a few demonstrations for adapting the reward, and generate a second environment (with new sprite positions) where we evaluate the adapted reward. In these meta-test time evaluations, we refer to the performance on the first environment as "training performance" (not to be confused with meta-training) and to performance on the second as "testing performance". We evaluate on 32 meta-test randomly generated tasks. Results. The are shown in FIG0, which illustrate test performance with in-distribution and out-of-distribution sprites. Our approach, MandRIL, achieves consistently better performance in both settings. Most significantly, our approach performs well even with single-digit numbers of demonstrations. By comparison, alternative meta-learning methods generally overfit considerably, attaining good training performance (see Appendix. A for curves) but poor test performance. Learning the reward function from scratch is in fact the most competitive baseline -as the number of demonstrations increases, simply training the fully convolutional reward function from scratch on the new task is the only method that matches the performance of MandRIL when provided 20 or more demonstrations. However, with only a few demonstrations, MandRIL has substantially lower value difference. It is worth noting the performance of MandRIL on the out of distribution test setting FIG0: although the evaluation is on new sprites, MandRIL is still able to adapt via gradient descent and exceed the performance of learning from scratch and all other methods. Figure 5: Meta-test performance comparison with varying number of demonstrations using pre-trained weights fine-tuned over either a single task or all tasks. We find that pre-training on the full set of tasks hurts adaptation, while pre-training on a single task does not improve performance (comparable to random initialization). Our approach ManDRIL outperforms all these methods, which shows that explicitly optimizing for initial weights for fine-tuning robustly improves performance. Shaded regions show 95% confidence intervals. Finally, as it is common practice to fine-tune representations obtained from a supervised pre-training phase, we perform this comparison in Figure 5. We compare against an approach that follows the mean gradient across the tasks at meta-training time and is fine-tuned at meta-test time which we find is less effective than learning from a random initialization. We conclude that fine tuning reward functions learned in this manner is not an effective way of using prior task information. When using a single task for pre-training, we find that it performs comparable to random initialization. In contrast, we find that our approach, which explicitly optimizes for initial weights for fine-tuning, robustly improves performance. In this work, we present an approach that enables few-shot learning for reward functions of new tasks. We achieve this through a novel formulation of inverse reinforcement learning that learns to encode common structure across tasks. Using our meta-IRL approach, we show that we can leverage data from previous tasks to effectively learn deep neural network reward functions from raw pixel observations for new tasks, from only a handful of demonstrations. Our work paves the way for futures work that considers environments with unknown dynamics, or more fully probabilistic approaches to reward and goal inference. The input to our reward function for all experiments is a resized 80 × 80 RGB image, with an output space of 20×20 in the underlying MDP state space s. In order to compute the optimal policy, we use Q-iteration. In our experiments, we parameterize the reward function for all our reward functions starting from the same base learner. The first layer is a 8 × 8 convolution with a stride of 2, 256 filters and symmetric padding of 4. The second layer is a 4 × 4 with a stride of 2, 128 filters and symmetric of 1. The third and fourth layer are 3 × 3 convolutions with a stride of 1, 64 filters and symmetric padding of 1. The final layer is a 1 × 1 convolution. Our LSTM BID17 implementation is based on the variant used by Zaremba et al. BID54. The input to the LSTM at each time step is the location of the agent in the, we separately embed the (x, y)-coordinates. This is then used to predict an additional channel in to the base CNN architecture described above. We also experimented with conditioning the initial hidden state on image features from a separate CNN, but found that this did not improve performance. In our demo conditional model, we preserve the spatial information of the demonstrations by feeding in the state visitation map as a image-grid, upsampled with bi-linear interpolation, as an additional channel to the image. In our setup, both the demo-conditional models share the same convolutional architecture, but differ only in how they encode condition on the demonstrations. For all our methods, we optimzed our model with Adam . We turned over the learning rate α, the inner learning rate β of our approach and 2 weight decay on the initial parameters. In our LSTM learner, we experimented with different embedding sizes, as well as the dimensionality of the LSTM although we found that these hyperparameters did not impact performance. A negative we found was that bias transformation BID12 did not help in our experimental setting. The sprites in our environment are extracted directly from the StarCraft files. We used in total 100 random units for meta-training. Evaluation on new objects was performed with 5 randomly selected sprites. For computational efficiency, we create a meta-training set of 1000 tasks and cache the optimal policy and state visitations under the true cost. Our evaluation is over 32 tasks. Our set of sprites was divided into two categories: buildings and characters. Each characters had multiple poses (taken from different frames of animation, such as walking/running/flying), whereas buildings only had a single pose. During meta-training the units were randomly placed, but to avoid the possibility that the agent would not need to actively avoid obstacles, the units were placed away from the boundary of the image in both the meta-validation and meta-test set. The terrain in each environment was randomly generated using a set of tiles, each belonging to a specific category (e.g. grass, dirt, water). For each tile, we also specified a set of possible tiles for each of the 4-neighbors. Using these constraints on the neighbors, we generated random environment terrains using a graph traversal algorithm, where successor tiles were sampled randomly from this set of possible tiles. This process ed in randomly generated, seamless environments. The names of the units used in our experiments are as follows (names are from the original game files):The list of buildings used is: academy, assim, barrack, beacon, cerebrat, chemlab, chrysal, cocoon, comsat, control, depot, drydock, egg, extract, factory, fcolony, forge, gateway, genelab, geyser, hatchery, hive, infest, lair, larva, mutapit, nest, nexus, nukesilo, nydustpit, overlord, physics, probe, pylon, prism, pillbox, queen, rcluster, refinery, research, robotic, sbattery, scolony, spire, starbase, stargate, starport, temple, warm, weaponpl, wessel. The list of characters used is: acritter, arbiter, archives, archon, avenger, battlecr, brood, bugguy, carrier, civilian, defiler, dragoon, drone, dropship, firebat, gencore, ghost, guardian, hydra, intercep, jcritter, lurker, marine, missile, mutacham, mutalid, sapper, scout, scv, shuttle, snakey, spider, stank, tank, templar, trilob, ucereb, uikerr, ultra, vulture, witness, zealot, zergling. C META-TEST TRAINING PERFORMANCE Figure 6: Meta-test "training" performance with varying numbers of demonstrations (lower is better). This is the performance on the environment for which demonstrations are provided for adaptation. As the number of demonstrations increase, all methods are able to perform well in terms of training performance as they can simply overfit to the training environment without acquiring the right visual cues that allow them to generalize. However, we find comes at the cost comes of considerable overfitting as we discuss in Section. 5. We define the quality of reward function r θ parameterized by θ ∈ R k on task T with the MaxEnt IRL loss, L T IRL (θ), described in Section 4. The corresponding gradient is DISPLAYFORM0 where ∂ r θ /∂ θ is the k ×|S||A|-dimensional Jacobian matrix of the reward function r θ with respect to the parameters θ. Here, µ τ ∈ R |S||A| is the vector of state visitations under the trajectory τ (i.e. the vector whose elements are 1 if the corresponding state-action pair has been visited by the trajectory τ, and 0 otherwise), and µ D T = 1 |D T | τ ∈D T µ τ is the mean state visitations over all demonstrated trajectories in D T. Let φ T ∈ R k be the updated parameters after a single gradient step. Then DISPLAYFORM1 Let L test T be the MaxEnt IRL loss, where the expectation over trajectories is computed with respect to a test set that is disjoint from the set of demonstrations used to compute L test T (θ) in Eq. 10. We seek to minimize DISPLAYFORM2 over the parameters θ. To do so, we first compute the gradient of Eq. 11, which we derive here. Applying the chain rule DISPLAYFORM3 where in the last line we substitute in the gradient of the MaxEnt IRL loss in Eq. 9. In Eq. 12, we use the following notation:• ∂ φ T /∂ θ denotes the k × k-dimensional vector of partial derivatives ∂ φ T,i /∂ θ j,• ∂ r φ T /∂ φ T denotes the k × |S||A|-dimensional matrix of partial derivatives ∂ r φ T,i /∂ φ T,j,• and, ∂ L test T /∂ r φ T denotes the k-dimensional gradient vector of L test T with respect to r φ T.We will now focus on the term inside of the parentheses in Eq. 12, which is a k × k-dimensional matrix of partial derivatives. DISPLAYFORM4 where between the first and second lines, we apply the chain rule to expand the second term. In this expression, we make use of the following notation:• ∂ 2 r θ /∂ θ 2 denotes the k × |S||A|-dimensional matrix of second-order partial derivatives of the form ∂ 2 r θ,i /∂ θ 2 j, DISPLAYFORM5 • ∂ r θ /∂ θ denotes the k × |S||A|-dimensional matrix of partial derivatives of the form ∂ r θ,i /∂ θ j,• and, DISPLAYFORM6 ] is the |S||A|-dimensional Jacobian matrix of E τ [µ τ] with respect to the reward function r θ (we will examine in more detail exactly what this is below).When substituted back into Eq. 12, the ing gradient is equivalent to that in Eq. 8 in Section 4. In order to compute this gradient, however, we must take the gradient of the expectation E τ [µ τ] with respect to the reward function r θ. This can be done by expanding the expectation as follows DISPLAYFORM7 τ exp(µ τ r θ)(µ τ µ τ) − exp(µ τ r θ) (τ exp(µ τ r θ)) 2 τ (µ τ µ τ) exp(µ τ r θ) DISPLAYFORM8

Represent this text to find a much much longer article about it with say hundreds-thousands of words

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.

Represent the TLDR for finding the corresponding scientific article.

The overall goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels, through the use of f-divergences.


Question: Represent the article for finding a single-sentence TLDR:

Question: Recent work has focused on combining kernel methods and deep learning. With this in mind, we introduce Deepström networks -- a new architecture of neural networks which we use to replace top dense layers of standard convolutional architectures with an approximation of a kernel function by relying on the Nyström approximation. Our approach is easy highly flexible. It is compatible with any kernel function and it allows exploiting multiple kernels. We show that Deepström networks reach state-of-the-art performance on standard datasets like SVHN and CIFAR100. One benefit of the method lies in its limited number of learnable parameters which make it particularly suited for small training set sizes, e.g. from 5 to 20 samples per class. Finally we illustrate two ways of using multiple kernels, including a multiple Deepström setting, that exploits a kernel on each feature map output by the convolutional part of the model. Kernel machines and deep learning have mostly been investigated separately. Both have strengths and weaknesses and appear as complementary family of methods with respect to the settings where they are most relevant. Deep learning methods may learn from scratch relevant features from data and may work with huge quantities of data. Yet they actually require large amount of data to fully exploit their potential and may not perform well with limited training datasets. Moreover deep networks are complex and difficult to design and require lots of computing and memory resources both for training and for inference. Kernel machines are powerful tools for learning nonlinear relations in data and are well suited for problems with limited training sets. Their power comes from their ability to extend linear methods to nonlinear ones with theoretical guarantees. However, they do not scale well to the size of the training datasets and do not learn features from the data. They usually require a prior choice of a relevant kernel amongst the well known ones, or even require defining an appropriate kernel for the data at hand. Although most research in the field of deep learning seems to have evolved as a "parallel learning strategy" to the field of kernel methods, there are a number of studies at the interface of the two domains which investigated how some concepts can be transferred from one field to another. Mainly, there are two types of approaches that have been investigated to mix deep learning and kernels. Few works explored the design of deep kernels that would allow working with a hierarchy of representations as the one that has been popularized with deep learning (2; 14; 7; 6; 20; 23). Other studies focused on various ways to plug kernels into deep networks (13; 24; 5; 12; 25). This paper follows this latter line of research, it focuses on convolutional networks. Specifically, we propose Deepström networks which are built by replacing dense layers of a convolutional neural network by an adaptive approximation of a kernel function. Our work is inspired from Deep Fried Convnets which brings together convolutional neural networks and kernels via Fastfood, a kernel approximation technique based on random feature maps. We revisit this concept in the context of Nyström kernel approximation BID21. One key advantage of our method is its flexibility that enables the use of any kernel function. Indeed, since the Nyström approximation uses an explicit feature map from the data kernel matrix, it is not restricted to a specific kernel function and not limited only to RBF kernels, as in Fastfood approximation. This is particularly useful when one wants to use or learn multiple different kernels instead of a single kernel function, as we demonstrate here. In particular we investigate two different ways of using multiple kernels, one is a straightforward extension to using multiple kernels while the second is a multiple Deepström variant that exploits a Nyström kernel approximation for each of the feature map output by the convolutional part of the neural network. Furthermore the specific nature of our architecture makes it use only a limited number of parameters, which favours learning with small training sets as we demonstrate on targeted experiments. Our experiments on four datasets (MNIST, SVHN, CIFAR10 and CIFAR100) highlight three important features of our method. First our approach compares well to standard approaches in standard settings (using ful training sets) while requiring a reduced number of parameters compared to full deep networks and of the same order of magnitude as Deep Fried Convnets. This specific feature of our proposal makes it suitable for dealing with limited training set sizes as we show by considering experiments with tens or even fewer training samples per class. Finally the method may exploit multiple kernels, providing a new tool with which to approach the problem of multiple kernel learning (MKL), and enabling taking into account the rich information in multiple feature maps of convolution networks through multiple Deepström layers. The rest of the paper is organized as follows. We provide on kernel approximation via the Nyström and the random Fourier features methods and describe the Deep Fried Convnet architecture in Section 2. The detailed configuration of the proposed Deepström network is described in Section 3. We also show in Section 3 how Deepström networks can be used with multiple kernels. Section 4 reports experimental on MNIST, SVHN, CIFAR10 and CIFAR100 datasets to first provide a deeper understanding of the behaviour of our method with respect to the choice of the kernels and the combination of these, and second to compare it to state of the art baselines on classification tasks with respect to accuracy and to complexity issues, in particular in the small training set size setting. Kernel approximation methods have been proposed to make kernel methods scalable. Two popular methods are Nyström approximation BID21 and random features approximation BID15. The former approximates the kernel matrix by an efficient low-rank decomposition, while the latter is based on mapping input features into a low-dimensional feature space where dot products between features approximate well the kernel function. Nyström approximation BID21 It computes a low-rank approximation of the kernel matrix by randomly subsampling a subset of instances. Let consider a training set of n training samples, DISPLAYFORM0 which is selected from the training set. Assuming the subset includes the first samples, or rearranging the training samples this way, K may be rewritten as: DISPLAYFORM1 where K 11 is the Gram matrix on subset L. Nyström approximation is obtained as follows DISPLAYFORM2 From this approximation the Nyström nonlinear representation of a single example x is given by DISPLAYFORM3 where DISPLAYFORM4 Random features approximation It computes a low-dimensional feature mapφ of dimension q such that φ (·),φ(·) =k(·, ·) k(·, ·). Two instances of this method are Random Kitchen Sinks (RKS) and Fastfood (17; 9). RKS approximates a Radial Basis Function (RBF) kernel using a random feature map defined as DISPLAYFORM5 where z ∈ R p, Q ∈ R q×p and Q i,j are drawn randomly. If Q i,j are drawn according to a Gaussian distribution then the method is shown to approximate the Gaussian kernel, i.e. φ rks (DISPLAYFORM6 where σ is the hyper-parameter of the kernel. Note that σ is related to the parameters of the Gaussian distribution that generate the random features. The Fastfood method is a variant of RKS with reduced computational cost for the Gaussian kernel. It is based on approximating the matrix Q in Eq. 2, when q = p, by a product of diagonal and hadamard matrices according to DISPLAYFORM7 where S,G and B are diagonal matrices of size p × p, Π ∈ {0, 1} p×p is a random permutation matrix, H is a Hadamard matrix which does not requite to be stored, and σ is an hyperparameter. Matrix V may be used in place of Q in Eq. 2 to define the Fastfood nonlinear representation map DISPLAYFORM8 Note that this definition requires p to be a power of 2 to take advantage of the recursive structure of the Hadamard matrix. Note also that to reach a representation dimension q > p one may compute multiple V and concatenate the corresponding φ f f.Deep Fried Convnets Our attention in this work is especially focused on combining kernel approximation with deep learning architecture. Deep Fried Convnets is a deep learning architecture that replaces dense layers of a convolutional neural architecture by a Fastfood approximation of a kernel. This allows to take advantage of the low complexity cost in terms of computation and memory of Fastfood to reduce significantly the computation cost and the number of parameters of the fully-connected layers of the deep convolutional neural network. More formally, let conv(x) be the representation of the data sample x learned by a convolutional neural network. It may include a number of convolution blocks, each including convolution and pooling layers, batch normalization and nonlinear activation. In Deep Fried Convnets, an input x ∈ R d is mapped to the representation spaces conv(x) ∈ R p and then the Fastfood feature map φ f f is applied to the convolutional representation conv(x) instead of the fully-connected layers. The feature representation of x with Deep Fried Convnets is then (φ f f •conv)(x) ∈ R q. It is of note that this method is dedicated to RBF kernels. In, wo architectures have been proposed. The first one relies on the Fastfood kernel approximation method as described above. The second one is a variant of Fastfood called Adaptive-Fastfood. It involves learning the weights of matrices S, G and B through gradient descent rather than setting them randomly, while matrices Π and H are kept unchanged. In the next section we introduce Deepström Networks as an alternative to Deep Fried Convnets. They are based on Nyström approximation and are not limited to RBF kernels. They also allow the use of multiple different kernels and find an appropriate kernel function.3 Deepström NETWORKS In this section, we describe our new Deepström model which combines the desirable characteristics of Nyström approximation and convolutional neural networks. First, we start by revisiting the concept of Nyström kernel approximation from a feature map perspective. Nyström approximation from an empirical kernel map perspective The empirical kernel map is an explicit n-dimensional feature map that is obtained by applying the kernel function on the training data x i. It is defined as DISPLAYFORM9 An interesting feature of the empirical kernel map is that if we consider the inner product in R n ·, · M = ·, M · with a positive semi-definite (psd) matrix M, we can recover the kernel matrix K using the empirical kernel map by setting M equals to the inverse of the kernel matrix. In other words, DISPLAYFORM10 = K. Since K is a psd matrix, one can consider the feature φ emp: DISPLAYFORM11 as an explicit feature map that allows to reconstruct the DISPLAYFORM12 The Deepström network architecture involves a usual convolutional part, conv, including multiple convolutional blocks, and a Deepström layer which is then fed to (eventually multiple) standard dense layers up to the classification layer. The Deepström layer computes the kernel between the output of the conv block for a given input and the corrsponding representations of the trains samples in the subsample L.kernel matrix. This feature map is of dimension n and then is not interesting when the number of example is large. The feature map of the Nyström approximation is given by DISPLAYFORM13 T with x i ∈ L. From an empirical kernel map point of view,φ nys (x) can be seen as an "empirical kernel map" and K DISPLAYFORM14 11 as a metric in the "empirical feature space". From this viewpoint, we think that it could be useful to learn a metric W in the empirical feature space instead of assuming it to be equal to K 11. In a sense, this should allow to learn a kernel by learning its Nyström feature representation. In the following, we call the setting where W is learned by the network Adapative Deepström Network. Principle Deepström networks we propose are an alternative to Deep Fried Convnets. They are based on using the Nyström approximation rather than the Fastfood one to integrate any kernel function on top of convolutional layers of a deep net. Indeed, although Deep Fried Convnets yield state-of-the-art with a significant gain with respect to memory resource and to inference complexity, it is restricted to the Gaussian kernel Fastfood, which may not be always the best choice in practice. In addition, our method can deal with multiple kernels. Deepström nets are Neural Networks that make use of a nonlinear representation function computed with the Nyström approximation (see Figure 1). Starting from Deep Fried Convnets we replace φ f f with φ nys so that a Deepström net implements a function f (x) = (lc • φ nys • conv)(x). In order to compute the above Nyström representation of a sample x one must consider a subsample L of training instances. Since the kernel k is computed on the representations given by convolutional layers, the samples in L must be represented in the same space, and hence must be processed by the convolutional layers as well. Once convolutional representations are calculated, the kernel function may be computed with an input sample and each instance in L in order to get the k x,L, which is then linearly transformed by W before the linear classification layer (see Figure 1).Two main structural differences between Deep Fried Convnets and our Deepström nets: (I) Nyström has the flexibility to use different kernel functions and to combine multiple kernels, and (II) in contrast to Fastfood the Nyström approximation is data dependent. However, one problem arises with the computation of K We present a series of experimental that explore the potential of Deepström networks with respect to various classification settings. First we consider a rather standard setting and compare our approach with standard models on image classification tasks. We explore in particular the behaviour of Deepström networks with various kernels and stress the very limited subsample size needed to reach state-of-the-art accuracy. Next we investigate the use of Deepström networks in a small training set setting, which shows that our approach may allow to learn new classes with only very few training samples, taking advantage of the reduced number of parameters learned by our model. Before describing all these we detail the datasets used. Finally we investigate first the multiple kernel architecture and illustrate its interest when learning with RBF kernel to overcome the hyperparameter selection, and second we demonstrate the benefit of a multiple Deepström approach, combining kernels computed from individual feature maps. We conducted experiments on four well known image classification datasets: MNIST, SVHN, CIFAR10 and CIFAR100 BID7, details on these datasets are provided in Table 1. We pretrained the convolutional layers using standard architectures on both datasets: Lenet for MNIST and VGG19 BID18 for SVHN, CIFAR10 and CIFAR100. We slightly modified the filters' sizes in Lenet network to ensure that the dimension of data after the convolution blocks is a power of 2 (needed for the Deep Fried Convnets architecture).We compare three convolutional architectures in all conducted experiments. Pretrained convolutional parts are shared by the three architectures, which differ from the layers on top of it: Dense architectures use dense hidden layers, i.e. these are classical convnets architectures; Deep Fried implements the Fastfood approximation (Equation 3); Deepstrom stands for our proposal. For Dense architectures, we considered one hidden layer with relu activation function, and varied the output dimension as {2, 4, 8, 16, 32, 64, 128, 1024} in order to highlight accuracies as a function of the number of parameters. For the Fastfood approximation in Deep Fried Convnets we consider that φ f f is gained with one stack of random features to form V in equation 3, except in the experiments of section 4.3 which yields a representation dimension up to 5 times larger. Regarding our approach φ nys, we varied the subset size L ∈ {2, 4, 8, 16, 32, 64, 128}, we tested with the linear, the RBF, and the Chi2 kernels, and we chose as output dimension the same size as the subset sample size. Finally we explored the adaptive as well as non-adaptive variants. Models were learned to optimize the cross entropy criterion with Adam optimizer and a gradient step fixed to 1e −4. Dropout was used on representation layers with probability equal to 0.5. By default the RBF bandwidth was set to the inverse of the mean distance between the representations, after the convolutional part, of pairs of training samples. All experiments were performed with Keras and Tensorflow. Note that the aim of all the experiments below is to investigate the potential of out architecture, not to reach or beat state of the art on the datasets considered. We then compare gained with our architecture and with state-of-the-art models, given a shared convolutional model. Consequently, we did not use tricks such as data augmentation and extensive tuning and, in particular, we did not use the best known convolutional architecture for each of the dataset, we rather used a reasonable deep architecture, VGG19, for the three datasets CIFAR10, CIFAR100 and SVHN. We compare now Deepström networks to two similar architectures, Deep Fried Convnets and classical convolutional networks (inspired from VGG19 and Lenet depending on the dataset). We vary the number of parameters of each architecture in order to highlight classification accuracy with respect to needed memory space. FIG2 shows the compared networks accuracy with respect to the number of parameters, and ignore parameters for convolutions layers to ease the readability. We repeated each experiments 10 times and plot average scores with standard deviations. Deepström models of increasing complexity (number of parameters) correspond to the use of subsample of increasing size from 2 (leftmost point) to 128 (rightmost point). One may see that there is no need of a large subsample here. This may be explained since the convolutional part of the network has been learned to yield quite robust and stable representations of input images. We provide a figure in the Appendix that illustrates this. The Deepström network is able to reach state-of-the-art performance using much fewer parameters than both classical networks and Deep Fried Convnets. Moreover, we also observe smaller variations that points out the robustness of our model. The flexibility in the choice of the kernel function is a clear advantage of out method. The best kernel is clearly dependent on the dataset (linear on MNIST, Chi2 on SVHN and CIFAR100, RBF on CIFAR10). While Random Features in DeepFried are restricted to RBF kernels, we show for instance a gain by using the Chi2 Kernel (k(x 1, x 2) = ||x 1 −x 2 || 2 /(x 1 +x 2)) that had been used for image classification BID20. We also notice the benefit of adaptive variants of Deepström model, suggesting that our model is able to learn and adapt useful Kernel function. Finally, note that we obtained very similar with neural architectures exploiting two hidden layers instead of one after the convolution module conv. Here we explore the ability of our model to work with few training samples, from very few to tens of samples per class. It is an expected benefit of the method since the use of kernels could take advantage of small training samples. Note that we do not exactly deal with a real small training set setting. These preliminary experiments aim to show how the final layers of a convolutional model may be learned from very few samples, given a frozen convolutional model. We actually performed the following experiments by exploiting a trained convolution model conv that has been learned on the full CIFAR100 training set and investigate the performance of Deepström architectures as a function of the training set used to learn the classification layers. One perspective of this work is to exploit such a strategy for domain adaptation settings where the convolutional model is trained on a training set within a different domain as the classes to be recognized. Having at our disposal such a trained convolution model conv, we leverage on the additional information that one may easily include in our models, which is brought by the subsample set. Notice that this subsample may include unlabeled samples since their labels are not used for optimizing the model. Table 2 reports the comparison of network architectures on four datasets. We consider Adaptive Deepström using Linear, RBF or Chi2 kernels and compare with Dense and Adaptive Deepfried for training set sizes of 5, 10 and 20 samples per class. We only consider here adaptive variants since they brought better than their non adaptive counterparts. We obtain models with different complexities: by increasing the hidden layer size in standard convolutional models, or by stacking the number of matrices V in DeepFried (up to 8 times, more was untractable on our machines), and by increasing the subset size in Deepström. Reported are averaged over 30 runs. One may see first that Deepstrom architectures outperfom baselines on every setting except for 5 training samples per class on MNIST. The linear kernel performs well on MNIST but is significantly worse than baselines on harder datasets. At the opposite, both ADSR and ADSC significantly outperfom Adaptive DeepFried for any dataset and perform on par or significantly better than Dense architectures on the hardest CIFAR100 dataset. Moreover one sees that no single kernel based Deepstrom architecture dominate on all settings, showing the potential interest of combining multiple kernels as following experiments will show. We report here gained using multiple kernels in two different ways. First we exploited the Multiple Kernels strategy that we described in section 3 for exploiting multiple kernels in the output of the convolutional blocks, conv. FIG4 reports gained when using a combination of RBF kernels with various bandwidths and for different subsample sizes. Our multiple kernel strategy, exploiting kernels defined with various values of the hyperparameter allows automatically handling this hyper-parameter which usually requires to be tuned either through cross validation or to be manually chosen. The plots show the accuracy on the CIFAR10 dataset as a function of σ value, where the performance of the multiple kernel Deepström is shown as a horizontal line. Plots report for various subsample size equal to 2 (left), 4 (middle) and 8 (right), averaged over 10 runs. As may be seen, using our Multiple kernel strategy allows adapting the kernel combination optimally from the data without requiring any prior choice on the RBF bandwith hyper-parameter. Second, we investigated another architecture that exploits Multiple Deepström approximations as presented in section 3. Here we use in parallel multiple Nyström approximations where kernels are dedicated to deal each with the output of a single feature map of the conv part. TAB4 reports on CIFAR100. We show the best performances obtained for each method by grid-searching on various hyper-parameters depending on the models, within a similar range of number of parameters. For Dense model, we considered one or two hidden layers of 16, 64, 128, 1024, 2048 or 4096 neurons. Deepfried is the adaptive variant where we varied the number of stacks in 1, 3, 5, 7. Deepström is also the adaptive variant where the subsample size is in 16, 64, 128, 256, 512. We observe that both Deepström models outperform the considered baselines, demonstrating the interest in combining Deepström approximations. We proposed Deepström, a new hybrid architecture that mixes deep networks and kernel methods. It is based on the Nyström approximation that allow considering any kind of kernel function in contrast to Deep Fried Convnets. Our proposal allows reaching state of the art while significantly reducing the number of parameters on various datasets, enabling in particular learning from few samples. Moreover the method allows to easily deal with multiple kernels and with multiple Deepström architectures. FIG5 plots the 2-dimensional φ nys representations of some CIFAR10 test samples obtained with a subsample of size equal to 2 (while the number of classes is 10) and two different kernels. One may see here that the 10 classes are already significantly well separated in this low dimensional representation space, illustrating that a very small sized subsammple is already powerfull. Beside, we experienced that designing Deepström Convnets on lower level features output by lower level convolution blocks may yield state-of-the-art performance as well while requiring larger subsamples.

represent the short summary for finding the corresponding scientific article (long)

A new neural architecture where top dense layers of standard convolutional architectures are replaced with an approximation of a kernel function by relying on the Nyström approximation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: represent the scientific article to find a short tldr

Question: The main goal of this short paper is to inform the neural art community at large on the ethical ramifications of using models trained on the imagenet dataset, or using seed images from classes 445 -n02892767- [’bikini, two-piece’] and 459- n02837789- [’brassiere, bra, bandeau’] of the same. We discovered that many of the images belong to these classes were verifiably pornographic, shot in a non-consensual setting, voyeuristic and also entailed underage nudity. Akin to the \textit{ivory carving-illegal poaching} and \textit{diamond jewelry art-blood diamond} nexuses, we posit there is a similar moral conundrum at play here and would like to instigate a conversation amongst the neural artists in the community. The emergence of tools such as BigGAN and GAN-breeder has ushered in an exciting new flavor of generative digital art, generated using deep neural networks (See for a survey). A cursory search on twitter 1 reveals hundreds of interesting art-works created using BigGANs. There are many detailed blog-posts 2 on generating neural art by beginning with seed images and performing nifty experiments in the latent space of BigGANs. At the point of authoring this paper, (8 September 2019, 4 :54 PM PST),users on the GanBreeder app 3 had discovered 49652500 images. Further, Christie's, the British auction house behemoth, recently hailed the selling of the neural network generated Portrait of Edmond Belamy for an incredible $432, 500 as signalling the arrival of AI art on the world auction stage. Given the rapid growth of this field, we believe this to be the right time to have a conversation about a particularly dark ethical consequence of using such frameworks that entail models trained on the ImageNet dataset which has many images that are pornographic, non-consensual, voyeuristic and also entail underage nudity. We argue that this lack of consent in the seed images used to train the models trickles down to the final artform in a way similar to the blood-diamond syndrome in jewelry art. An example: Consider the neural art image in Fig 1 we generated using the GanBreeder app. On first appearance, it is not very evident as to what the constituent seed classes are that went into the creation of this neural artwork image. When we solicited volunteers online to critique the artwork (See the collection of responses (Table 2) in the supplementary material), none had an inkling regarding a rather sinister trickle down effect at play here. As it turns out, we craftily generated this image using hand-picked specific instances of children images emanating from what we will showcase are two problematic seed image classes: Bikini and Brassiere. More specifically, for this particular image, we set the Gene weights to be: [Bikini: 42.35, Brassiere: 31.66, Comic Book -84.84]. We'd like to strongly emphasize at this juncture that the problem does not emanate from a visual patriarchal mindset, whereby we associate female undergarment imagery to be somehow unethical, but the root cause lies in the fact that many of the images were curated into the dataset (at least with regards to the 2 above mentioned classes) were voyeuristic, pornographic, non-consensual and also entailed underage nudity. 2 Root cause: Absence of referencing consent during the curation of the imagenet dataset The emergence of the ImageNet dataset is widely considered to be a pivotal moment 4 in the deep learning revolution that transformed the domain computer vision. Two highly cited papers (with more than 10000 citations each), authored by Deng et al in 2009 and authored by Russakovsky et al in 2015, provide deep insights into the procedure used to curate the dataset. In the 2009 paper, subsections 3.1-Collecting Candidate Images and 3.2-Cleaning Candidate Images are dedicated towards the algorithms used to collect and clean the dataset and also to elucidate the specific ways in which the Amazon Mechanical Turk (AMT) platform was harnessed to scale the dataset. Similarly the entirety of Section-3-Dataset construction at large scale in is dedicated towards extending the procedures for the 2015 release. It is indeed disappointing that neither the 2009 nor the 2015 versions of the endeavors required the AMT workers to check if the images they were asked to categorize and draw bounding boxes over, were ethically viable for usage. More specifically, in imagery pertaining to anthropocentric content, such as undergarment clothing, there was no attempt made towards assessing if the images entailed explicit consent given by the people in the images. In fact, none of the following words in the set [ethics, permission, voyeurism, consent] are mentioned in either of the two papers. As such, we have a plethora of images specifically belonging to the two categories detailed in Table 1, that have serious ethical shortcomings. In Fig 2, we showcase the gallery of images from the two classes categorized into four sub-categories: Non-consensual/Voyeuristic, Personal, Verifiably pornographic and Underage / Children. In Fig 2, we also include images that were also incorrectly categorized (Specifically there was no brassieres being sported by the subjects in the images) (Sub-figure (a) ) and those that involved male subjects indulging in lecherous tomfoolery (Sub-figure (e) ). In this paper, we expose a certain unethical facet of neural art that emerges from usage of nonconsensual images that are present in certain specific classes of the imagenet dataset. These images born out of an unethical (and in some cases, illegal) act of voyeuristic non-consensual photography predominantly targeting women (as well as children), might implicitly poison the sanctity of the artworks that eventually emerge. This work is complementary to works such as [10; 11] that have explored the unspoken ethical dimensions of harnessing cheap crowd-sourced platforms such as AMT for scientific research in the first place, which we firmly believe is also an important issue to be considered. We'd like to begin by admitting the self-contradictory facet of raising this specific issue by using a flagship example image that was generated using the very same ethically dubious procedure that we are targeting to root out. Secondly, we'd like to inform the reader that we have indeed raised the issue with curators of the imagenet dataset, but to no avail and plan to update this dissemination if and when they respond. This one reminds me of a mix between graffiti and paper mache using newspaper with color images or magazines. My attention is immediately drawn to near the top of the image which, at first glance, appears to be a red halo of sorts, but upon further consideration, looks to be long black branching horns on a glowing red . My attention then went to the center top portion, where the "horns" were coming from, which appeared to be the head or skull of a moose or something similar. The body of the creature appears to be of human-like form in a crucifix position, of sorts. The image appears more and more chaotic the further down one looks. Antisymmetric: left side is very artistic, rich in flavor and shades; right is more monotonic but has more texture. Reminds me of the two different sides of the brain through the anti-symmetry C-Data Scientist, Facebook Inc Futurism It's visually confusing in the sense that I couldn't tell if I was looking at a 3D object with a colorful or a painting. It's not just abstract, but also mysteriously detailed in areas to the point that I doubt that a human created these E -Senior software engineer, Mt View The symmetry implies a sort of intentionally. I get a sense of Picasso mixed with Frieda Callo here. F-Data Scientist, SF Reminds me of a bee and very colorful flowers, but with some nightmarish masks hidden in some places. Very tropical Table 2: Responses received for the neural art image in Fig 1

There's non-consensual and pornographic images in the ImageNet dataset

Represent the TLDR for finding the corresponding scientific article

Our paper analyses the tremendous representational power of networks especially with 'skip connections', which may be used as a method  for better generalization.