Question: represent the scientific article to find a short tldr

Question: Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives. But people can communicate objectives to each other simply by describing or demonstrating them. How can we build learning algorithms that will allow us to tell machines what we want them to do? In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments. We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network. We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a language-conditioned policy leads to poor performance. Figure 1: A task where an agent (green triangle) must execute the command "go to the fruit bowl." This is a simple example where the reward function is easier to specify than the policy. While reinforcement learning provides a powerful and flexible framework for describing and solving control tasks, it requires the practitioner to specify objectives in terms of reward functions. Engineering reward functions is often done by experienced practitioners and researchers, and even then can pose a significant challenge, such as when working with complex image-based observations. While researchers have investigated alternative means of specifying objectives, such as learning from demonstration BID1, or through binary preferences BID5, language is often a more natural and desirable way for humans to communicate goals. A common approach to building natural language interfaces for reinforcement learning agents is to build language-conditioned policies that directly map observations and language commands to a sequence of actions that perform the desired task. However, this requires the policy to solve two challenging problems together: understanding how to plan and solve tasks in the physical world, and understanding the language command itself. The trained policy must simultaneously interpret a command and plan through possibly complicated environment dynamics. The performance of the system then hinges entirely on its ability to generalize to new environments -if either the language interpretation or the physical control fail to generalize, the entire system will fail. We can recognize instead that the role of language in such a system is to communicate the goal, and rather than mapping language directly to policies, we propose to learn how to convert language-defined goals into reward functions. In this manner, the agent can learn how to plan and perform the task on its own via reinforcement learning, directly interacting with the environment, without relying on zero-shot transfer of policies. A simple example is shown in Figure 1, where an agent is tasked with navigating through a house. If an agent is commanded "go to the fruit bowl", a valid reward function could simply be a fruit bowl detector from first-person views of the agent. However, if we were to learn a mapping from language to actions, given the same goal description, the model would need to generate a different plan for each house. In this work, we investigate the feasibility of grounding free-form natural language commands as reward functions using inverse reinforcement learning (IRL). Learning language-conditioned rewards poses unique computational problems. IRL methods generally require solving a reinforcement learning problem as an inner-loop BID26, or rely on potentially unstable adversarial optimization procedures BID8 BID10. This is compounded by the fact that we wish to train our model across multiple tasks, meaning the IRL problem itself is an inner-loop. In order to isolate the language-learning problem from the difficulties in solving reinforcement learning and adversarial learning problems, we base our method on an exact MaxEnt IRL BID26 procedure, which requires full knowledge of environment dynamics to train a language-conditioned reward function represented by a deep neural network. While using exact IRL procedures may seem limiting, in many cases (such as indoor robotic navigation) full environment dynamics are available, and this formulation allows us to remove the difficulty of using RL from the training procedure. The crucial insight is that we can use dynamic programming methods during training to learn a reward function that maps from observations, but we do not need knowledge of dynamics to use the reward function, meaning during test time we can evaluate using a reinforcement learning agent without knowledge of the underlying environment dynamics. We evaluate our method on a dataset of realistic indoor house navigation and pick-and-place tasks using the SUNCG dataset, with natural language commands. We demonstrate that our approach generalizes not only to novel tasks, but also to entirely new scenes, while directly learning a language-conditioned policy leads to poor performance and fails to generalize. A popular class of approaches to language grounding in reinforcement learning is to directly train a policy that consumes language as an input. Several works adopt a behavioral cloning approach, where the model is trained using supervised learning with language-action sequences pairs BID0 BID15 BID23. A second approach is to forego demonstrations but instead reward an agent whenever the desired task is completed BID21 BID17 BID11 BID3 ). This approach requires reward functions (the task completion detector) to be hand-designed for the training tasks considered. Another related approach is semantic parsing, which has also been used to convert language into an executable form that corresponds to actions within an environment BID9 BID16 BID24. In a related task to instruction following, BID6 consider an embodied question-answering task where an agent must produce an answer to a question, where the relevant information lies within the environment. They adopt a hybrid approach, where they pretrain with supervised learning but also give the agent reward for completing intermediate tasks. Overall, our experiments show that policy-based approaches have worse generalization performance to new environments, because the policy must rely on zero-shot generalization at test time as we show in Section 6.3. While in this paper we argue for the performance benefits of a reward-based approach, a reason one may want to adopt a policy-based approach over a reward-based one is if one cannot run RL to train a new policy in a new environment, such as for time or safety reasons. A second approach to the language grounding problem is to learn a mapping from language to reward functions. There are several other works that apply IRL or IRL-like procedures to the problem of language grounding. Perhaps most closely related to our work is BID14, which also aims to learn a language-conditioned reward function via IRL. However, this method requires an extensively hand-designed, symbolic reward function class, whereas we use generic, differentiable function approximators that can handle arbitrary observations, including raw images. BID2; BID25 also learn language-conditioned reward functions, but do not perform IRL, meaning that the objective does not correspond to matching the expert's trajectory distribution. BID25 train a task-completion classifier, but do not evaluate their reward on control problems. The strategy they use is similar to directly regressing onto a ground-truth reward function, which we include a comparison to in Section 6 to as an oracle baseline. BID2 adopt an adversarial approach similar to GAIL BID12, and use the learned discriminator as the reward function. While this produces a reward function, it does not provide any guarantees that the ing reward function can be reoptimized in new environments to yield behavior similar to the expert. We believe our work is the first to apply language-conditioned inverse reinforcement learning to environments with image observations and deep neural networks, and we show that our rewards generalize to novel tasks and environments. We build off of the MaxEnt IRL model BID27, which considers an entropy-regularized Markov decision process (MDP), defined by the tuple (S, A, T, r, γ, ρ 0). S, A are the state and action spaces respectively and γ ∈ is the discount factor. T (s |s, a) represents the transition distribution or dynamics. We additionally consider partially-observed environments, where each state is associated with an observation within an observations space o ∈ O.The goal of "forward" reinforcement learning is to find the optimal policy π *. Let r(τ) = T t=0 γ t r(s t, a t) denote the returns of a trajectory, where τ denotes a sequence of states and actions DISPLAYFORM0 Inverse reinforcement learning (IRL) seeks to infer the reward function r(s, a) given a set of expert demonstrations D = {τ 1, ..., τ N},. In IRL, we assume the demonstrations are drawn from an optimal policy π * (a|s). We can interpret the IRL problem as solving the maximum likelihood problem: DISPLAYFORM1 In the MaxEnt IRL framework, optimal trajectories are observed with probabilities proportional to the exponentiated returns, meaning p(τ) ∝ exp{r(τ)} BID27. Thus, learning a reward function r θ (τ) is equivalent to fitting an energy-based model p θ (τ) ∝ exp{r θ (τ)} to the maximum likelihood objective in Eqn 1. The gradient to update the reward function is BID26: DISPLAYFORM2 where ρ D (s, a) represents the state-action marginal of the demonstrations, and ρ * θ (s, a) represents the state-action marginal of the optimal policy under reward r θ (s, a). A unique challenge of the language-conditioned IRL problem, compared to standard IRL, is that the goal is to learn a reward function that generalizes across multiple tasks. While standard IRL methods are typically trained and evaluated on the same task, we want our language-conditioned reward function to produce correct behavior when presented with new tasks. Several previous works consider a multi-task scenario, such as in a Bayesian or meta-learning setting BID13 BID7 BID4. We adopt a similar approach adapted for the language-IRL problem, and formalize the notion of a task, denoted by ξ, as an MDP, where individual tasks may not share the same state spaces, dynamics or reward functions. Each task is associated with a context c ξ which is a unique identifier (i.e. an indicator vector) for that task. Thus, we wish to optimize the following multi-task objective, where τ ξ denotes expert demonstrations for that task: Sample task ξ, demonstrations d ξ, and language L ξ. DISPLAYFORM0 Compute optimal q * (s, a) using q-iteration and ρ * (s, a) using the forward algorithm. Update reward r θ with the gradient (ρ DISPLAYFORM0 In order to optimize this objective, we first require that all tasks share the same observation space and action space, and the reward to be a function of the observation, rather than of the state. For example, in our experiments, all observations are in the form of 32x24 images taken from simulated houses, but the state space for each house is allowed to differ (i.e., the houses have different layouts). This means the same reward can be used across all MDPs even though the state spaces differ. Second, we share the reward function across all tasks, but substitute a language command L ξ as a proxy for the context c ξ, ing in a model p θ (τ ξ, L ξ) that takes as input language, states, and actions. For computational efficiency we run stochastic gradient descent on the objective in Eqn. 3 by sampling over the set of environments on each iteration. We learn language-conditioned reward functions using maximum causal entropy IRL, adapted for a multi-task setting and rewards represented by language-conditioned convolutional neural networks. While during training we use dynamic programming methods that require dynamics knowledge, we do not need knowledge of dynamics to evaluate the reward function. Thus, at test time we can use standard model-free RL algorithms to learn the task from the inferred reward function in new environments. Our algorithm is briefly summarized in Algorithm 1. In order to take gradient steps on the objective of Eqn. 3, we update our reward function in terms of the Maximum Entropy IRL gradient BID26 according to Eqn. 2. The stochastic gradient update (for a single task ξ) adapted to our case is: DISPLAYFORM0 Where o(s) denotes the observation for state s. Note that during training we need access to the ground truth states s. While the update depends on the underlying state, the reward itself is only a function of the observation, the action, and the language. This enables us to evaluate the reward without knowing the underlying state space and dynamics of the environment. While requiring dynamics knowledge during training may seem limiting, in practice many environments we may wish to train a robot in can easily be mapped. This training strategy is analogous to training a robot in only known environments such as a laboratory, but the ing reward can be used in unknown environments. In order to compute ρ * θ (s, a), one normally has to first compute the optimal policy with respect to reward r θ (o, a) using reinforcement learning, and then compute the occupancy measure using the forward algorithm for Markov chains to compute the state visitation distributions at each time-step. Because this embeds a difficult RL optimization problem within nested inner-loops, this quickly becomes computationally intractable. Thus, we train in tabular environments with known dynamics, where we can compute optimal policies exactly using Q-iteration. However, we emphasize that this is only a training time restriction, and knowledge of dynamics is not required to evaluate the rewards. Our network architecture is shown in FIG0. The network has two main modalities of input: a variable-length language input represented by a sequence of one-hot vectors (one vector for each tokenized word), and a panoramic image observation. The language embedding is formed by processing the language input sequence through an LSTM network, and the final time-step of the topmost layer is used as a fixed-dimensional embedding e language of the input command. The agent receives image observations in the form of four 32x24 image observations, one for each cardinal direction view (N, S, E, W). The convolutional neural network (CNN) consists of a sequence of convolutional and max pool layers, with the final operation being a channel-wise global pooling operation that produces an image embedding of the same length as the language embedding. Each image is passed through an identical CNN with shared weights, and the outputs are summed together to form the image embedding. That is, e image = CNN(img N) + CNN(img S) + CNN(img E) + CNN(img W).Finally, these two embeddings are element-wise multiplied and passed through a fully-connected network (FC) to produce a reward output. Letting denote elementwise multiplication, we have r = FC(e image e language).We found that the max global-pooling architecture in the CNN was able to select out objects from a scene and allow the language embedding to modulate which features to attend to. We selected our architecture via a hyper-parameter search, and found that the choice of using an element-wise multiplication versus a concatenation for combining embeddings had no appreciable performance difference, but a global pooling architecture performed significantly better than using a fully connected layer at the end of the CNN. We evaluate our method within a collection of simulated indoor house environments, built on top of the SUNCG BID22 dataset. The SUNCG dataset provides a large repository of complex and realistic 3D environments which we find very suitable for our goals. An example task from our environment is shown in FIG1. An example of a successful execution of a task is showin in FIG3. We consider two typical kinds of tasks that an indoor robot may wish to perform:• Navigation (NAV): In the navigation task, the agent is given a location which corresponds to a room or object, and the agent must navigate through the house to reach the target location. For example, in FIG1, the target could be "cup" or "laptop" or "living room". FIG1. We only use the semantic labels as input to our model. • Pick-and-place (PICK): In the pick and place task, the agent must move an object from one location to another. For example, in Fig. 1 the task could be to move the cup from to the sink to the kitchen table. Each environment corresponds to one 3D scene, which is discretized into a grid to form a tabular environment where the grid coordinates plus agent orientation (N, S, E, W) correspond to the state of the agent. The agent receives observations with two components: one is a free-form language command, and one is a first-person panoramic image of the environment. Because the agent can move objects without directly looking at them, the panoramic view gives the agent a full view of its surroundings. The panoramic image is formed from 4 semantic image observations, one for each orientation of the agent. Each semantic image observation is 32x32 pixels and contains 61 channels, one per semantic image class. Each agent is equipped with 4 actions: step forward one grid tile, turn left or right, or interact with an object. We generate language commands based on a preset grammar, and using names of objects and locations associated with the task. These are of the form "go to X" for NAV tasks, or "move X to Y" for PICK tasks, where X and Y stand for names of locations and objects within the environment. We explicitly do not use step-by-step instruction language such as "turn left, walk down the hallway, go through the door", as these commands remove the planning aspect of the problem and tell the agent directly which actions to take in order to solve the problem. The interact action only has meaning within the PICK task. Executing this action will either pick up an object if the agent is within a 1 meter of an object, or drop an object if the agent is currently holding an object. To limit the size of the state space, within a single task, there is only one object an agent may interact with and two locations the object can be in. This setup only increases the size of the state-space by a factor of 3. However, different objects may be placed in different locations across environments, meaning the model still must learn to detect the object in the correct location rather than memorizing the specific object and location associated with a single task. In order to evaluate how well different methods generalize, we split our dataset of tasks into three segments: a training set, and two test sets -"task" and "house". The "task" test set contains tasks within the same houses as training, but requires the agent to interact with novel combinations of objects and locations. The "house" test set contains tasks on entirely new houses that were not in the training set. The purpose of this split is to investigate varying degrees of generalization: the "task" test set requires the model to execute novel language commands, but using landmarks and objects which were seen during training. The "house" test set adds another layer of difficulty, requiring the model to detect familiar objects situated in entirely new scenes. In total, our dataset contains 1413 tasks (716 PICK, 697 NAV). Across all tasks, there are 14 objects, and 76 different house layouts. There are 1004 tasks (71%) in the training set, 236 (17%) in the "task" test set, and 173 (12%) in the "house" test set. We evaluate two methods for reward learning. LC-RL refers to the language-conditioned IRL method outlined in Section 5, which takes as input demonstration and language pairs and learns a shared reward function across all tasks. In particular, we use 10 demonstrations per task, sampled from the computed optimal policy. To provide an upper bound for performance, we can also regress directly onto the ground-truth rewards, a method we label as "Reward Regression". While this is not possible in a practical scenario, this evaluation serves to show oracle performance on our task. Note that our method does not require access to ground-truth rewards, and only uses demonstrations. Success rates for each reward-learning method are evaluated using two policy learning procedures. Q-iteration (QI) computes the optimal policy exactly using dynamic programming, which we report in TAB1. We also experiment with reoptimizing the learned reward using DQN BID18, a sample-based RL method that does not require ground-truth knowledge of the environment dynamics. We use the position, orientation, and whether an object is held as the observation (this is identical to the state representation). This experiment represents the testing use-case where we can evaluate the reward at test-time in novel, unmapped environments despite the fact that at training time we require dynamics knowledge. However, because the probability that the random policy receives rewards on our task is tiny, we found that epsilon-greedy exploration was not enough. Thus, we also report using a reward shaping term with a state-based potential equal to the optimal value function BID19. We note that this shaping term does require dynamics knowledge compute, but we include this to highlight the difficulty of the RL problem even if reward learning is done properly. We also compare against two baselines derived from GAIL BID12, using the learned discriminator as the "reward" function. We first compare to AGILE BID2, which modifies GAIL to use a goal-based discriminator and false negative filtering, using DQN as a policy optimizer and ρ = 0.25. We found it difficult to learn rewards using a reinforcement learningbased policy optimizer, and the model was only able to solve the simpler NAV environments. This experiment emphasizes the gap between using a sampling-based solver and an exact solver during the training of reward-based methods. To create a more fair comparison, we also compare against GAIL using a dynamic programming solver (labeled GAIL-Exact), and we see that the performance is comparable to LC-RL on training environments, but performs significantly worse on test environments. These are in line with our intuitions -GAIL and IRL are equivalent in training scenarios BID12, but the discriminator of GAIL does not correspond to the true reward function, and thus performs worse when evaluated in novel environments. In order compare against a policy-learning approach, we compare against an optimal behavioral cloning baseline. We train the optimal cloning baseline by computing the exact optimal policy using Q-iteration, and perform supervised learning to regress directly on to the optimal action probabilities. To make a fair comparison, we keep the policy architecture identical to the reward architecture, except we add two additional inputs: the orientation indicator and and indicator on whether the ob- ject (during PICK tasks) is held by the agent or not. Each indicator is transformed by an embedding lookup, and all embeddings are element-wise multiplied along with with the language and image embeddings in the original architecture. Our main experimental on reward learning are reported in TAB1, and experiments in reoptimizing the learned reward function are reported in TAB2. Qualitative with diagrams of learned reward functions can be found in Appendix B. Additional supplementary material can be viewed at https://sites.google.com/view/language-irl, and experiment hyperparameters are detailed in Appendix A.We found that both LC-RL and Reward Regression were able to learn reward functions which generalize to both novel tasks and novel house layouts, and both achieve significant performance over the policy-based approach. As expected, we found that Reward Regression has superior performance when compared to LC-RL, due to the fact that it uses oracle ground-truth supervision. We include examples of learned reward functions for both methods in Appendix B. We found that a common error made by the learned rewards, aside from simply misidentifying objects and locations, was rewarding the agent for reaching the goal position without placing the object down on PICK tasks. This is reflected in the as the performance on PICK tasks is much lower than that of NAV tasks. Additionally, there is some ambiguity in the language commands, as the same environment may contain multiple copies of a single object or location, and we do not consider the case when agents can ask for additional clarification (for example, there are 2 beds in FIG1).We observed especially poor performance from the cloning baseline on both training as well as testing environments, even though it was trained by directly regressing onto the optimal policy. We suspect that it is significantly more difficult for the cloning baseline to learn across multiple environments. Our language is high-level and only consists of descriptions of the goal task (such as "move the cup to the bathroom") rather than step-by-step instructions used in other work such as BID15 that allow the policy to follow a sequence of instructions. This makes the task much more difficult for a policy-learning agent as it needs to learn a mapping from language to house layouts instead of blindly following the actions specified in the language command. Regarding re-optimization of the learned rewards, we found that DQN with epsilon-greedy exploration alone achieved poor performance compared to the exact solver and comparable performance to the cloning baseline (however, note that our cloning baseline was regressing onto the exact optimal actions). Adding a shaping term based on the value-function improves , but computing this shaping term requires ground-truth knowledge of the environment dynamics. We also note that it appears that rewards learned through regression are easier to re-optimize than rewards learned through IRL. One explanation for this is that IRL rewards appear more "noisy" (for example, see reward plots in Appendix B, because small variations in the reward may not affect the trajectories taken by the optimal policy if a large reward occurs at the goal position. However, while RL is training it may never see the large reward and thus is heavily influenced by small, spurious variations in the reward. Nevertheless, with proper exploration methods, we believe that language-conditioned reward learning provides a performant and conceptually simple method for grounding language as concrete tasks an agent can perform within an interactive environment. In this paper, we introduced LC-RL, an algorithm for scalable training of language-conditioned reward functions represented by neural networks. Our method restricts training to tractable domains with known dynamics, but learns a reward function which can be used with standard RL methods in environments with unknown dynamics. We demonstrate that the reward-learning approach to instruction following outperforms the policy-learning when evaluated in test environments, because the reward-learning enables an agent to learn and interact within the test environment rather than relying on zero-shot policy transfer. For our environment, we give the agent a time limit 30 time-steps to complete a task. For the purpose of reward regression and generating demonstrations, the environment gives a reward of 10 when the agent successfully completes the task. We sample demonstrations from the optimal policy using this ground-truth reward. Our MDP solvers use a discount of γ = 0.99.For our model, we used 10 demonstrations per environment to train IRL, and optimized with Adam using a learning rate of 5 * 10 −4 .For our convolutional neural network, we used a 5x5 convolution with 16 filters, followed by a 3x3 convolution with 32 filters. The size of each embedding was 32. The final fully connected layer had sizes of 32, and 1 (for the final output). We did not find significant performance differences increasing the number of filters or embedding sizes. We selected our architecture through a hyper-parameter sweep, with train and test accuracies presented below (averaged over 3 seeds each). The main architectures we swept through were whether to produce the image embedding via a global pooling layer (labeled Pooling) versus a single fully connected layer (labeled FC) versus FiLM BID20, and whether to combine the language and image embeddings using a point-wise multiplication or a pooling operation (labeled Mult) versus concatenation (labeled Concat). Below is an example of a learned reward (and the computed value function) from our IRL model. The task presented here is to bring the fruit bowl (green arrow) to the bathroom (red arrow). In the top row, we plot the reward function, and in the bottom row we plot the ing value function. The left column shows the rewards/values before the object (fruit bowl) is acquired, and the right column shows the rewards/values after. Note that before the object is acquired, the value directs the agent to the fruit bowl, and once the object is found, the value directs the agent to the bathroom. In these figures, a blue shaded square means a high values and red means low. The green-outlined tiles correspond to all locations within a 1-meter radius of the fruit bowl, or all tiles an agent can pick up the bowl from. The red-outlined tile likewise represents all tiles within a 1-meter radius of the drop-off location in the bathroom. The blue-outlined square represents the starting location of the agent. Next, we show rewards learned by 3 different methods: inverse reinforcement learning (IRL), GAIL, and reward regression. Again, low rewards are denoted by red squares and high rewards are denoted by blue squares. For each task, we also include a birds-eye view of task, where the object is highlighted in green and the agent is denoted by a green triangle. In general, we find that rewards learned by IRL and GAIL tend to be noisy and contain small artifacts. This is not unexpected, as both of these methods are sample-based and observe demonstrations instead of ground-truth rewards as supervision. We believe that such artifacts are detrimental when using RL to reoptimize the learned reward, as without adequate exploration RL cannot find the large reward at the true goal state, and instead ends up finding local minima. Figure 6: Learned rewards and a corresponding birds-eye view rollout for the task "go to fruit bowl". Figure 7: Learned rewards and a corresponding birds-eye view rollout for the task "move pan to living room". The running time for dynamic programming algorithms for solving MDPs (such as q-iteration) scales with the size of the state space, and in our environments we found reward evaluation becoming a major bottleneck in runtime. One major optimization we make to our algorithm is to cache computation on repeated observations. Computation-wise, the main bottleneck in Algorithm 1 is evaluating and back-propagating the reward function at all states and actions, rather than Q-iteration itself (even though it carries a cubic run-time dependency on the size of the state space). However, in many environments this is extremely wasteful. For example, in the house depicted in FIG1, information about where the cup is located must be included in the state. However, if our observations are images of what the robot sees, whether the cup is in the kitchen or in the bathroom has no impact on the images inside the living room. This means that we should only need to evaluate our reward in each living room images once for both locations of the cup. A major factor in speeding up our computation was to cache such repeated computation. In practice we found this to be significant, ing in 10-100 times speedups on reward computation depending on the structure of the environment.

represent the short summary for finding the corresponding scientific article (long)

We ground language commands in a high-dimensional visual environment by learning language-conditioned rewards using inverse reinforcement learning.

Represent the TLDR for finding the corresponding scientific article

DCEM learns latent domains for optimization problems and helps bridge the gap between model-based and model-free RL --- we create a differentiable controller and fine-tune parts of it with PPO


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Many biological learning systems such as the mushroom body, hippocampus, and cerebellum are built from sparsely connected networks of neurons. For a new understanding of such networks, we study the function spaces induced by sparse random features and characterize what functions may and may not be learned. A network with d inputs per neuron is found to be equivalent to an additive model of order d, whereas with a degree distribution the network combines additive terms of different orders. We identify three specific advantages of sparsity: additive function approximation is a powerful inductive bias that limits the curse of dimensionality, sparse networks are stable to outlier noise in the inputs, and sparse random features are scalable. Thus, even simple brain architectures can be powerful function approximators. Finally, we hope that this work helps popularize kernel theories of networks among computational neuroscientists. Kernel function spaces are popular among machine learning researchers as a potentially tractable framework for understanding artificial neural networks trained via gradient descent [e.g. 1, 2, 3, 4, 5, 6]. Artificial neural networks are an area of intense interest due to their often surprising empirical performance on a number of challenging problems and our still incomplete theoretical understanding. Yet computational neuroscientists have not widely applied these new theoretical tools to describe the ability of biological networks to perform function approximation. The idea of using fixed random weights in a neural network is primordial, and was a part of Rosenblatt's perceptron model of the retina. Random features have then resurfaced under many guises: random centers in radial basis function networks, functional link networks, Gaussian processes (GPs), and so-called extreme learning machines; see for a review. Random feature networks, where the neurons are initialized with random weights and only the readout layer is trained, were proposed by Rahimi and Recht in order to improve the performance of kernel methods and can perform well for many problems. In parallel to these developments in machine learning, computational neuroscientists have also studied the properties of random networks with a goal towards understanding neurons in real brains. To a first approximation, many neuronal circuits seem to be randomly organized. However, the recent theory of random features appears to be mostly unknown to the greater computational neuroscience community. Here, we study random feature networks with sparse connectivity: the hidden neurons each receive input from a random, sparse subset of input neurons. This is inspired by the observation that the connectivity in a variety of predominantly feedforward brain networks is approximately random and sparse. These brain areas include the cerebellar cortex, invertebrate mushroom body, and dentate gyrus of the hippocampus. All of these areas perform pattern separation and associative learning. The cerebellum is important for motor control, while the mushroom body and dentate gyrus are The function shown is the sparse random feature approximation to an additive sum of sines, learned from poorly distributed samples (red crosses). Additivity offers structure which may be leveraged for fast and efficient learning. general learning and memory areas for invertebrates and vertebrates, respectively, and may have evolved from a similar structure in the ancient bilaterian ancestor. Recent work has argued that the sparsity observed in these areas may be optimized to balance the dimensionality of representation with wiring cost. Sparse connectivity has been used to compress artificial networks and speed up computation, whereas convolutions are a kind of structured sparsity. We show that sparse random features approximate additive kernels with arbitrary orders of interaction. The in-degree of the hidden neurons d sets the order of interaction. When the degrees of the neurons are drawn from a distribution, the ing kernel contains a weighted mixture of interactions. These sparse features offer advantages of generalization in high-dimensions, stability under perturbations of their input, and computational and biological efficiency. Now we will introduce the mathematical setting and review how random features give rise to kernels. The simplest artificial neural network contains a single hidden layer, of size m, receiving input from a layer of size l (Figure 1). The activity in the hidden layer is given by, for i ∈ [m], Here each φ i is a feature in the hidden layer, h is the nonlinearity, W = (w 1, w 2, . . ., w m) ∈ R l×m are the input to mixed weights, and b ∈ R m are their biases. We can write this in vector notation as Random features networks draw their input-hidden layer weights at random. Let the weights w i and biases b i in the feature expansion be sampled i.i.d. from a distribution µ on R l+1. Under mild assumptions, the inner product of the feature vectors for two inputs converges to its expectation We identify the limit with a reproducing kernel k(x, x) induced by the random features, since the limiting function is an inner product and thus always positive semidefinite. The kernel defines an associated reproducing kernel Hilbert space (RKHS) of functions. For a finite network of width m, the inner product 1 m φ(x) φ(x) is a randomized approximation to the kernel k(x, x). We now turn to our main : the general form of the random feature kernels with sparse, independent weights. For simplicity, we start with a regular model and then generalize the to networks with varying in-degree. Two kernels that can be computed in closed form are highlighted. Fix an in-degree d, where 1 ≤ d ≤ l, and let µ|d be a distribution on R d which induce, together with some nonlinearity h, the kernel denote this set of neighbors. Second, sample w ji ∼ µ|d if j ∈ N i and otherwise set w ji = 0. We find that the ing kernel Here x N denotes the length d vector of x restricted to the neighborhood N, with the other l − d entries in x ignored. More generally, the in-degrees may be chosen independently according to a degree distribution, so that d becomes a random variable. Let D(d) be the probability mass function of the hidden node in-degrees. Conditional on node i having degree d i, the in-neighborhood N i is chosen uniformly at random among the l di possible sets. Then the induced kernel becomes For example, if every layer-two node chooses its inputs independently with probability p, the is the probability mass function of the binomial distribution Bin(l, p). The regular model is a special case of with Extending the proof techniques in yields: C ) many features (the proof is contained in Appendix C). Two simple examples With Gaussian weights and regular d = 1, we find that (see Appendix B) 4 Advantages of sparse connectivity The regular degree kernel is a sum of kernels that only depend on combinations of d inputs, making it an additive kernel of order d. The general expression for the degree distribution kernel illustrates that sparsity leads to a mixture of additive kernels of different orders. These have been referred to as additive GPs, but these kind of models have a long history as generalized additive models [e.g. 28, 32]. For the regular degree model with d = 1, the sum in is over neighborhoods of size one, simply the individual indices of the input space. Thus, for any two input neighborhoods N 1 and N 2, we have |N 1 ∩ N 2 | = ∅, and the RKHS corresponding to k reg 1 (x, x) is the direct sum of the subspaces H = H 1 ⊕... ⊕ H l. Thus regular d = 1 defines a first-order additive model, where, all pairwise terms. These interactions are defined by the structure of the terms k d (x N, x N). Finally, the degree distribution D(d) determines how much weight to place on different degrees of interaction. Generalization from fewer examples in high dimensions Stone proved that first-order additive models do not suffer from the curse of dimensionality, as the excess risk does not depend on the dimension l. Kandasamy and Yu extended this to dth-order additive models and found a bound on the excess risk of O(l 2d n −2s for kernels with polynomial or exponential eigenvalue decay rates (n is the number of samples and the constants s and C parametrize rates). Without additivity, these weaken to O(n −2s 2s+l) and O(C l /n), much worse when l d. Similarity to dropout Dropout regularization in deep networks has been analyzed in a kernel/GP framework, leading to with D = Bin(l, p) for a particular base kernel. Dropout may thus improve generalization by enforcing approximate additivity, for the reasons above. Equations and are similar: They differ only by the presence of an 0 -"norm" versus an 1 -norm and the presence of the sign function. Both norms are stable to outlying coordinates in an input x. This property also holds for different nonlinearities and 1 < d l, since every feature φ i (x) only depends on d inputs, and therefore only a minority of the m features will be affected by the few outliers. 1 Sufficiently sparse features will then be less affected by sparse noise than a fully-connected network, offering denoising advantages [e.g . 20]. A regressor f (x) = α φ(x) built from these features is stable so long as α p is small, since |f (x) − f (x)| ≤ α p φ(x) − φ(x) q for any Hölder conjugates 1/p + 1/q = 1. Thus if x = x + e where e contains a small number of nonzero entries, then f (x) ≈ f (x) since φ(x) ≈ φ(x). Stability also may guarantee the robustness of these networks to sparse adversarial attacks, although exactly the conditions under which these approximations hold (p = ∞, q = 1 is an interesting case) we leave for future work. Computational Sparse random features give potentially huge improvements in scaling. Direct implementations of additive models incur a large cost for d > 1, since requires a sum over time to compute the Gram matrix of n examples and O(nl d) operations to evaluate f (x). In our case, since the random features method is primal, we need to perform O(nmd) computations to evaluate the feature matrix and the cost of evaluating f (x) remains O(md). 3 Sparse matrix-vector multiplication makes evaluation faster than the O(ml) time it takes when connectivity is dense. For ridge regression, we have the usual advantages that computing an estimator takes O(nm 2 + nmd) time and O(nm + md) memory, rather than O(n 3) time and O(n 2) memory for a naïve kernel ridge method. Biological In a small animal such as a flying insect, space is extremely limited. Sparsity offers a huge advantage in terms of wiring cost. Additive approximation also means that such animals can learn much more quickly, as seen in the mushroom body. While the previous computational points do not apply as well to biology, since real neurons operate in parallel, fewer operations translate into lower metabolic cost for the animal. Inspired by their ubiquity in biology, we have studied sparse random networks of neurons using the theory of random features, finding the advantages of additivity, stability, and scalability. This theory shows that sparse networks such as those found in the mushroom body, cerebellum, and hippocampus can be powerful function approximators. Kernel theories of neural circuits may be more broadly applicable in the field of computational neuroscience. Expanding the theory of dimensionality in neuroscience Learning is easier in additive function spaces because they are low-dimensional, a possible explanation for few-shot learning in biological systems. Our theory is complementary to existing theories of dimensionality in neural systems, which defined dimensionality using a skewness measure of covariance eigenvalues. Kernel theory extends this concept, measuring dimensionality similarly in the space of nonlinear functions spanned by the kernel. Limitations We model biological neurons as simple scalar functions, completely ignoring time and neuromodulatory context. It seems possible that a kernel theory could be developed for timeand context-dependent features. Our networks suppose i.i.d. weights, but weights that follow Dale's law should also be considered. We have not studied the sparsity of activity, postulated to be relevant in cerebellum. It remains to be demonstrated how the theory can make concrete, testable predictions, e.g. whether this theory may explain identity versus concentration encoding of odors or the discrimination/generalization tradeoff under experimental conditions. Appendices: Additive function approximation in the brain As said in the main text, Kandasamy and Yu created a theory of the generalization properties of higher-order additive models. They supplemented this with an empirical study of a number of datasets using their Shrunk Additive Least Squares Approximation (SALSA) implementation of the additive kernel ridge regression (KRR). Their data and code were obtained from https: //github.com/kirthevasank/salsa. We compared the performance of SALSA to the sparse random feature approximation of the same kernel. We employ random sparse Fourier features with Gaussian weights N (0, σ 2 I) with σ = 0.05 · √ dn 1/5 in order to match the Gaussian radial basis function used by Kandasamy and Yu. We use m = 300l features for every problem, with regular degree d selected equal to the one chosen by SALSA. The regressor on the features is cross-validated ridge regression (RidgeCV from scikit-learn) with ridge penalty selected from 5 logarithmically spaced points between 10 −4 · n and 10 2 · n. In Figure 2, we compare the performance of sparse random features to SALSA. Generally, the training and testing errors of the sparse model are slightly higher than for the kernel method, except for the forestfires dataset. We studied the speed of learning for a test function as well. The function to be learned f (x) was a sparse polynomial plus a linear term: The linear term took a ∼ N (0, I), the polynomial p was chosen to have 3 terms of degree 3 with In Figure 3, we show the test error as well as the selected ridge penalty for different values of d and n. With a small amount of data (n < 250), the model with d = 1 has the lowest test error, since this "simplest" model is less likely to overfit. On the other hand, in the intermediate data regime (250 < n < 400), the model with d = 3 does best. For large amounts of data (n > 400), all of the models with interactions d ≥ 3 do roughly the same. Note that with the RBF kernel the RKHS can still capture the degree 3 polynomial model. However, we see that the more complex models have a higher ridge penalty selected. The penalty is able to adaptively control this complexity given enough data. Here we show that sparse random features are stable for spike-and-slab input noise. In this example, the truth follows a linear model, where we have random input points x i ∼ N (0, I) and linear observations y i = x i β for i = 1,..., n and β ∼ N (0, I). However, we only have access to sparsely corruputed inputs w i = x i + e i, where e i = 0 with probability 1 − p and e i = x − x i with probability p, x ∼ N (0, σ 2 I). That is, the corrupted inputs are replaced with pure noise. We use p = 0.03 1 and σ = 6 1 so that the noise is sparse but large when it occurs. In Table 1 we show the performance of various methods on this regression problem given the corrupted data (W, y). Note that if the practitioner has access to the uncorrupted data X, linear regression succeeds with a perfect score of 1. Using kernel ridge regression with k(x, x) = 1 − Table 1: Scores (R 2 coefficient) of various regression models on linear data with corrupted inputs. In the presence of these errors, linear regression fails to acheive as good a test score as the kernel method, which is almost as good as trimming before performing regression and better than the robust Huber estimator. Figure 4: Kernel eigenvalue amplification while (left) varying p with σ = 6 fixed, and (right) varying σ with p = 0.03 fixed. Plotted is the ratio of eigenvalues of the kernel matrix corrupted by noise to those without any corruption, ordered from largest to smallest in magnitude. We see that the sparse feature kernel shows little noise amplification when it is sparse (right), even for large amplitude. On the other hand, less sparse noise does get amplified (left). best performance is attained by trimming the outliers and then performing linear regression. However, this is meant to illustrate our point that sparse random features and their corresponding kernels may be useful when dealing with noisy inputs in a learning problem. In Figure 4 we show another way of measuring this stability property. We compute the eigenvalues of the kernel matrix on a fixed dataset of size n = 800 points both with noise and without noise. Plotted are the ratio of the noisy to noiseless eigenvalues, in decibels, which we call the amplification and is a measure of how corrupted the kernel matrix is by this noise. The main trend that we see is, for fixed p = 3, changing the amplitude of the noise σ does not lead to significant amplification, especially of the early eigenvalues which are of largest magnitude. On the other hand, making the outliers denser does lead to more amplification of all the eigenvalues. The eigenspace spanned by the largest eigenvalues is the most "important" for any learning problem. We will now describe a number of common random features and the kernels they generate with fully-connected weights. Later on, we will see how these change as sparsity is introduced in the input-hidden connections. Translation invariant kernels The classical random features sample Gaussian weights w ∼ N (0, σ −2 I), uniform biases b ∼ U [−a, a], and employ the Fourier nonlinearity h(·) = cos(·). This leads to the Gaussian radial basis function kernel In fact, every translation-invariant kernel arises from Fourier nonlinearities for some distributions of weights and biases (Bôchner's theorem). Moment generating function kernels The exponential function is more similar to the kinds of monotone firing rate curves found in biological neurons. In this case, we have k(x, x) = E exp(w (x + x) + 2b). We can often evaluate this expectation using moment generating functions. For example, if w and b are independent, which is a common assumption, then where E (exp(w (x + x)) is the moment generating function for the marginal distribution of w, and E exp(2b) is just a constant that scales the kernel. For multivariate Gaussian weights w ∼ N (m, Σ) this becomes This equation becomes more interpretable if m = 0 and Σ = σ −2 I and the input data are normalized: This highlights that dot product kernels k(x, x) = v(x x), where v: R → R, are radial basis functions on the sphere S l−1 = {x ∈ R l : x 2 = 1}. The eigenbasis of these kernels are the spherical harmonics. Arc-cosine kernels This class of kernels is also induced by monotone "neuronal" nonlinearities and leads to different radial basis functions on the sphere. Consider standard normal weights w ∼ N (0, I) and nonlinearities which are threshold polynomial functions +, where Θ(·) is the Heaviside step function. The kernel in this case is given by for a known function J p (θ) where θ = arccos. Note that arc-cosine kernels are also dot product kernels. Also, if the weights are drawn as w ∼ N (0, σ −2 I), the terms x are replaced by x/σ, but this does not affect θ. With p = 0, corresponding to the step function nonlinearity, we have J 0 (θ) = π − θ, and the ing kernel does not depend on x or x: Sign nonlinearity We also consider a shifted version of the step function nonlinearity, the sign function sgn(z), equal to +1 when z > 0, −1 when z < 0, and zero when and w ∼ P, where P is any spherically symmetric distribution, such as a Gaussian. Then, where e = (x − x)/ x − x 2. The factor E(|w e|) in front of the norm is just a function of the radial part of the distribution P, which we should set inversely proportional to The sparsest networks possible have d = 1, leading to first-order additive kernels. Here we look at two simple nonlinearities where we can perform the sum and obtain an explicit formula for the additive kernel. In both cases, the kernels are simply related to a robust distance metric. This suggests that such kernels may be useful in cases where there are outlier coordinates in the input data. Step function nonlinearity We again consider the step function nonlinearity h(·) = Θ(·), which in the case of fully-connected Gaussian weights leads to the degree p = 0 arc-cosine kernel k(x, For a scalar a, normalization leads to a/ a = sgn(a). Therefore, θ = arccos (sgn(x i) sgn(x i)) = 0 if sgn(x i) = sgn(x i) and π otherwise. Performing the sum in, we find that the kernel becomes This kernel is equal to one minus the normalized Hamming distance of vectors sgn(x) and sgn(x). The fully-connected kernel, on the other hand, uses the full angle between the vectors x and x. The sparsity can be seen as inducing a "quantization," via the sign function, on these vectors. Finally, if the data are in the binary hypercube, with x and x ∈ {−1, +1} l, then the kernel is exactly one minus the normalized Hamming distance. Sign nonlinearity We now consider a slightly different nonlinearity, the sign function. It will turn out that the kernel is quite different than for the step function. This has h(·) = sgn(·) = 2Θ(·) − 1. Choosing P (w) = 1 2 δ(w + 1) + 1 2 δ(w − 1) and a 2 = −a 1 = a recovers the "random stump" of Rahimi and Recht. Despite the fact that sign is just a shifted version of the step function, the kernels are quite different: the sign nonlinearity does not exhibit the quantization effect and depends on the 1 -norm rather than the 0 -"norm". We now show a basic uniform convergence for any random features, not necessarily sparse, that use Lipschitz continuous nonlinearities. Recall the definition of a Lipschitz function: holds for all x, y ∈ X. Here, · is a norm on X (the 2 -norm unless otherwise specified). Assuming that h is Lipschitz and some regularity assumptions on the distribution µ, the random feature expansion approximates the kernel uniformly over X. As far as we are aware, this has not been stated previously, although it appears to be known (see Bach ) and is very similar to Claim 1 in Rahimi and Recht which holds only for random Fourier features (see also Sutherland and Schneider and Sriperumbudur and Szabo for improved in this case). The rates we obtain for Lipschitz nonlinearities are not essentially different than those obtained in the Fourier features case. As for the examples we have given, the only ones which are not Lipschitz are the step function (order 0 arc-cosine kernel) and sign nonlinearities. Since these functions are discontinuous, their convergence to the kernel occurs in a weaker than uniform sense. However, our does apply to the rectified linear nonlinearity (order 1 arc-cosine kernel), which is non-differentiable at zero but 1-Lipschitz and widely applied in artificial neural networks. The proof of the following Theorem appears at the end of this section. Assume that x ∈ X ⊂ R l and that X is compact, ∆ = diam(X), and the null vector 0 ∈ X. Let the weights and biases (w, b) follow the distribution µ on R l+1 with finite second moments. Let h(·) be a nonlinearity which is L-Lipschitz continuous and define the random feature φ: R l → R by φ(x) = h(w x − b). We assume that the following hold for all x ∈ X: |φ(x)| ≤ κ almost surely, E |φ(x)| 2 < ∞, and ≤ with probability at least Sample complexity Theorem 1 guarantees uniform approximation up to error using m = O features. This is precisely the same dependence on l and as for random Fourier features. Note that also found that m should scale linearly with l to minimize error in a particular classification task. A limitation of Theorem 1 is that it only shows approximation of the limiting kernel rather than direct approximation of functions in the RKHS. A more detailed analysis of the convergence to RKHS is contained in the work of Bach, whereas Rudi and Rosasco directly analyze the generalization ability of these approximations. Sun et al. show even faster rates which also apply to SVMs, assuming that the features are compatible ("optimized") for the learning problem. Also, the techniques of Sutherland and Schneider and Sriperumbudur and Szabo could be used to improve our constants and prove convergence in other L p norms. In the sparse case, we must extend our probability space to capture the randomness of the degrees, the neighborhoods conditional on the degree, and the weight vectors conditional on the degree and neighborhood. The degrees are distributed independently according to d i ∼ D, with some abuse of notation since we also use D(d) to represent the probability mass function. We shall always think of the neighborhoods N ∼ ν|d as chosen uniformly among all d element subsets, where ν|d represents this conditional distribution. Finally, given a neighborhood of some degree, the nonzero weights and bias are drawn from a distribution (w, b) ∼ µ|d on R d+1. For simpler notation, we do not show any dependence on the neighborhood here, since we will always take the actual weight values to not depend on the particular neighborhood N. However, strictly speaking, the weights do depend on N because that determines their support. Finally, we use E to denote expectation over all variables (degree, neighborhood, and weights), whereas we use E µ|d for the expectation under µ|d for a given degree. Corollary 2 (Kernel approximation with sparse features). Assume that x ∈ X ⊂ R l and that X is compact, ∆ = diam(X), and the null vector 0 ∈ X. Let the degrees d follow the degree distribution D on [l]. For every d ∈ [l], let µ|d denote the conditional distributions for (w, b) on R d+1 and assume that these have finite second moments. Let h(·) be a nonlinearity which is L-Lipschitz continuous, and define the random feature φ: R l → R by φ(x) = h(w x − b), where w follows the degree distribution model. We assume that the following hold for all x N ∈ X N with |N | = d, and for all 1 ≤ d ≤ l: |φ(x N)| 2 ≤ κ almost surely under µ|d, E |φ(x N)| 2 |d < ∞, and, with probability at least The kernels k are given by equations and. Proof. It suffices to show that conditions on the conditional distributions µ|d, d ∈ [l], imply conditions in Theorem 1. Conditions and clearly hold, since the distribution D has finite support. By construction,, which concludes the proof. Differences of sparsity The only difference we find with sparse random features is in the terms E w 2 and E w, since sparsity adds variance to the weights. This suggests that scaling the weights so that E µ|d w 2 is constant for all d is a good idea. For example, setting With this choice, the number of sparse features needed to achieve an error is the same as in the dense case, up to a small constant factor. This is perhaps remarkable since there could be as many as 2 l terms in the expression of k dist D (x, x). However, the random feature expansion does not need to approximate all of these terms well, just their average. Proof of Theorem 1. We follow the approach of Claim 1 in, a similar for random Fourier features but which crucially uses the fact that the trigonometric functions are differentiable and bounded. For simplicity of notation, let ξ = (x, x) and define the direct sum norm on X + = X ⊕ X as ξ + = x + x. Under this norm X + is a Banach space but not a Hilbert space, however this will not matter. For i = 1,..., m, let f i (ξ) = φ i (x)φ i (x), and note that these g i are i.i.d., centered random variables. By assumptions and, f i and g i are absolutely integrable and k(x, x) = E φ i (x)φ i (x). Denote their mean bȳ Our goal is to show that |ḡ(ξ)| ≤ for all ξ ∈ X + with sufficiently high probability. The space X + is compact and 2l-dimensional, and it has diameter at most twice the diameter of X under the sum norm. Thus we can cover X + with an -net using at most T = (4∆/R) 2l balls of radius R. Call the centers of these balls ξ i for i = 1,..., T, and letL denote the Lipschitz constant ofḡ with respect to the sum norm. Then we can show that |ḡ(ξ)| ≤ for all ξ ∈ X + if we show that we have that f i has Lipschitz constant κL w i. This implies that g i has Lipschitz constant ≤ κL(w i + E w). LetL denote the Lipschitz constant ofḡ. Note that EL ≤ 2κLE w. Also, Markov Now we would like to show that |ḡ(ξ i)| ≤ /2 for all i = 1,..., T anchors in the -net. A straightforward application of Hoeffding's inequality and a union bound shows that since |f i (ξ)| ≤ κ 2. Combining equations and in a probability of failure Pr sup Set R = (a/b) 1 2l+2, so that the probability has the form, 2a for all l ∈ N, assuming ∆κL E w 2 + 3(E w) 2 >. Considering the complementary event concludes the proof.

Represent the short summary for finding the corresponding scientific article (long)

We advocate for random features as a theory of biological neural networks, focusing on sparsely connected networks

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.