Question:
represent the article for finding a single-sentence tldr
Examples:
'It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned. In this paper, we draw inspiration from the recent success of sequence-to-sequence models for structured prediction problems to develop policies over discretized spaces. Central to this method is the realization that complex functions over high dimensional spaces can be modeled by neural networks that predict one dimension at a time. Specifically, we show how Q-values and policies over continuous spaces can be modeled using a next step prediction model over discretized dimensions. With this parameterization, it is possible to both leverage the compositional structure of action spaces during learning, as well as compute maxima over action spaces (approximately). On a simple example task we demonstrate empirically that our method can perform global search, which effectively gets around the local optimization issues that plague DDPG. We apply the technique to off-policy (Q-learning) methods and show that our method can achieve the state-of-the-art for off-policy methods on several continuous control tasks. Reinforcement learning has long been considered as a general framework applicable to a broad range of problems. However, the approaches used to tackle discrete and continuous action spaces have been fundamentally different. In discrete domains, algorithms such as Q-learning leverage backups through Bellman equations and dynamic programming to solve problems effectively. These strategies have led to the use of deep neural networks to learn policies and value functions that can achieve superhuman accuracy in several games (; where actions lie in discrete domains. This success spurred the development of RL techniques that use deep neural networks for continuous control problems BID12 BID20 . The gains in these domains, however, have not been as outsized as they have been for discrete action domains. This disparity is, in part, a of the inherent difficulty in maximizing an arbitrary function on a continuous domain, even in low-dimensional settings. Furthermore, it becomes harder to apply dynamic programming methods to back up value function estimates from successor states to parent states in continuous control problems. Several of the recent continuous control reinforcement learning approaches attempt to borrow characteristics from discrete problems by proposing models that allow maximization and backups more easily BID12 .One way in which continuous control can avail itself of the above advantages is to discretize each of the dimensions of continuous control action spaces. As noted in, doing this naively, however, would create an exponentially large discrete space of actions. For example with M dimensions being discretized into N bins, the problem would balloon to a discrete space with M N possible actions. We leverage the recent success of sequence-to-sequence type models BID32 to train such discretized models, without falling into the trap of requiring an exponentially large number of actions. Our method relies on a technique that was first introduced in BID3, which allows us to escape the curse of dimensionality in high dimensional spaces by modeling complicated probability distributions using the chain rule decomposition. In this paper, we similarly parameterize functions of interest -Q-values -using a decomposition of the joint function into a sequence of conditional values tied together with the bellman operator. With this formulation, we are able to achieve fine-grained discretization of individual domains, without an explosion in the number of parameters; at the same time we can model arbitrarily complex distributions while maintaining the ability to perform (approximate) global maximization. These benefits come at the cost of shifting the exponentially complex action space into an exponentially complex MDP BID5 BID10. In many settings, however, there are relationships between transitions that can be leveraged and large regions of good solutions, which means that this exponential space need not be fully explored. Existing work using neural networks to perform approximate exponential search is evidence of this BID37; BID2.While this strategy can be applied to most function approximation settings in RL, we focus on off-policy settings with an algorithm akin to DQN. Empirical on an illustrative multimodal problem demonstrates how our model is able to perform global maximization, avoiding the exploration problems faced by algorithms like NAF BID13 and DDPG. We also show the effectiveness of our method on a range of benchmark continuous control problems from hopper to humanoid. In this paper, we introduce the idea of building continuous control algorithms utilizing sequential, or autoregressive, models that predict over action spaces one dimension at a time. Here, we use discrete distributions over each dimension (achieved by discretizing each continuous dimension into bins) and apply it using off-policy learning. We briefly describe the notation we use in this paper. Let s t ∈ R L be the observed state of the agent, a ∈ R N be the N dimensional action space, and E be the stochastic environment in which the agent operates. Finally, let a i:j = a i · · · a j T be the vector obtained by taking the sub-range/slice of a DISPLAYFORM0 At each step t, the agent takes an action a t, receives a reward r t from the environment and transitions stochastically to a new state s t+1 according to (possibly unknown) dynamics p E (s t+1 |s t, a t). An episode consists of a sequence of such steps (s t, a t, r t, s t+1), with t = 1 · · · H where H is the last time step. An episode terminates when a stopping criterion F (s t+1) is true (for example when a game is lost, or when the number of steps is greater than some threshold length H max).Let R t = H i=t γ i−1 r i be the discounted reward received by the agent starting at step t of an episode. As with standard reinforcement learning, the goal of our agent is to learn a policy π (s t) that maximizes the expected future reward E [R H] it would receive from the environment by following this policy. Because this paper is focused on off-policy learning with Q-Learning BID39, we will provide a brief description of the algorithm. Q-learning is an off-policy algorithm that learns an action-value function Q (s, a) and a corresponding greedy-policy, π Q (s) = argmax a Q (s, a). The model is trained by finding the fixed point of the Bellman operator, i.e. DISPLAYFORM0 This is done by minimizing the Bellman Error, over the exploration distribution, ρ β (s) DISPLAYFORM1 Traditionally, Q is represented as a table of state action pairs or with linear function approximators or shallow neural networks BID39 BID35. Recently, there has been an effort to apply these techniques to more complex domains using non-linear function approximators that are deep neural networks (; . In these models, a Deep Q-Network (DQN) parameterized by parameters, θ, is used to predict Q-values, i.e. Q(s, a) = f (s, a; θ). The DQN parameters, θ, are trained by performing gradient descent on the error in equation 2, without taking a gradient through the Q-values of the successor states (although, see BID1 for an approach that takes this into account).Since the greedy policy, π Q (s), uses the action value with the maximum Q-value, it is essential that any parametric form of Q be able to find a maxima easily with respect to actions. For a DQN where the output layer predicts the Q-values for each of the discrete outputs, it is easy to find this max -it is simply the action corresponding to the index of the output with the highest estimated Q-value. In continuous action problems, it can be tricky to formulate a parametric form of the Q-value where it is easy to find such a maxima. Existing techniques either use a restrictive functional form, such as NAF BID13. DDPG employs a second neural network to approximate this max, in addition to the Q function approximator. This second network is trained to maximize / ascend the Q function as follows: DISPLAYFORM2 where ρ β is the state distribution explored by some behavioral policy, β and µ(·; θ µ) is the deterministic policy. In this work we modify the form of our Q-value function while still retaining the ability to find local maxima over actions for use in a greedy policy. In this section, we outline our proposed model, Sequential DQN (SDQN). This model decomposes the original MDP model with N -D actions to a similar MDP which contains sequences of 1-D actions. By doing this, we have 2 layer hierarchy of MDP -the "upper" containing the original environment, and the "lower" containing the transformed, stretched out, MDP. Both MDP model the same environment. We then combine these MDP by noting equality of Q values at certain states and doing bellman backups against this equality. See figure 1 for a pictorial view of this hierarchy. Consider an environment with states s t and actions a ∈ R N. We can perform a transformation to this environment into a similar environment replacing each N -D action into a sequence of N 1-D actions. This introduces a new MDP consisting of states u st k where superscript denotes alignment to the state s t, above, and subscript k to denote time offset on the lower MDP from s t. As a , u st k = (s t, a 1:k) is a tuple containing the state s t from original MDP and a history of additional states in the new MDP -in our case, a concatenation of actions a 1:k previously selected. The transitions of this new MDP can be defined by two rules: when all 1-D actions are taken we compute 1 step in the N -D environment receiving a new state, s t+1, a reward r t, and resetting a. In all other transitions, we append the previously selected action in a and receive 0 reward. This transformation reduces the N -D actions to a series of 1-D actions. We can now discretize the 1-D output space and directly apply Q-learning. Note that we could apply this strategy to continuous values, without discretization, by choosing a conditional distribution, such as a mixture of 1-D Gaussians, over which a maxima can easily be found. As such, this approach is equally applicable to pure continuous domains as compared to discrete approximations. The downside to this transformation is that it increases the number of steps needed to solve the transformed MDP. In practice, this transformation makes learning a Q-function considerably harder. The extra steps of dynamic programming coupled with learned function approximators causes large overestimation and stability issues. This can be avoided by learning Q-values for both MDPs at the same time and performing the bellman backup from the lower to the upper MDP for the transitions, s t, where Q-values should be equal. We define Q U (s, a) ∈ R, a ∈ R N, as a function that is the Q-value for the top MDP. Next, we define Q L (u, a i) ∈ R where a i ∈ R as the Q value for the lower MDP. We would like to have consistent Q-values across both MDPs when they perform one step in the environment. To make this possible, we must define how the time discounting works. We define the lower MDP to have zero discount for all steps except for when the real environment changes state. Thus, the discount is 0 for all all u st k where k < N, and the same as the top MDP when k = N. By doing this, the following is then true: DISPLAYFORM0 where u st N −1 contains the upper state and the previous N − 1 actions: (s t, a t 1:N −1). This equality allows us to "short circuit" the backups. Backups are only needed up until the point where the upper MDP can be used improving training and stability. During training, we parameterize Q U and Q L as neural networks. We learn Q U via TD-0 learning by minimizing: DISPLAYFORM1 Next, we learn Q L by also doing Q-learning, but we make use of of the equality noted in equation 3 and zero discounting. There is no new information nor environment dynamics for this MDP. As such we can draw samples from the same replay buffer used for learning Q U. For states u st k where k < N we minimize the bellman error as follows: DISPLAYFORM2 When Q U and Q L should be equal, as defined in equation 3, we do not backup we instead enforce soft equality by MSE. DISPLAYFORM3 In practice, as in DQN, we can also make use of target networks and/or double DQN BID15 when training Q U and Q L for increased stability. When using this model as a policy we compute the argmax over each action dimension of the lower MDP. As with DQN, we employ exploration when training with either epsilon greedy exploration or Boltzmann exploration. Q U is a MLP whose inputs are state and actions and outputs are Q values. Unlike in DDPG, the loss function does not need to be smooth with respect to actions. As such, we also feed in discretized representation of each action dimension to make training simpler. We worked with two parameterizations for Q L. First, we looked at a recurrent LSTM model BID17. This model has shared weights and passes information via hidden activations from one action dimension to another. The input at each time step is a function of the current state from the upper MDP, s t, and a single action dimension, a i. As it's an LSTM, the hidden state is capable of accumulating the previous actions. Second, we looked at a version with separate weights for each step of the lower MDP. The lower MDP does not have a fixed size input as the amount of action dimensions it has as inputs varies. To combat this, we use N separate models that are switched between depending on the state index of the lower MDP. We call these distinct models Q i where i ∈ [1, N]. These models are feed forward neural networks that take as input a concatenation of all previous action selections, a 1 t: i, as well as the upper state, s t. Doing this in switching Q L with the respective Q i in every use. Empirically we found that this weight separation led to more stable training. In more complex domains, such as vision based control tasks for example, one should untie only a subset of the weights and keep common components -a vision system -the same. In practice, we found that for the simplistic domains we worked in fully untied weights was sufficient. Architecture exploration for these kinds of models is still ongoing work. For full detail of model architectures and training procedures selection see Appendix C.3 RELATED WORK Our work was motivated by two distinct desires -to learn policies over exponentially large discrete action spaces, and to approximate value functions over high dimensional continuous action spaces effectively. In our paper we used a sequential parameterization of policies that help us to achieve this without making an assumption about the actual functional form of the model. Other prior work attempts to handle high dimensional action spaces by assuming specific decompositions. For example, BID26 were able to scale up learning to extremely large action sizes by factoring the action value function and use product of experts to learn policies. An alternative strategy was proposed in using action embeddings and applying k-nearest neighbors to reduce scaling of action sizes. By laying out actions on a hypercube, BID23 are able to perform a binary search over actions ing in a logarithmic search for the optimal action. Their method is similar to SDQN, as both construct a Q-value from sub Q-values. Their approach presupposes these constraints, however, and optimizes the Bellman equation by optimizing hyperplanes independently thus enabling optimizing via linear programming. Our approach is iterative and refines the action selection, which contrasts to their independent sub-plane maximization. and BID22 proposes a transformation similar to ours where a continuous action MDP is converted to a sequence of transitions representing a binary search over the continuous actions. In our setting, we used a 2-layer hierarchy of variable width as opposed to a binary tree. Additionally, we used the original MDP as part of our training procedure to reduce estimation error. We found this to be critical to reduce overestimation error when working with function approximators. Along with the development of discrete space algorithms, researchers have innovated specialized solutions to learn over continuous state and action environments including BID30 BID13. More recently, novel deep RL approaches have been developed for continuous state and action problems. TRPO BID27 and A3C uses a stocastic policy parameterized by diagonal covariance Gaussian distributions. NAF BID13 relies on quadratic advantage function enabling closed form optimization of the optimal action. Other methods structure the network in a way such that they are convex in the actions while being non-convex with respect to states BID0 or use a linear policy BID24.In the context of reinforcement learning, sequential or autoregressive policies have previously been used to describe exponentially large action spaces such as the space of neural architectures, BID40 and over sequences of words BID2 BID29. These approaches rely on policy gradient methods whereas we explore off-policy methods. Hierarchical/options based methods, including BID9 ) which perform spatial abstraction or BID34 that perform temporal abstraction pose another way to factor action spaces. These methods refine their action selection from time where our approaches operates on the same timescale and factors the action space. A vast literature on constructing sequential models to solve tasks exists outside of RL. These models are a natural fit when the data is generated in a sequential process such as in language modeling This in more sample efficiency but yields poor approximations of the Q surface outside of where the policy is. Right: Reward achieved over time. DDPG quickly converges to a local maximum. SDQN has high variance performance initially as it searches the space, but then quickly converges to the global maximum as the Q surface estimate becomes more accurate. BID4. One of the first and most effective deep learned sequence-to-sequence models for language modeling was proposed in BID33, which used an encoder-decoder architecture. In other domains, techniques such as NADE BID19 To consider the effectiveness of our algorithm, we consider a deterministic environment with a single time step, and a 2D action space. This can be thought of as being a two-armed bandit problem with deterministic rewards, or as a search problem in 2D action space. We chose our reward function to be a multimodal distribution as shown in the first column in FIG2. A large suboptimal mode and a smaller optimal mode exist. As with bandit problems, this formulation helps us isolate the ability of our method to find an optimal policy, without the confounding effect that arises from backing up rewards via the Bellman operator for sequential problems. As in traditional RL, we do exploration while learning. We consider uniformly sampling (-greedy with = 1) as well as sampling data from a normal distribution centered at the current policy -we refer to this as "local." A visualization of the final Q surfaces as well as training curves can be found in FIG2. DDPG uses local optimization to learn a policy on a constantly changing estimate of Q values predicted by a critic. The form of the Q distribution is flexible and as such there is no closed form properties we can make use of for learning a policy. As such, gradient descent, a local optimization algorithm, is used. This algorithm can get stuck in a sub-optimal policy. We hypothesize that these local maximum in policy space exist in more realistic simulated environments as well. Traditionally, deep learning methods use local optimizers and avoid local minima or maxima by working in a high dimensional parameter space BID8. In RL, however, the action space of a policy is relatively small dimensional thus it is much more likely that they exist. For example, in the hopper environment, a common failure mode we experienced when training algorithms like DDPG is to learn to balance instead of moving forward and hopping. We contrast this to SDQN. As expected, this model is capable of completely representing the Q surface (under the limits of discretization). The optimization of the policy is not done locally however enabling convergence to the optimal policy. Much like DDPG, the Q surface learned can be done on uniform, off policy, data. Unlike DDPG, however, the policy will not get stuck in a local maximum. In the uniform behavior policy setting, the model slowly reaches the right solution.1 With a behavior policy that is closer to being on-policy (such as the stochastic Gaussian greedy policy referred to above), the rate of convergence increases. Much of the error occurs from selecting over estimated actions. When sampling more on policy, the over estimated data points get sampled more frequently ing in faster training. To evaluate the relative performance of these models we perform a series of experiments on common continuous control tasks. We test the hopper (3-D action space), swimmer (2-D action space), half cheetah (6-D action space), walker2d (6-D action space) and the humanoid environment (17-D action space) from the OpenAI gym suite BID6. 2 We performed a wide hyper parameter search over various parameters in our models (described in Appendix C), and selected the best performing runs. We then ran 10 random seeds of the same hyper parameters to evaluate consistency and to get a more realistic estimate of performance. We believe this replication is necessary as many of these algorithms are not only sensitive to both hyper parameters but random seeds. First, we look at learning curves of some of the environments tested in FIG4. Our method quickly achieves good policies much faster than DDPG. For a more qualitative analysis, we use the best reward achieved while training averaged across over 25,000 steps and with evaluations sampled every 5,000 steps. Again we perform an average over 10 different random seeds. This metric gives a much better sense of stability than the traditionally reported instantaneous max reward achieved during training. We compare our algorithm to the current state-of-the-art in off-policy continuous control: DDPG. Through careful model selection and extensive hyper parameter tuning, we train DDPG agents with performance better than previously published for some of these tasks. Despite this search, however, we believe that there is still space for significant performance gain for all the models given different neural network architectures and hyper parameters. See for discussion on implementation variability and performance. Results can be seen in Figure 4. Our algorithm achieves better performance on four of the five environments we tested. Unlike existing continuous control algorithms, we have a choice over the number of discritization bins we choose, B. To test the effect of this we first take the best performing half cheetah hyper parameter configuration found above, and rerun it varying the number of bins. For statistical significance we run 10 trials per tested value of B. Results can be found in Figure 5. These suggest that SDQN is robust to this hyper parameter, working well in all bin amounts greater than 4. Lower than 4 bins does not yield enough fine grain enough control to solve this task effectively. Next we look to the effect of action order. In most existing environments there is no implicit "ordering" to environments. Given the small action space dimensionality, we hypothesized that this ordering would not matter. We test this hypothesis by taking our hyper parameters for half cheetah found in section 4.2 and reran them with random action ordering, 10 times for statistical significance. Half cheetah has 6 action dimensions thus we are only exploring a subset of orderings. Seed 0 represents the ordering used when finding the original hyper parameters. Results can be found in Figure 5. While there is some variability, the overall changes in performance are small validating our original assumption. Conceptually, our approach centers on the idea that action selection at each stage can be factored and sequentially selected. In this work we use 1-D action spaces that are discretized as our base component. Existing work in the image modeling domain suggests that using a mixture of logistic units BID25 greatly speeds up training and would also satisfy our need for a closed form max. Additionally, this work imposes a prespecified ordering of actions which may negatively impact training for certain classes of problems (with much larger number of action dimensions). To address this, we could learn to factor the action space into the sequential order for continuous action spaces or learn to group action sets for discrete action spaces. Another promising direction is to combine this approximate max action with gradient based optimization procedure. This would relieve some of the complexity of the modeling task of the maxing network, at the cost of increased compute when sampling from the policy. Finally, the work presented here is exclusively on off-policy methods. We chose to focus on these methods due to their sample efficiency. Use of an sequential policies with discretized actions could also be used as the policy for any stochastic policy optimization algorithm such as TRPO BID27 or A3C . In this work we present a continuous control algorithm that utilize discretized action spaces and sequential models. The technique we propose is an off-policy RL algorithm that utilizes sequential prediction and discretization. We decompose our model into a hierarchy of Q function. The effectiveness of our method is demonstrated on illustrative and benchmark tasks, as well as on more complex continuous control tasks. Sampling an Action To gain insight into the characteristics of Q that our SDQN algorithm learns, we visualized from the hopper environment as it is complex but has a small dimensional action space. Figure B.3: Exploration of the sub-DQN during after training. The top row shows the Q i predictions for a given frame (action dimensions correspond to the joint starting at the top and moving toward the bottom -action 3 is the ankle joint). The bottom row shows the corresponding rendering of the current state. For insensitive parts of the gait, such as when the hopper is in the air (e.g. frame 430, 442, 490, 502), the network learns to be agnostic to the choice of actions; this is reflected in the flat Q-value distribution, viewed as a function of action index. On the other hand, for critical parts of the gait, such as when the hopper is in contact with the ground (e.g. frames 446, 478), the network learns that certain actions are much better than others, and the Q-distribution is no longer a flat line. This reflects the fact that taking wrong actions in these regimes could lead to bad such as tripping, yielding a lower reward. First we compute each action dimension's Q distribution, Q L / Q i, and compare those distributions to that of the top MDP for the full action dimentionality, Q U. A figure containing these distributions and corresponding state visualization can be found in Figure B.3.For most states in the hopper walk cycle, the Q distribution is very flat. This implies that small changes in the action taken in a state will have little impact on future reward. This makes sense as the system can recover from any action taken in this frame. However, this is not true for all statescertain critical states exist, such as when the hopper is pushing off, where not selecting the correct action value greatly degrades performance. This can be seen in frame 466.Our algorithm is trained with a number of soft constraints. First, if fully converged, we would expect Q i−1 >= Q i as every new sub-action taken should maintain or improve the expected future discounted reward. Additionally, we would expect Q N (s, a) = Q U (s, a) (from equation 6). In the majority of frames these properties seem correct, but there is certainly room for improvement. Next, we attempt to look at Q surfaces in a more global manner. We plot 2D cross sections for each pair of actions and assume the third dimension is zero. Figure B.4 shows the . As seen in the previous visualization, the surface of both the sequential Q surface and the Q U is not smooth, which is expected as the environment action space for Hopper is expected to be highly non-linear. Some regions of the surface seem quite noisy which is not expected. Interestingly though, these regions of noise do not seem to lower the performance of the final policy. In Q-learning, only the maximum Q value regions have any impact on the taken policy. Future work is needed to better characterize this effect. We would like to explore techniques that use "soft"; BID28; BID14. These techniques will use more of the Q surface thus smooth the representations. Additionally, we notice that the dimensions of the autoregressive model are modeled differently. The last action, a 3 has considerably more noise than the previous two action dimensions. This large difference in the smoothness and shape of the surfaces demonstrates that the order of the actions dimensions matters. This figure suggests that the model has a harder time learning sharp features in the a 1 dimension. In future work, we would like to explore learned orderings, or bidirectional models, to combat this. Finally, the form of Q U is extremely noisy and has many cube artifacts. The input of this function is both a one hot quantized action, as well as the floating point representation. It appears the model uses the quantization as its main feature and learns a sharp Q surface. The most sensitive hyper parameters were the learning rate of the two networks, reward scale, and finally, discount factor. Parameters such as batch size, quantization amounts, and network sizes mattered to a lesser extent. We found it best to have exploration be done on less than 10% of the transitions. We didn't see any particular value below this that gave better performance. In our experiments, the structure of model used also played a large impact in performance, such as, using tied versus untied weights for each sequential prediction model. In future work, we would like to lower the amount of hyper parameters needed in these algorithms and study the effects of various hyper parameters more thoroughly. In this model, we looked at a number of configurations. Of the greatest importance is the form of the model itself. We looked at an LSTM parameterization as well as an untied weight variant. The untied weight variant's parameter search is listed below. To compute Q i we first take the state and previous actions and do one fully connected layer of size "embedding size". We then concatenate these representations and feed them into a 2 hidden layer MLP with "hidden size" units. The output of this network is "quantization bins" wide with no activation. return a' == 'A method to do Q-learning on continuous action spaces by predicting a sequence of discretized 1-D actions.' != 'We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.'

Question: The gap between the empirical success of deep learning and the lack of strong theoretical guarantees calls for studying simpler models. By observing that a ReLU neuron is a product of a linear function with a gate (the latter determines whether the neuron is active or not), where both share a jointly trained weight vector, we propose to decouple the two. We introduce GaLU networks — networks in which each neuron is a product of a Linear Unit, defined by a weight vector which is being trained, with a Gate, defined by a different weight vector which is not being trained. Generally speaking, given a base model and a simpler version of it, the two parameters that determine the quality of the simpler version are whether its practical performance is close enough to the base model and whether it is easier to analyze it theoretically. We show that GaLU networks perform similarly to ReLU networks on standard datasets and we initiate a study of their theoretical properties, demonstrating that they are indeed easier to analyze. We believe that further research of GaLU networks may be fruitful for the development of a theory of deep learning. An artificial neuron with the ReLU activation function is the function f w (x): R d → R such that f w (x) = max{x w, 0} = 1 x w≥0 · x w.The latter formulation demonstrates that the parameter vector w has a dual role; it acts both as a filter or a gate that decides if the neuron is active or not, and as linear weights that control the value of the neuron if it is active. We introduce an alternative neuron, called Gated Linear Unit or GaLU for short, which decouples between those roles. A 0 − 1 GaLU neuron is a function g w,u (x): R d → R such that g w,u (x) = 1 x u≥0 · x w. GaLU neurons, and therefore GaLU networks, are at least as expressive as their ReLU counterparts, since f w = g w,w. On the other hand, GaLU networks appear problematic from an optimization perspective, because the parameter u cannot be trained using gradient based optimization (since ∇ u g w,u (x) is always zero). In other words, training GaLU networks with gradient based algorithms is equivalent to initializing the vector u and keeping it constant thereafter. A more general definition of a GaLU network is given in section 2.The main claim of the paper is that GaLU networks are on one hand as effective as ReLU networks on real world datasets (section 3) while on the other hand they are easier to analyze and understand (section 4). Many recent works attempt to understand deep learning by considering simpler models, that would allow theoretical analysis while preserving some of the properties of networks of practical utility. Our model is most closely related to two such proposals: linear networks and non-linear networks in which only the readout layer is being trained. Deep linear networks is a popular model for analysis that lead to impressive theoretical (e.g. BID14 ; BID4 ; BID7). Linear networks are useful in order to understand how well gradient-based optimization algorithms work on non-convex problems. The weakness of linear network is that their expressive power is very limited: linear networks can only express linear functions. It means that their usefulness to understand the practical success of standard networks is somewhat limited. Training only the readout layer is an alternative attempt to understand deep learning through simpler models, that also gave theoretical interesting (e.g. BID13 BID8 ; BID2). The idea is that all the layers but the last one implement a non-linear constant transformation, and the last layer is learning a linear function on top of this transformation. The weakness of this model is that there is a big practical difference between training all the layers of a network and training only the last one. Our model is similar in certain aspects to both of those models, but it enjoys a much better practical utility than either one. See section 3 for an empirical comparison. Recall the definition of a basic GaLU neuron given in equation 1. We consider a more general GaLU neuron of the form g w,u,σ (x) = σ(x u) · x w for some non-linear scalar function σ: R → R. If σ is differentiable, we could train the vectors u with gradient based algorithms, but the focus of this paper is on untrained gates. That is, we assume that the vectors {u} are kept to their initial values throughout the optimization procedure and only the linear part of the GaLU neurons is being optimized. GaLU networks with a single hidden layer have the following property: for any given example, the values of the gates in the network remain constant. In networks with more than one hidden layer this not true. Consider a standard fully connected feed-forward network, let x be the input to the network and let x, x,... be the inputs to intermediate layers of the network. The output of a GaLU neuron at layer i will be σ(x (i−1) u) · x (i−1) w. So while the filter parameter vector, u,is not optimized upon, the value of the gate, σ(x (i−1) u), can change as x (i−1) changes. This adds an additional complication to the dynamics of the optimization that we wish to avoid. An alternative way to define a GaLU neuron at layer i is σ(x u) · x (i−1) w. In that case, the value of the gate is determined by the original input, and only the linear part depends on the output of the previous layer of the network. We call such a neuron a GaLU0 neuron, and a GaLU0 network is a network where all the neurons are GaLU0 neurons. In GaLU0 networks the gate values remain constant along the training, producing simpler dynamics. In order to check the hypothesis that effectiveness of ReLU networks stems mostly from the ability to train the linear part of the neurons, and not the gate part, we tested both GaLU0 1 and ReLU networks on the standard MNIST and Fashion-MNIST BID17 datasets. For both, we used PCA to reduce the input dimension to 64, and then trained a two hidden layers fully-conneted networks on them, with k hidden neurons at each hidden layer. FIG0 summarizes the , showing that GaLU0 and ReLU achieve similar , both outperforming linear networks of the same size. Training only the readout layer of a ReLU network gave much poorer (which were omitted from the graphs for clarity). All models were trained using the same architectures: two fully connected hidden layers with k neurons. The input dimension was reduced to 64 with PCA.Consider a GaLU network with a single hidden layer of k neurons: DISPLAYFORM0 A convenient property of a GaLU neuron is that it is linear in the weights w j, hence, α j g wj,uj (x) = g αj wj,uj (x). It means that the network can be rewritten as DISPLAYFORM1 withw j = α j w j. Because we want to optimize over the weights w 1,..., w k, α 1,..., α k, we might as well optimize over the reparameterizationw 1,...,w k without losing expressive power. It means that in a GaLU network of this form, it is sufficient to train the first layer of the network, as the readout layer adds nothing to the expressiveness of the network. The previous term can be further simplified: DISPLAYFORM2 So it turns out that a GaLU network is nothing more than a random non-linear transformation Φ u: R d → R kd and then a linear function. There are different notions for the expressivity of a model, and one of the simplest ones is the finitesample expressivity over a random sample. This notion fits well to our model, because we are not interested in the absolute expressivity of a GaLU network, but of the expressivity of a GaLU network with random filters. So the question is how well does a randomly-initialized network can fit a random sample. Note that given the constant filters, solving for the best weights is a convex problem. Hence, there is no "expressivity -optimization gap" in GaLU networks -every expressivity is immediately also an optimization . DISPLAYFORM0 and y 1,..., y m ∼ N, all of which are independent. Clearly, it is impossible to generalize from the sample to unseen examples; the best possible test loss is 1, and is achieved by the constant prediction 0. However, it is an interesting problem because it allows us to measure the expressivity of GaLU networks, by showing how much overfit we can expect from the network for a non-adversarial sample. Equivalently, it tells us how well the network can perform memorization tasks, where the only solution is to memorize the entire sample. We train the network for the standard mean-squareerror regression loss. Because the network is simply linear function over a constant non-linear transformation, and because we use the MSE loss, there is a closed form solution to the optimization problem DISPLAYFORM1 withX + being a pseudo-inverse ofX. This gives us DISPLAYFORM2 be arbitrary vectors. DefineX as above. Let y 1,..., y m ∼ N be independent random normal variables. Define the expected squared loss on the training set, for weights w, as L S (w). Then, DISPLAYFORM3 Proof Every vector y = (y 1, . . ., y m) ∈ R m can be decomposed to a sum y = a + b where a is in the span of the columns ofX and b is in the null space ofX. It follows that min w L S (w) = b 2 /m. The claim follows because if y ∼ N (0, I m) then the expected value of b 2 is m − rank X.It is always true that rank(X) ≤ min{m, kd}. Empirical experimentation shows that if x 1,..., x m, u 1,..., u k ∼ N (0, I d) then with high probability rank(X) = min{m, kd}. The fact that the GaLU network turned out to be only a linear function on top of a non-linear transformation seems to be a peculiar mathematical accident, with little relevance to standard networks. So we empirically tested the behavior of both ReLU and GaLU networks on the above model. It turns out that ReLU outperforms GaLU by a small margin -it is never better than GaLU with double the number of neurons, and is often worse than that. ReLU can outperform GaLU, even though it is less expressive, because we don't train the value of the the filters u 1,..., u k at all for the GaLU networks. It turns out that SGD over a ReLU network converges to better filters than a simple random initialization. One way to measure how much better those filters are is by trying to improve the initial filters of the GaLU network by randomly replacing them. Consider for example the simple algorithm given in algorithm 1.Running this algorithm improves the of the GaLU networks, making them more competitive with the ReLU ones. FIG1 summarizes our . An important fact about artificial neural networks is that they have small generalization error in many real-life problems. Otherwise they wouldn't be very useful as a learning algorithm. ZhangAlgorithm 1 Improve GaLU filters Input: A sample S, number of neurons k, number of iterations n. Initialize u 1, u 2,..., u k randomly. Find an optimal solution w 1,..., w k. for i = 1 to n do Pick j ∼ Uniform {1, 2, ..., k}. Pickũ j randomly. Find an optimal solution for a GaLU network with filters u 1,..., u j−1,ũ j, u j+1,..., u k. If the new solution is better than the current one, update u j =ũ j. end for One of the main experiments they run was to train the network over a sample with randomized labels, and to observe that the network still achieved small training loss (but large test loss, naturally). So any generalization bound that can be applied to the randomized sample is necessarily too weak to explain the generalization of the natural sample. As our goal is to show that GaLU networks exhibit similar phenomena as ReLU networks, but may be easier to analyze, we first construct a similar experiment to that of BID18 and compare the performance of GaLU and ReLU networks. Consider the following natural model. Let c 1,..., c n ∼ N (0, I d) be n clusters centers, each one with a random labels b 1,..., b n. A data point (x, y) is generated by picking a random index i ∼ Uniform {1, 2, . . ., n}, and setting x = c i + ξ for ξ ∼ N (0, σ We fixed the number of samples m = 1000, the input dimension d = 30, the number of clusters n = 30, the number of hidden neurons k = 30 and σ x = 0.1. We calculated the train and test errors for different values of σ y and p and for a GaLU and ReLU networks. The are summarized in FIG3 . We can clearly see that GaLU and ReLU have similar statistical behavior, and that while the train error is always small, as the labels become noisier the generalization error increases. This matches the spirit of experiments reported in BID18 . Next, we turn to an analysis of this phenomenon. Since one hidden layer GaLU networks can be cast as linear predictors, we can rely on classic norm-based generalization bounds for linear predictors. In particular, for p ∈ {1, 2}, consider the class of linear predictors H p = {x → x w : w p ≤ B p}. BID15. This also induces an upper bound on the gap between the test and train loss (see again BID15 for Lipschitz loss functions and see BID16 for the relation between Rademacher complexity and the generalization of smooth losses such as the squared loss). The question is whether the 1 / 2 norm of w is correlated with the amount of noise in the data. To study this, we depict the gap between train and test error as a function of the norm of w for GaLU networks. As can be seen in FIG5, for both the 1 and 2 norm, there is a clear linear relation between w 2 p and the generalization gap. While the constants are far from what the bounds state, the linear correlation is very clear. Note that figure 4 deals with GaLU networks that were trained as linear functions (by using the closed form solution for the MSE loss), and indeed shows that such network with such training behave as the theory states for linear predictors. We do not get the same behavior when we (unnecessarily) train both layers of the network using SGD. This matches the discussion in Section 5 of BID18, where the correlation between the 2 norm of the weights in a ReLU network and the test loss is discussed, and it is argued that there are more factors that affect the generalization properties. Indeed, many followup works show different capacity measures that may be more adequate for studying the generalization of deep learning (See for example). We next show a rather different analysis for a particular instance of linear regression. Consider a simple linear regression using the MSE, and denote the train and test loss by DISPLAYFORM0 Given a training set S, the MSE estimator is defined as w(S):= arg min w L S (w).We start with the following lemma. Lemma 1 (Follows from Corollary 2 of BID12) For a scalar σ ≥ 0 and a vector β ∈ R d, let D σ,β be the distribution over R d × R which is defined by the following generative DISPLAYFORM1 This lemma provides a complete analysis for the following experiment, which is similar to the experiments reported by BID18. We compare two distributions, the first is D σ,β for some vector β ∈ R d and for σ being close to 0, and the second is D 1,0. Note that the first distribution corresponds to a case in which we would like to be able to generalize, while the second distribution corresponds to a case in which we are fitting random noise and do not expect to generalize. We set the training set size to be m = d + 2 and we analyze the MSE estimator, w(S). As the lemma shows, the expected training losses on the first and second distributions are DISPLAYFORM2 respectively. Hence, the training loss should be small on both of the distributions. In contrast, the expected test loss on the first distribution is DISPLAYFORM3 while the expected test loss on the second distribution is DISPLAYFORM4 We see that while the train loss can be small on both distributions, in the test loss we see a big gap between the first distribution (assuming σ 1/ √ d) and the second distribution of purely random labels. This is exactly the type of phenomenon reported in BID18 -a sample with a small amount of noise achieves both small train and test losses, but a sample with random labels achieves a small train loss but a large test loss. Note that this is a natural property of the least squares solution, without any explicit regularization, picking a minimal-norm solution or using a specific algorithm for solving the problem. Lemma 1 gives us a very sharp analysis of linear regression. Unfortunately, the assumptions of Lemma 1 (which are based on the assumptions of Corollary 2 in BID12) are too strong -we need that m > d + 1 and that the instances will be generated based on a Gaussian distribution. While BID12 also includes asymptotic that are applicable for a larger set of distributions, we leave the application of them to GaLU networks for future work. In the analysis of the R d → R case we used the fact that a GaLU neuron g w,u is linear in the parameter w, and it allowed us to rephrase the problem as a convex problem. In the R d → R d case the situation is not as simple. In this case, every hidden neuron has d outgoing edges, and so we cannot use the same reparametrization trick as before. Even so, the output of a GaLU neuron is still linear in the parameter w. It means that for convex loss functions, finding the optimal weights for the first layer, keeping the weights of the second one constant, is a convex problem. The same doesn't hold for ReLU networks. Finding the optimal weights for the second layer, keeping the weights of the first one constant, is also a convex problem. Even more specifically, the optimization problem over the two layers is biconvex (see BID3 for a survey). So instead of applying SGD, we can apply biconvex optimization algorithms, such as Alternate Convex Search (ACS). In the case of the MSE loss, there is a closed form solution for each step of ACS, and using it outperforms SGD for small enough samples 2. Even though it is of limited practical use, this algorithm might be interesting for the derivation of theoretical bounds for such networks. In addition, it turns out that as we increase the output dimension d, GaLU and ReLU networks becomes more similar. In section 4.1.1 we measured the difference between ReLU and GaLU for the problem where all the variables are i.i.d. N, and it turned out that ReLU outperforms GaLU to a small extent. We repeated this experiment with larger d, and saw that the difference between the two vanished quickly (see FIG6). ber of neurons k such that a one hidden layer network achieves MSE< 0.3 on the random regression problem. As the output dimension d grows, more neurons are needed. As demonstrated in figure 2, GaLU networks needs more neurons than ReLU networks for output dimension d = 1. For larger d GaLU is slightly better, but it is clear that the two networks exhibit very similar behavior. We used fixed sample size (m = 1024) and input dimension (d = 32) in the generation of this graph. The standard paradigm in deep learning is to use neurons of the form σ x w for some differentiable non linear function σ: R → R. In this article we proposed a different kind of neurons, σ i,j · x w, where σ i,j is some function of the example and the neuron index that remains constant along the training. Those networks achieve similar to those of their standard counterparts, and they are easier to analyze and understand. To the extent that our arguments are convincing, it gives new directions for further research. Better understanding of the one hidden layer case (from section 5) seems feasible. And as GaLU and ReLU networks behave identically for this problem, it gives us reasons to hope that understanding the behavior of GaLU networks would also explain ReLU networks and maybe other non-linearities as well. As for deeper network, it is also not beyond hope that GaLU0 networks would allow some better theoretical analysis than what we have so far.

represent the short summary for finding the corresponding scientific article (long).

We propose Gated Linear Unit networks — a model that performs similarly to ReLU networks on real data while being much easier to analyze theoretically.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Machine learning systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from a different distribution from the one used for training. With their growing use in critical applications, it becomes important to develop systems that are able to accurately quantify its predictive uncertainty and screen out these anomalous inputs. However, unlike standard learning tasks, there is currently no well established guiding principle for designing architectures that can accurately quantify uncertainty. Moreover, commonly used OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples. To address these problems, we first seek to identify guiding principles for designing uncertainty-aware architectures, by proposing Neural Architecture Distribution Search (NADS). Unlike standard neural architecture search methods which seek for a single best performing architecture, NADS searches for a distribution of architectures that perform well on a given task, allowing us to identify building blocks common among all uncertainty aware architectures. With this formulation, we are able to optimize a stochastic outlier detection objective and construct an ensemble of models to perform OoD detection. We perform multiple OoD detection experiments and observe that our NADS performs favorably compared to state-of-the-art OoD detection methods. Detecting anomalous data is crucial for safely applying machine learning in autonomous systems for critical applications and for AI safety . Such anomalous data can come in settings such as in autonomous driving ), disease monitoring , and fault detection (b). In these situations, it is important for these systems to reliably detect abnormal inputs so that their occurrence can be overseen by a human, or the system can proceed using a more conservative policy. The widespread use of deep learning models within these autonomous systems have aggravated this issue. Despite having high performance in many predictive tasks, deep networks tend to give high confidence predictions on Out-of-Distribution (OoD) data . Moreover, commonly used OoD detection approaches are prone to errors and even assign higher likelihoods to samples from other datasets . Unlike common machine learning tasks such as image classification, segmentation, and speech recognition, there are currently no well established guidelines for designing architectures that can accurately screen out OoD data and quantify its uncertainty. Such a gap in our knowledge makes Neural Architecture Search (NAS) a promising option to explore the better design of uncertaintyaware models . NAS algorithms attempt to find an optimal neural network architecture for a specific task. Existing efforts have primarily focused on searching for architectures that perform well on image classification or segmentation. However, it is unclear whether architecture components that are beneficial for image classification and segmentation models would also lead to better uncertainty quantification and thereafter be effective for OoD detection. Moreover, previous work on deep uncertainty quantification shows that ensembles can help calibrate OoD classifier based methods, as well as improve OoD detection performance of likelihood estimation models . Because of this, instead of a single best performing architecture for uncertainty awareness, one might consider a distribution of wellperforming architectures. Along this direction, designing an optimization objective which leads to uncertainty-aware models is also not straightforward. With no access to labels, unsupervised/self-supervised generative models which maximize the likelihood of in-distribution data become the primary tools for uncertainty quantification (a). However, these models counter-intuitively assign high likelihoods to OoD data (a; ; a; Shafaei et al.). Because of this, maximizing the log-likelihood is inadequate for OoD detection. On the other hand, proposed using the Widely Applicable Information Criterion (WAIC) , a penalized log-likelihood score, as the OoD detection criterion. However, the score was approximated using an ensemble of models that was trained on maximizing the likelihood and did not directly optimize the WAIC score. To this end, we propose a novel Neural Architecture Distribution Search (NADS) framework to identify common building blocks that naturally incorporate model uncertainty quantification and compose good OoD detection models. NADS is an architecture search method designed to search for a distribution of well-performing architectures, instead of a single best architecture by formulating the architecture search problem as a stochastic optimization problem. Using NADS, we optimize the WAIC score of the architecture distribution, a score that was shown to be robust towards model uncertainty. Such an optimization problem with a stochastic objective over a probability distribution of architectures is unamenable to traditional NAS optimization strategies. We make this optimization problem tractable by taking advantage of weight sharing between different architectures, as well as through a parameterization of the architecture distribution, which allows for a continuous relaxation of the discrete search problem. Using the learned posterior architecture distribution, we construct a Bayesian ensemble of deep models to perform OoD detection. Finally, we perform multiple OoD detection experiments to show the efficacy of our proposed method. Neural Architecture Search (NAS) algorithms aim to automatically discover an optimal neural network architecture instead of using a hand-crafted one for a specific task. Previous work on NAS has achieved successes in image classification , image segmentation , object detection , structured prediction , and generative adversarial networks . However, there has been no NAS algorithm developed for uncertainty quantificaton and OoD detection. NAS consists of three components: the proxy task, the search space, and the optimization algorithm. Prior work in specifying the search space either searches for an entire architecture directly, or searches for small cells and arrange them in a pre-defined way. Optimization algorithms that have been used for NAS include reinforcement learning (; ;), Bayesian optimization , random search , Monte Carlo tree search , and gradient-based optimization methods (b;). To efficiently evaluate the performance of discovered architectures and guide the search, the design of the proxy task is critical. Existing proxy tasks include leveraging shared parameters , predicting performance using a surrogate model (a), and early stopping ). To our best knowledge, all existing NAS algorithms seek a single best performing architecture. In comparison, searching for a distribution of architectures allows us to analyze the common building blocks that all of the candidate architectures have. Moreover, this technique can also complement ensemble methods by creating a more diverse set of models for the ensemble decision, an important ingredient for deep uncertainty quantification . Prior work on uncertainty quantification and OoD detection for deep models can be divided into model-dependent (; ;), and model-independent techniques (; ;). Model-dependent techniques aim to yield confidence measures p(y|x) for a model's prediction y when given input data x. However, a limitation of model-dependent OoD detection is that they may discard information regarding the data distribution p(x) when learning the task specific model p(y|x). This could happen when certain features of the data are irrelevant for the predictive task, causing information loss regarding the data distribution p(x). Moreover, existing methods to calibrate model uncertainty estimates assume access to OoD data during training (; b). Although the OoD data may not come from the testing distribution, this assumes that the structure of OoD data is known ahead of time, which can be incorrect in settings such as active/online learning where new training distributions are regularly encountered. On the other hand, model-independent techniques seek to estimate the likelihood of the data distribution p(x). These techniques include Variational Autoencoders (VAEs) , generative adversarial networks (GANs) , autoregressive models , and invertible flow-based models . Among these techniques, invertible models offer exact computation of the data likelihood, making them attractive for likelihood estimation. Moreover, they do not require OoD samples during training, making them applicable to any OoD detection scenario. Thus in this paper, we focus on searching for invertible flow-based architectures, though the presented techniques are also applicable to other likelihood estimation models. Along this direction, recent work has discovered that likelihood-based models can assign higher likelihoods to OoD data compared to in-distribution data (a;) (see Figure 13 for an example). One hypothesis for such a phenomenon is that most data points lie within the typical set of a distribution, instead of the region of high likelihood (b). Thus, Nalisnick et al. (2019b) recommend to estimate the entropy using multiple data samples to screen out OoD data instead of using the likelihood. Other uncertainty quantification formulations can also be related to entropy estimation . However, it is not always realistic to test multiple data points in practical data streams, as testing data often come one sample at a time and are never well-organized into in-distribution or out-of-distribution groups. With this in mind, model ensembling becomes a natural consideration to formulate entropy estimation. Instead of averaging the entropy over multiple data points, model ensembles produce multiple estimates of the data likelihood, thus "augmenting" one data point into as many data points as needed to reliably estimate the entropy. However, care must be taken to ensure that the model ensemble produces likelihood estimates that agree with one another on in-distribution data, while also being diverse enough to discriminate OoD data likelihoods. In what follows, we propose NADS as a method that can identify distributions of architectures for uncertainty quantification. Using a loss function that accounts for the diversity of architectures within the distribution, NADS allows us to construct an ensemble of models that can reliably detect OoD data. Putting Neural Architecture Distribution Search (NADS) under a common NAS framework , we break down our search formulation into three main components: the proxy task, the search space, and the optimization method. Specifying these components for NADS with the ultimate goal of uncertainty quantification for OoD detection is not immediately obvious. For example, naively using data likelihood maximization as a proxy task would run into the issue pointed out by Nalisnick et al. (2019a), with models assigning higher likelihoods to OoD data. On the other hand, the search space needs to be large enough to include a diverse range of architectures, yet still allowing a search algorithm to traverse it efficiently. In the following sections, we motivate our decision on these three choices and describe these components for NADS in detail. The first component of NADS is the training objective that guides the neural architecture search. Different from existing NAS methods, our aim is to derive an ensemble of deep models to improve model uncertainty quantification and OoD detection. To this end, instead of searching for architectures which maximize the likelihood of in-distribution data, which may cause our model to incorrectly assign high likelihoods to OoD data, we instead seek architectures that can perform entropy estimation by maximizing the Widely Applicable Information Criteria (WAIC) of the training data. The WAIC score is a Bayesian adjusted metric to calculate the marginal likelihood . This metric has been shown by to be robust towards the pitfall causing likelihood estimation models to assign high likelihoods to OoD data. The score is defined as follows: Here, E[·] and V[·] denote expectation and variance respectively, which are taken over all architectures α sampled from the posterior architecture distribution p(α). Such a strategy captures model uncertainty in a Bayesian fashion, improving OoD detection. Intuitively, minimizing the variance of training data likelihoods allows its likelihood distribution to remain tight which, by proxy, minimizes the overlap of in-distribution and out-of-distribution likelihoods, thus making them separable. Under this objective function, we search for an optimal distribution of network architectures p(α) by deriving the corresponding parameters that characterize p(α). Because the score requires aggregating the from multiple architectures α, optimizing such a score using existing search methods can be intractable, as they typically only consider a single architecture at a time. Later, we will show how to circumvent this problem in our optimization formulation. NADS constructs a layer-wise search space with a pre-defined macro-architecture, where each layer can have a different architecture component. Such a search space has been studied by (; b;), where it shows to be both expressive and scalable/efficient. The macro-architecture closely follows the Glow architecture presented in. Here, each layer consists of an actnorm, an invertible 1 × 1 convolution, and an affine coupling layer. Instead of pre-defining the affine coupling layer, we allow it to be optimized by our architecture search. The search space can be viewed in Figure 1. Here, each operational block of the affine coupling layer is selected from a list of candidate operations that include 3 × 3 average pooling, 3 × 3 max pooling, skip-connections, 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated convolutions, identity, and zero. We choose this search space to answer the following questions towards better architectures for OoD detection: • What topology of connections between layers is best for uncertainty quantification? Traditional likelihood estimation architectures focus only on feedforward connections without adding any skip-connection structures. However, adding skip-connections may improve optimization speed and stability. • Are more features/filters better for OoD detection? More feature outputs of each layer should lead to a more expressive model. However, if many of those features are redundant, it may slow down learning, overfitting nuisances and ing in sub-optimal models. • Which operations are best for OoD detection? Intuitively, operations such as max/average pooling should not be preferred, as they discard information of the original data point "too aggressively". However, this intuition remains to be confirmed. Having specified our proxy task and search space, we now describe our optimization method for NADS. Several difficulties arise when attempting to optimize this setup. First, optimizing p(α), a distribution over high-dimensional discrete random variables α, jointly with the network parameters is intractable as, at worst, each network's optimal parameters would need to be individually identified. Second, even if we relax the discrete search space, the objective function involves computing an expectation and variance over all possible discrete architectures. To alleviate these problems, we first introduce a continuous relaxation for the discrete search space, allowing us to approximately optimize the discrete architectures through backpropagation and weight sharing between common architecture blocks. We then approximate the stochastic objective by using Monte Carlo samples to estimate the expectation and variance. Specifically, let A denote our discrete architecture search space and α ∈ A be an architecture in this space. Let l θ * (α) be the loss function of architecture α with its parameters set to θ * such that it satisfies θ * = arg min θ l(θ|α) for some loss function l(·). We are interested in finding a distribution p φ (α) parameterized by φ that minimizes the expected loss of an architecture α sampled from it. We denote this loss function as Solving L(φ) for arbitrary parameterizations of p φ (α) can be intractable, as the inner loss function l θ * (α) involves searching for the optimal parameters θ * of a neural network architecture α. Moreover, the outer expectation causes backpropagation to be inapplicable due to the discrete random architecture variable α. We adopt a tractable optimization paradigm to circumvent this problem through a specific reparameterization of the architecture distribution p φ (α), allowing us to backpropagate through the outer expectation and jointly optimize φ and θ. For clarity of exposition, we first focus on sampling an architecture with a single hidden layer. In this setting, we intend to find a probability vector φ = [φ 1, . . ., φ K] with which we randomly pick a single operation from a list of the random categorical indicator vector sampled from φ, where b i is 1 if the i th operation is chosen, and zero otherwise. Note that b is equivalent to the discrete architecture variable α in this setting. With this, we can write the random output y of the hidden layer given input x as To make optimization tractable, we relax the discrete mask b to be a continuous random variableb using the Gumbel-Softmax reparameterization as follows: Here, g 1... g k ∼ − log(− log(u)) where u ∼ Unif, and τ is a temperature parameter. For low values of τ,b approaches a sample of a categorical random variable, recovering the original discrete problem. While for high values,b will equally weigh the K operations . Using this, we can compute backpropagation by approximating the gradient of the discrete architecture α with the gradient of the continuously relaxed categorical random variableb, as With this backpropagation gradient defined, generalizing the above setting to architectures with multiple layers simply involves recursively applying the above gradient relaxation to each layer. With this formulation, we can gradually remove the continuous relaxation and sample discrete architectures by annealing the temperature parameter τ. With this, we are able to optimize the architecture distribution p φ (α) and sample candidate architectures for further retraining, finetuning, or evaluation. By sampling M architectures from the distribution, we are able to approximate the WAIC score expectation and variance terms as: Figure 2: Summary of our architecture search findings: the most likely architecture structure for each block K found by NADS. We applied our architecture search on five datasets: CelebA (Liu et al.), CIFAR-10, CIFAR-100, , SVHN , and MNIST (LeCun). In all experiments, we used the Adam optimizer with a fixed learning rate of 1 × 10 −5 with a batch size of 4 for 10000 iterations. We approximate the WAIC score using M = 4 architecture samples, and set the temperature parameter τ = 1.5. The number of layers and latent dimensions is the same as in the original Glow architecture , with 4 blocks and 32 flows per block. Images were resized to 64 × 64 as inputs to the model. With this setup, we found that we are able to identify neural architectures in less than 1 GPU day. Our findings are summarized in Figure 2, while more samples from our architecture search can be seen in Appendix C. Observing the most likely architecture components found on all of the datasets, a number of notable observations can be made: • The first few layers have a simple feedforward structure, with either only a few convolutional operations or average pooling operations. On the other hand, more complicated structures with skip connections are preferred in the deeper layers of the network. We hypothesize that in the first few layers, simple feature extractors are sufficient to represent the data well. Indeed, recent work on analyzing neural networks for image data have shown that the first few layers have filters that are very similar to SIFT features or wavelet bases . • The max pooling operation is almost never selected by the architecture search. This confirms our hypothesis that operations that discard information about the data is unsuitable for OoD detection. However, to our surprise, average pooling is preferred in the first layers of the network. We hypothesize that average pooling has a less severe effect in discarding information, as it can be thought of as a convolutional filter with uniform weights. • The deeper layers prefer a more complicated structure, with some components recovering the skip connection structure of ResNets . We hypothesize that deeper layers may require more skip connections in order to feed a strong signal for the first few layers. This increases the speed and stability of training. Moreover, a larger number of features can be extracted using the more complicated architecture. Interestingly enough, we found that the architectures that we sample from our NADS perform well in image generation without further retraining, as shown in Appendix D. Using the architectures sampled from our search, we create a Bayesian ensemble of models to estimate the WAIC score. Each model of our ensemble is weighted according to its probability, as in. The log-likelihood estimate as well as the variance of this model ensemble Here,'Baseline' denotes the method proposed by. Subcaptions denote training-testing set pairs. Additional figures are provided in Appendix G. is given as follows: Intuitively, we are weighing each member of the ensemble by their posterior architecture distribution p φ (α), a measure of how likely each architecture is in optimizing the WAIC score. We note that for our setup, V[log p αi (x)] is zero for each model in our ensemble; however, for models which do have variance estimates, such as models that incorporate variational dropout; ), this term may be nonzero. Using these estimates, we are able to approximate the WAIC score in equation. We trained our proposed method on 4 datasets: CIFAR-10, CIFAR-100 , SVHN , and MNIST (LeCun). In all experiments, we randomly sampled an , and Outlier Exposure (OE) (b ensemble of M = 5 models from the posterior architecture distribution p φ * (α) found by NADS. Although these models can sufficiently perform image synthesis without retraining as shown in Appendix D, we observed that further retraining these architectures led to a significant improvement in OoD detection. Because of this, we retrained each architecture on data likelihood maximization for 150000 iterations using Adam with a learning rate of 1 × 10 −5. We first show the effects of increasing the ensemble size in Figure 3 and Appendix F. Here, we can see that increasing the ensemble size causes the OoD WAIC scores to decrease as their corresponding histograms shift away from the training data WAIC scores, thus improving OoD detection performance. Next, we compare our ensemble search method against a traditional ensembling method that uses a single Glow architecture trained with multiple random initializations. As shown in Table 2, we find that our method is superior compared to the traditional ensembling method when compared on OoD detection using CIFAR-10 as the training distribution. We then compared our NADS ensemble OoD detection method for screening out samples from datasets that the original model was not trained on. For SVHN, we used the Texture, Places, LSUN, and CIFAR-10 as the OoD dataset. For CIFAR-10 and CIFAR-100, we used the SVHN, Texture, Places, LSUN, CIFAR-100 (CIFAR-10 for CIFAR-100) datasets, as well as the Gaussian and Rademacher distributions as the OoD dataset. Finally, for MNIST, we used the not-MNIST, F-MNIST, and K-MNIST datasets. We compared our method against a baseline method that uses maximum softmax probability (MSP) , as well as two popular OoD detection methods: ODIN and Outlier Exposure (OE) (b). ODIN attempts to calibrate the uncertainty estimates of an existing model by reweighing its output softmax score using a temperature parameter and through random perturbations of the input data. For this, we use DenseNet as the base model as described in . On the other hand, OE models are trained to minimize a loss regularized by an outlier exposure loss term, a loss term that requires access to OoD samples. As shown in Table 1 and Table 3, our method outperforms the baseline MSP and ODIN significantly while performing better or comparably with OE, which requires OoD data during training, albeit not from the testing distribution. We plot Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves in Figure 4 and Appendix G for more comprehensive comparison. In particular, our method consistently achieves high area under PR curve (AUPR%), showing that we are especially capable of screening out OoD data in settings where their occurrence is rare. Such a feature is important in situations where anomalies are sparse, yet have disastrous consequences. Notably, ODIN underperforms in screening out many OoD datasets, despite being able to reach the original reported performance when testing on LSUN using a CIFAR10 trained model. This suggests that ODIN may not be stable for use on different anomalous distributions. Unlike NAS for common learning tasks, specifying a model and an objective to optimize for uncertainty estimation and outlier detection is not straightforward. Moreover, using a single model may not be sufficient to accurately quantify uncertainty and successfully screen out OoD data. We developed a novel neural architecture distribution search (NADS) formulation to identify a random ensemble of architectures that perform well on a given task. Instead of seeking to maximize the likelihood of in-distribution data which may cause OoD samples to be mistakenly given a higher likelihood, we developed a search algorithm to optimize the WAIC score, a Bayesian adjusted estimation of the data entropy. Using this formulation, we have identified several key features that make up good uncertainty quantification architectures, namely a simple structure in the shallower layers, use of information preserving operations, and a larger, more expressive structure with skip connections for deeper layers to ensure optimization stability. Using the architecture distribution learned by NADS, we then constructed an ensemble of models to estimate the data entropy using the WAIC score. We demonstrated the superiority of our method to existing OoD detection methods and showed that our method has highly competitive performance without requiring access to OoD samples. Overall, NADS as a new uncertainty-aware architecture search strategy enables model uncertainty quantification that is critical for more robust and generalizable deep learning, a crucial step in safely applying deep learning to healthcare, autonomous driving, and disaster response. A FIXED MODEL ABLATION STUDY

We propose an architecture search method to identify a distribution of architectures and use it to construct a Bayesian ensemble for outlier detection.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We propose a framework to learn confidence-calibrated networks by designing a novel loss function that incorporates predictive uncertainty estimated through stochastic inferences.