Question: represent the scientific article to find a short tldr

Question: Specifying reward functions is difficult, which motivates the area of reward inference: learning rewards from human behavior. The starting assumption in the area is that human behavior is optimal given the desired reward function, but in reality people have many different forms of irrationality, from noise to myopia to risk aversion and beyond. This fact seems like it will be strictly harmful to reward inference: it is already hard to infer the reward from rational behavior, and noise and systematic biases make actions have less direct of a relationship to the reward. Our insight in this work is that, contrary to expectations, irrationality can actually help rather than hinder reward inference. For some types and amounts of irrationality, the expert now produces more varied policies compared to rational behavior, which help disambiguate among different reward parameters -- those that otherwise correspond to the same rational behavior. We put this to the test in a systematic analysis of the effect of irrationality on reward inference. We start by covering the space of irrationalities as deviations from the Bellman update, simulate expert behavior, and measure the accuracy of inference to contrast the different types and study the gains and losses. We provide a mutual information-based analysis of our findings, and wrap up by discussing the need to accurately model irrationality, as well as to what extent we might expect (or be able to train) real people to exhibit helpful irrationalities when teaching rewards to learners. The application of reinforcement learning (RL) in increasingly complex environments has been most successful for problems that are already represented by a specified reward function (; ; . Unfortunately, not only do real-world tasks usually lack an explicit exogenously-specified reward function, but attempting to specify one tends to lead to unexpected side-effects as the agent is faced with new situations . This has motivated the area of reward inference: the process of estimating a reward function from human inputs. The inputs are traditionally demonstrations, leading to inverse reinforcement learning (IRL) or inverse optimal control (IOC) (; ; ;). Recent work has expanded the range of inputs significantly,to comparisons (; ;), natural language instructions , physical corrections , proxy rewards ), or scalar reward values . The central assumption behind these methods is that human behavior is rational, i.e. optimal with respect to the desired reward (cumulative, in expectation). Unfortunately, decades of research in behavioral economics and cognitive science has unearthed a deluge of irrationalities, i.e. of ways in which people deviate from optimal decision making: hyperbolic discounting, scope insensitivity, optimism bias, decision noise, certainty effects, loss aversion, status quo bias, etc. Work on reward inference has predominantly used one model of irrationality: decision-making noise, where the probability of an action relates to the value that action has. The most widely used model by far is a Bolzmann distribution stemming from the Luce-Sherpard rule (; ;) and the principle of maximum (causal) entropy in (;, which we will refer to as Bolzmann-rationality . Recent work has started to incorporate systematic biases though, like risk-aversion , having the wrong dynamics belief , and myopia and hyperbolic discounting . Learning from irrational experts feels like daunting task: reward inference is already hard with rational behavior, but now a learner needs to make sense of behavior that is noisy or systematically biased. Our goal in this work is to characterize just how muddied the waters are -how (and how much) do different irrationalities affect reward inference? Our insight is that, contrary to expectations, irrationality can actually help, rather than hinder, reward inference. Our explanation is that how good reward inference is depends on the mutual information between the policies produced by the expert and the reward parameters to be inferred. While it is often possible for two reward parameters to produce the same rational behavior, irrationalities can sometimes produce different behaviors that disambiguate between those same two reward parameters. For instance, noise can help when it is related to the value function, as Boltzmann noise is, because it distinguishes the difference in values even when the optimal action stays the same. Optimism can be helpful because the expert takes fewer risk-avoiding actions and acts more directly on their goal. Overall, we contribute 1) an analysis and comparison of the effects of different biases on reward inference testing our insight, 2) a way to systematically formalize and cover the space of irrationalities in order to conduct such an analysis, and 3) evidence for the importance of assuming the right type of irrationality during inference. Our good news is that irrationalities can indeed be an ally for inference. Of course, this is not always true -the details of which irrationality type and how much of it also matter. We see these as opening the door to a better understanding of reward inference, as well as to practical ways of making inference easier by asking for the right kind of expert demonstrations -after all, in some cases it might be easier for people to act optimistically or myopically than to act rationally. Our reinforce that optimal teaching is different from optimal doing, but point out that some forms of teaching might actually be easier than doing. Our goal is to explore the effect irrationalities have on reward inference if the learner knows about them -we explore the need for the learner to accurately model irrationalities in section 4.2. While ideally we would recruit human subjects with different irrationalities and measure how well we can learn rewards, this is prohibitive because we do not get to dictate someone's irrationality type: people exhibit a mix of them, some yet to be discovered. Further, measuring accuracy of inference is complicated by the fact that we do not have ground truth access to the desired reward: the learner can measure agreement with some test set, but the test set itself is produced subject to the same irrationalities that produced the training data. As experimenters, we would remain deluded about the human's true intentions and preferences. To address this issue, we simulate expert behavior subject to different irrationalities based on ground truth reward functions, run reward inference, and measure the performance against the ground truth, i.e. the accuracy of a Bayesian posterior on the reward function given the (simulated) expert's inputs. There are many possible irrationalities that people exhibit , far more than what we could study in one paper. They come with varying degrees of mathematical formalization and replication across human studies. To provide good coverage of this space, we start from the Bellman update, and systematically manipulate its terms and operators to produce a variety of different irrationalities that deviate from the optimal MDP policy in complementary ways. For instance, operating on the discount factor can model more myopic behavior, while operating on the transition function can model optimism or the illusion of control. Figure 1 summarizes our approach, which we detail below. Figure 1: We modify the components of the Bellman update to cover different types of irrationalities: changing the max into a softmax to capture noise, changing the transition function to capture optimism/pessimism or the illusion of control, changing the reward values to capture the nonlinear perception of gains and losses (prospect theory), changing the average reward over time into a maximum (extremal), and changing the discounting to capture more myopic decision-making. The rational expert does value iteration using the Bellman update from figure 1. Our models change this update to produce different types of non-rational behavior. Boltzmann-rationality modifies the maximum over actions max a with a Boltzmann operator with parameter β: ) This models that people will not be perfect, but rather noisily pick actions in a way that is related to the Qvalue of those actions. The constant β is called the rationality constant, because as β → ∞, the human choices approach perfect rationality (optimality), whereas β = 0 produces uniformly random choices. This is the standard assumption for reward inference that does not assume perfect rationality, because it easily transforms the rationality assumption into a probability distribution over actions, enabling learners to make sense of imperfect demonstrations that otherwise do not match up with any reward parameters. Our next set of irrationalities manipulate the transition function away from reality. Illusion of Control. Humans often overestimate their ability to control random events. To model this, we consider experts that use the Bellman update: where T n (s |s, a) ∝ (T (s |s, a)) n. As n → ∞, the demonstrator acts as if it exists in a deterministic environment. As n → 0, the expert acts as if it had an equal chance of transitioning to every possible successor state. At n = 1, the expert is the rational expert. Optimism/Pessimism. Humans tend to systematically overestimate their chance experiencing of positive over negative events. We model this using experts that modify the probability they get outcomes based on the value of those outcomes: where T 1/τ (s |s, a) ∝ T (s |s, a)e (r(s,a,s)+γVi(s))/τ. 1/τ controls how pessimistic or optimistic the expert is. As 1/τ → +∞, the expert becomes increasingly certain that good transitions will happen. As 1/τ → −∞, the expert becomes increasingly certain that bad transitions will happen. As 1/τ → 0, the expert approaches the rational expert. Next, we consider experts that use the modified Bellman update: where f: R → R is some scalar function. This is equivalent to solving the MDP with reward f • r. This allows us to model human behavior such as loss aversion and scope insensitivity. inspires us to consider a particular family of reward transforms: c controls how loss averse the expert is. As c → ∞, the expert primarily focuses on avoiding negative rewards. As c → 0, the expert focuses on maximizing positive rewards and 2.2.5 MODIFYING THE SUM BETWEEN REWARD AND FUTURE VALUE: EXTREMAL Extremal. Humans seem to exhibit duration neglect, sometimes only caring about the maximum intensity of an experiennce . We model this using experts that use the Bellman step: These experts maximize the expected maximum reward along a trajectory, instead of the expected sum of rewards. As α → 1, the expert maximizes the expected maximum reward they achieve along their full trajectory. As α → 0, the expert becomes greedy, and only cares about the reward they achieve in the next timestep. Myopic Discount. In practice, humans are often myopic, only considering immediate rewards. One way to model this is to decrease gamma in the Bellman update. At γ = 1, this is the rational expert. As γ → 0, the expert becomes greedy and only acts to maximize immediate reward. Myopic VI. As another way to model human myopia, we consider a expert that performs only h steps of Bellman updates. That is, this expert cares equally about rewards for horizon h, and discount to 0 reward after that. As h → ∞, this expert becomes rational. If h = 1, this expert only cares about the immediate reward. Hyperbolic Discounting. Human also exhibit hyperbolic discounting, with a high discount rate for the immediate future and a low discount rate for the far future. formulate this as the following Bellman update: k modulates how much the expert prefers rewards now versus the future. As k → 0, this expert becomes the rational expert. 3.1 EXPERIMENTAL DESIGN Simulation Environment. To reduce possible confounding from our choice of environment, we used a small 5x5 gridworld where the irrationalities nonetheless cause experts to exhibit different behavior. Our gridworld consists of three types of cells: ice, holes, and rewards. The expert can start in any ice cell. At each ice cell, the expert can move in one of the four cardinal directions. With Figure 2: The log loss (lower = better) of the posterior as a function of the parameter we vary for each irrationality type. These six irrationalities all have parameter settings that outperform rational experts. For the models that interpolate to rational expert, we denote the value that is closest to rational using a dashed vertical line. probability 0.8, they will go in that direction. With probability 0.2, they will instead go in one of the two adjacent directions. Holes and rewards are terminal states, and return the expert back to their start state. They receive a penalty of −10 for falling into a hole and θ i ∈ for entering into the ith reward cell. Dependent Measures. To separate the inference difficulty caused by suboptimal inference from the difficulty caused by expert irrationality, we perform the exact Bayesian update on the trajectory θ , which gives us the posterior on θ given ξ: We use two metrics to measure the difficulty of inference The first is the expected log loss of this posterior, or negative log-likelihood: A low log loss implies that we are assigning a high likelihood to the true θ. As we are performing exact Bayesian inference with the true model P (ξ|θ) and prior P (θ), the log loss is equal to the entropy of the posterior H(θ|ξ). The second metric is the L 2 -distance between the mean posterior θ and the actual theta: The closer the inferred posterior mean of θ is to the actual value θ *, the lower the loss. For each irrationality type, we calculate the performance of reward inference on trajectories of a fixed length T, with respect to the two metrics above. To sample a trajectory of length T from a expert, we fix θ * and start state s. Then, we perform the expert's (possibly modified) Bellman updates until convergence to recover the policy π θ *. Finally, we generate rollouts starting from state s until T state, action pairs have been sampled from π θ *. Figure 3: A best case analysis for each irrationality type: the log loss/L 2 distance from mean (lower=better) for experts, as a function of the length of trajectory observed. Each irrationality uses the parameter value that is most informative. As discussed in section 3.2, different irrationality types have different slopes and converge to different values. In addition, the best performing irrationality type according to log loss is not the best performing type according to L 2 loss. Impact of Each Irrationality. We found that of the 8 irrationalities we studied, 6 had parameter settings that lead to lower log loss than the rational expert. We report how the parameter influences the log loss for each of these experts in figure 2. 1 For T = 30, Optimism with 1/τ = 3.16 performed the best, followed by Boltzmann with β = 100 and Hyperbolic with k = 0.1. Both forms of Myopia also outperformed the rational expert, with best performance occurring at γ = 0.9 and h = 5. Finally, the Extremal expert also slightly outperformed the rational expert, with best performance at α = 0.9. Notably, in every case, neither the most irrational expert nor the perfectly rational expert was the most informative. Impact of Data for Different Irrationalities. Next, we investigate how the quality of inference varies as we increase the length of the observed trajectory T. We report our for the best performing parameter for each irrationality type in figure 3. Interestingly, while both metrics decrease monotonically regardless of irrationality type, the rate at which they decrease differs by the irrationality type, and the best performing irrationality type according to log loss (Optimism) is not the best performing type according to L 2 distance (Boltzmann). What is behind these differences? To explain these , we use the notion of mutual information I(X; Y) between two variables, defined as: The mutual information measures how much our uncertainty about X decreases by observing Y. For reward inference, the term we care about is the mutual information between the expert's trajectory and the reward parameters The mutual information I(θ; ξ) is equal to a constant minus the posterior log loss under the true model. A expert with mutual information will cause the learner to have a lower posterior log loss., 4): when the reward is sufficiently large, the expert becomes convinced that no action it takes will lead to the reward, leading it to perform random actions. Figure 5: (a) Boltzmann-rationality produces different policies for θ * = vs. θ * =: when ||θ|| is larger, the policy becomes closer to that of the rational expert. (b) A Myopic expert produces different policies for θ * = vs. θ * =: while the rational expert always detours around the hole and attempts to reach the larger reward, myopia causes the myopic expert to go for the smaller source of reward when it is non-zero. By the information processing inequality, we have the bound I(θ; ξ) ≤ I(θ; π). To have higher mutual information, different θs should be mapped to different policies πs. Indeed, we found that the experts that were able to outperform the rational expert were able to disambiguate between θs that the rational expert could not. To visualize this, we show examples of how the policy of several irrational experts differ when the rational expert's policies are identical in figures 4 and 5. We plot the correlation between I(θ; ξ) and I(θ; π) in figure 6. Experts that have more informative policies tend to have more informative trajectories, but the correlation is not perfect. Notably, the Optimism expert has the most informative trajectories of length 30, but has less informative policies than the Boltzmann expert. In the limit of infinite data from every state, we would have I(θ; ξ) → I(θ; π). However, as each trajectory begins from the same start state, and not every state is reachable with every policy, the bound is not achievable in general, even if we observe an arbitrarily large number of trajectories. This highlights the need for off-policy data in reward inference tasks. We show that, contrary to what we might expect, suboptimal experts can actually help an agent learn the reward function. Optimism bias, myopia (via heavier discounting or hyperbolic discounting), Figure 6: The informativeness of policies correlates with the informativeness of trajectories of length 30, as discussed in section 3.2 and noise via Boltzmann rationality were the most informative irrationalities in our environments, far surpassing the performance of the rational expert for their ideal settings. Our contribution overall was to identify a systematic set of irrationalities by looking at deviations in the terms of the Bellman update, and show that being irrational is not automatically harmful to inference by quantifying and comparing the inference performance for these different types. Estimating expert irrationality. One major limitation of our work is that our findings hold for when the learner knows the type and parameter value of the irrationality. In practice, reward inference will require solving the difficult task of estimating the irrationality type and degree . We still need to quantify to what extent these still hold given uncertainty about the irrationality model. It does, however, seem crucial to reward inference that learners do reason explicitly about irrationality -not only is the learner unable to take advantage of the irrationality to make better inference if it does not model it, but actually reward inference in general suffers tremendously if the learner assumes the wrong type. In figure 10 in the Appendix, we compare inference with the true model vs. with assuming a Boltzmann model as default. The are quite striking: not knowing the irrationality harms inference tremendously. Whether irrationalities help, this means that it is really important to model them. Generalization to other environments. A second limitation of our work is that we only tested these models in a limited range of environments. Further work is needed to test generalization of our findings across different MDPs of interest. Our analysis of mutual information lends credence to the Boltzmann rationality generalizing well: these policies are much more varied with the reward parameters. In contrast, how useful the optimism bias is depends on the task: if we know about what to avoid already, as was the case for our learner, the bias is useful; if, on the other hand, we would know the goal but do not know what to avoid, the bias can hinder inference. Overall, this paper merely points out that there is a lot of richness to the ways in which these biases affect inference, and provides a quantitative comparison for a starting domain -much more is needed to gain a deeper understanding of this phenomenon. Applications to real humans. A third limitation is that we do not know where real humans lie. Do they have the helpful irrationality types? Do they fall in the range of parameters for these types that help inference? And what happens when types combine? While these questions are daunting, there is also a hidden opportunity here: what if we could influence humans to exhibit helpful types of irrationality? It might be much easier for them, for instance, to act myopically than to act rationally. In the end, reward inference is the confluence of two factors: how well the robot learns, and how well the teacher teaches. Our point out that it might be easier than previously thought to be a good teacher -even easier than being a rational expert., 1.78, 3.16, 5.62, 10, 17.8, 31.6, 56.2, 100, 178, 316, 562, 1000,1780, 3160, 5620, To enable exact inference, we discretized θ, using 5 evenly spaced points for each θ i. Our specific grid is included in figures 4 and 5 As there are two reward cells, this gives us 25 possible distinct reward parameters. We assumed a uniform prior on the reward parameter. We list the parameter values we search over for each policy in table 1. Except for myopic γ and myopic h, we use γ = 0.99. For myopic h, we use γ = 1. From each start state, we sample 10 trajectories of each length for each reward parameter, policy combination. We include the plots for the log loss of trajectories from the Prospect Theory and Illusion of Control experts in 7 In addition, we include the plots for the L 2 loss for all 8 irrationalities in figures 8 and figure 9. Given that several types of irrationality can help inference when the form of irrationality is known, a natural question to ask is how important is it to known the irrationality exactly. To investigate this, we plot the log loss of the posterior of a learner who falsely assumes that the expert is Boltzmann- Figure 8: The L 2 distance (lower = better) of posterior mean of θ to the true θ *,s as a function of the parameter we vary for each irrationality type. These six irrationalities all have parameter settings that outperform rational experts. For the models that interpolate to rational expert, we denote the value that is closest to rational using a dashed vertical line. A comparison of reward inference using a correct model of the irrationality type, versus always using a Boltzman model. (Lower log loss = better.) The inference impairment from using the misspecified irrationality model (Boltzmann) greatly outweighs the variation in inference performance caused by the various irrationality types themselves. Hence, compared to using a misspecified model of irrationality, expert irrationality is not in itself a major impairment to reward inference, and sometimes expert irrationality can even helps when a model of the irrationality is known. rational with β = 100. Where applicable, the log loss is averaged over possible hyperparameter settings for the expert. We report the in figure 10. The log loss of the posterior if we wrongly imagine the expert is Boltzmann-rational far outweighs differences between particular irrationality types. Fundamentally, misspecification is bad for inference because different experts might exhibit the same action only under different reward parameters. For example, consider figure the case where the actual expert is myopic, with small n. Then the myopic agent might go toward a closer reward even if it is much smaller, as shown in figure 11. This would cause the learner to falsely infer that the closer reward is quite large, leading to a posterior with extremely high log loss when the reward is actually smaller. Figure 11: An example of why assuming Boltzmann is bad for a myopic agent -the Boltzmann rational agent would take this trajectory only if the reward at the bottom was not much less than the reward at the top. The myopic agent with n ≤ 4, however, only "sees" the reward at the bottom. Consequently, inferring the preferences of the myopic agent as if it were Boltzmann leads to poor performance in this case.

represent the tldr for finding the corresponding scientific article
For example, 'We suggest the sufficient number of bits for representing weights of DNNs and the optimum bits are conservative when solving real problems.' should be close to 'Most deep neural networks (DNNs) require complex models to achieve high performance. Parameter quantization is widely used for reducing the implementation complexities. Previous studies on quantization were mostly based on extensive simulation using training data. We choose a different approach and attempt to measure the per-parameter capacity of DNN models and interpret the to obtain insights on optimum quantization of parameters. This research uses artificially generated data and generic forms of fully connected DNNs, convolutional neural networks, and recurrent neural networks. We conduct memorization and classification tests to study the effects of the number and precision of the parameters on the performance. The model and the per-parameter capacities are assessed by measuring the mutual information between the input and the classified output. We also extend the memorization capacity measurement to image classification and language modeling tasks. To get insight for parameter quantization when performing real tasks, the training and test performances are compared. Deep neural networks (DNNs) have achieved impressive performance on various machine learning tasks. Several DNN architectures are known, and the most famous ones are fully connected DNNs (FCDNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs).It is known that neural networks do not need full floating-point precision for inference BID10 BID16 BID23. A 32-bit floating-point parameter can be reduced to 8-bit, 4-bit, 2-bit, or 1-bit, but this can incur performance degradation. Therefore, precision should be optimized, which is primarily conducted by extensive computer simulations using training data. This not only takes much time for optimization but also can incorrectly predict the performance in real environments when the characteristics of input data are different from the training data. In this study, we attempt to measure the capacity of DNNs, including FCDNN, CNN, and RNN, using a memorization and classification task that applies random binary input data. The per-parameter capacities of various models are estimated by measuring the mutual information between the input data and the classification output. Then, the fixed-point performances of the models are measured to determine the relation between the quantization sensitivity and the per-parameter capacity. The memorization capacity analysis are extended to real models for performing image classification and language modeling, by which the parameter quantization sensitivity is compared between memorization and generalization tasks. The contributions of this paper are as follows.• We experimentally measure the memorization capacity of DNNs and estimate the perparameter capacity. The capacity per parameter is between 2.3 bits to 3.7 bits, according to the network structure, which is FCDNN, CNN, or RNN. The value is fairly independent of the model size.• We show that the performance of the quantized networks is closely related to the capacity per parameter, and FCDNNs show the most resilient quantization performance while RNNs suffer most from parameter quantization. The network size hardly effects the quantization performance when DNN models are trained to use full capacity.• We explain that severe quantization, such as binary or ternary weights, can be employed without much performance degradation when the networks are in the over-parameter region.• We suggest the sufficient number of bits for representing weights of neural networks, which are approximately 6 bits, 8 bits, and 10 bits for FCDNNs, CNNs, and RNNs, respectively. This estimate of the number of bits for implementing neural networks is very important considering that many accelerators are designed without any specific training data or applications.• The study with real-models shows that neural networks are more resilient to quantization when performing generalization tasks than conducting memorization. Thus, the optimum bits obtained with the memorization tasks are conservative and safe estimate when solving real problems. The paper is organized as follows. In Section 2, previous works on neural network capacity and fixedpoint optimization are briefly presented. Section 3 explains the capacity measurement methods for DNN models. Section 4 presents parameter capacity measurement for FCDNNs, CNNs, and RNNs. The quantization performances measured on DNNs are presented in Section 5. Concluding remarks follow in Section 6. The capacity of neural networks has been studied since the early days of DNN research. Although the capacity can be defined in many ways, it is related to the learnability of networks. The capacity of networks is shown as the number of uncorrelated random samples that can be memorized BID8. A single-layer perceptron with n parameters can memorize at least 2n random samples BID11. In other words, the network can always construct a hyperplane with n parameters that divides 2n samples. Additionally, the capacity of a three-layer perceptron is proportional to the number of parameters BID0. Recently, RNNs were trained with random data to measure the capacity per parameter BID6. Our study is strongly motivated by this research, and extends it to the quantization performance interpretation of generic DNN models, including FCDNN, CNN, and RNN. Recent studies have showed that neural networks have a generalization ability even if the expressive capacity of the model is sufficiently large BID33 BID2. In this paper, we also discuss the effect of network quantization when performing generalization tasks. Early works on neural network quantization usually employed 16-bit parameters obtained by directly quantizing the floating-point numbers BID10. Recently, a retraining technique was developed to improve the performance of quantized networks BID16 BID23. Retraining-based quantization was applied to CNN and RNN models, showing superior performance compared to directly quantized ones BID1 BID31. Many studies attempting extreme quantization have been published, such as 2-bit ternary BID16 BID21 BID35, 1-bit binary weight quantization, and XNOR networks BID7 BID28. Some aggressive model compression techniques also employed vector quantization or table look-up BID12 BID4. However, not all CNNs show the same quantization performance. For example, AlexNet BID20 shows almost the same performance with only 1-bit quantized parameters. However, the same quantization technique incurs a very severe performance loss when applied to ResNet BID28. A previous study shows that large sized networks are more resilient to severe quantization than smaller ones. Theoretical works and many practical implementation optimization techniques have been studied BID13 BID17 BID19 BID30 BID24. Recent work increases the number of network parameters to preserve the performance under low-precision quantization BID26. Our works are not targeted to a specific data or model, but introduce the general understanding of parameter quantization. We assess the network capacity of DNN models using a random data memorization and classification task BID6. In this task, N random binary vectors, X, are generated and each is randomly and uniformly assigned to the output label Y. The size of the binary vector depends on the DNN model. For FCDNN, the input X is a one dimensional vector whose size is determined by the hidden layer dimension. In CNN, the input needs to be a 2-D or 3-D tensor. Input samples of CNNs are generated by concatenating and reshaping random binary vectors. During the training process, the DNN is trained to correctly predict the label, which is 0 or 1, of the random input X. As the number of input data size, N, increases, the classification accuracy drops because of the limited memorization capacity. Note that the accuracy for the memorization task refers to the training performance after convergence because there is no proper test dataset for random training samples. The capacity is measured using the mutual information, defined as a measure of the amount of information that one random variable contains about another random variable (Cover & BID9 . The mutual information of a trained network with N input samples is calculated as follows: DISPLAYFORM0 where p is the mean classification accuracy for all samples under trained parameter θ. The network capacity is defined as DISPLAYFORM1 The accuracy, p, may vary depending on the training method of the model. We find N and p that maximize the mutual information of the networks by iteratively training the models. This optimization employs both grid search-and Bayesian optimization-based hyper-parameter tuning BID5 . The optimization procedure consists of three stages. First, we try to find the largest input data size whose accuracy is slightly lower than 1. Second, we perform a grid search to determine the boundary values of the hyper-parameters. The searched hyper-parameters can include initialization, optimizer, initial learning rate, learning rate decay factor, batch size, and optimizer variables. Finally, we conduct hyper-parameter tuning within the search space using Scikit-learn library BID27 . We add the number of training samples N as a hyper-parameter and use the mutual information of Eq. FORMULA0 as the metric for the optimization. Quantization of model parameters perturbs the trained network, therefore, fixed-point training or retraining with full-precision backpropagation is usually needed BID16 BID21 BID7 BID34 . However, the performance of the quantized networks does not always meet that of the floating-point models, even after retraining. This suggests that model capacity is reduced by quantization, especially when the number of bits used is very small. In this research, we observe the memorization capacity degradation caused by quantization in generic FCDNN, CNN, and RNN models. The uniform quantization is used for the sake of convenient arithmetic, and the same step size is assigned to each layer in the FCDNN, each kernel in the CNN, or each weight matrix in the LSTM layer. The bias values are not quantized, because they have a large dynamic range. It is important to note that the weights connected to the output are not quantized, because their optimum bit-widths depend on the number of labels in the output. Quantization is performed from floating-point to 8-bit, 6-bit, 5-bit, 4-bit, 3-bit, and 2-bit precision, in sequence. Retraining is performed after every quantization, but requires only a small number of epochs, because only fine-tuning is needed BID16 . We compare the generalization performance of floating-point and fixed-point DNNs by visualizing the loss surface. Loss is measured by applying Gaussian random noise to the parameters of the trained network as shown in Eq.. DISPLAYFORM0 Here, L(θ) is the loss according to the network parameters. The distribution of weights may vary depending on the model size and learning method. We apply the normalized filter noise to the θ noise for fair comparison on different models BID22.We employ two real networks, one is for image classification with CIFAR-10 dataset and the other is language modeling with Penn Tree Bank (PTB) dataset. One large and one small model are trained for these networks. We quantize those networks with the precision of 8, 6, 4, and 2 bits and analyze the variation of the surface according to the precision. θ noise is added to the quantized parameters. In order to reduce the error due to randomness of noise, all loss values are measured with 10 different trials and the average values are plotted. The capacities of FCDNNs, CNNs, and RNNs are measured via the memorization task explained in Section 3.1. The models used for the test employ floating-point parameters. The training data for FCDNNs is a 1-D vector of size n in. N input data are used as for the training data. The output, Y, is the randomly assigned label, either 0 or 1, for each input. Thus, inputs, X and Y, are represented as X ∈ {0, 1} N ×nin and Y ∈ {0, 1} N, respectively. The input data dimension, n in, should be larger than log 2 N so that no overlapped data is contained among N input data. In the experiments for FCDNNs, the input vector dimension, n in, is chosen to be equal to the number of units in the hidden layer. We conduct experiments for FCDNNs with hidden layer dimensions of 32, 64, 128, and 256, and with hidden layer depths of 1, 2, 3, and 4. The initialization method chosen is the'He' initialization BID14 and gradients are updated following the rule in SGD, with momentum, which shows the best performance in our grid search. The initial learning rate for hyper parameter tuning is chosen between 0.001 and 0.05 on the log scale. The decay factor and momentum are set to have even distance values in the linear scale between 0.1 and 0.5 and between 0.6 and 0.99, respectively. For each model, experiments are conducted to measure the accuracy of memorization while increasing the size of the input data, N. Note that only the training error is measured in this memorization task, because there is no unseen data. Experimental are based upon the best accuracy obtained when attempted with different hyper parameters. The capacity of the model is estimated according to Eq., where p is the training accuracy. The experimentally obtained memorization capacities of the FCDNN models are presented in FIG1, where depths of 1, 2, 3, and 4, and widths of 32, 64, 128, and 256 are used. When the number of hidden layers is the same, the amount of data that can be almost perfectly memorized quadruples when the dimension of the hidden layer is doubled. This means that the memorization capacity is linearly proportional to the number of parameters. Similarly, the FCDNN models with 2, 3, or 4 hidden layer depths can memorize 2, 3, or 4 times the input data as compared to the single layer DNN, respectively. FIG2 shows the memorization accuracy and the mutual information obtained using Eq. on the FCDNN. The model is composed of three layers and the hidden layer of size 64. Here, we find that the amount of mutual information steadily increases as the input data size grows. However, it begins to drop as the input size grows farther, and the memorization accuracy drops. By analyzing the accuracy trend of the model, it is possible to distinguish the input data size into three regions: the over-parameterized, the maximum performance, and the under-parameterized sections, as shown in FIG2. For example, if the model is trained to memorize only 10,000 data, it can be regarded as over-parameterized. The number of data that can be memorized by maximally utilizing all the parameters is between 30,000 and 40,000. In over-parameterized regions, performance can be maintained, even if the capacity of the networks is reduced. The per-parameter capacity of FCDNNs is shown in FIG2. Regardless of the width or depth, one parameter has a capacity of 1.7 to 2.5 bits, and FCDNNs have an average of 2.3-bit capacity per parameter. This is consistent with theoretical study BID11 BID0. The total capacity of the model may be interpreted as an optimal storage that can store a maximum of random binary samples BID11 BID3. The capacity of CNNs is also measured via a similar memorization task. CNNs can have a variety of structures according to the number of channels, the size of the kernels, and the number of layers. The kernel size of CNNs in this test are either (3 × 3) or (5 × 5), which are the same for all layers, the number of convolution layers from 3 to 9. The dimensions of the inputs are n height = n width = 32 and n channel = 1 for all experiments. Three max-pooling operations are applied to reduce the number of parameters in the fully connected layer. CNN structures used in our experiments are shown in Supplementary materials. The CNN models contain not only convolution layers but also fully connected layers. Thus, the per-parameter capacity for convolution layers is calculated after subtracting the capacity for fully connected layers from the measured total capacity. We assume the per-parameter capacity of the fully connected layer as 2.3 bits to calculate the capacity for convolution layers. As shown in FIG2, the convolution layers have the per-parameter capacity of between 2.86 and 3.09 except the smallest model, which is higher than that of FCDNNs. The average capacity per parameter of the tested models is 3.0 bits. Results show that the per-parameter capacity of CNNs is higher than that of FCDNNs, even when CNNs memorize uncorrelated data. Note that one parameter of FCDNNs is used only once for each inference. However, the parameter of CNNs is used multiple times. This parameter-sharing nature of CNNs seems to increase the amount of information that one parameter can store. It has been shown that the various structures of RNNs all have similar capacity per parameter of 3 to 6 bits BID6. We train RNNs with a dataset with no sequence correlation to show the capacity of the parameters. The random input dataset is composed of inputs, X ∈ {0, 1} N ×nseq×nin and labels Y ∈ {0, 1} N, which are uniformly set to 0 or 1. The training loss is calculated using the cross-entropy of the label at the output of the last step. We train RNNs with a single LSTM layer of 32-D. The input dimension, n in, is also 32-D and the amount of unrolling sequence, n seq, is five-step. It has been reported that unrolling of five-step almost saturates the performance in this setup BID6. We apply 5 input random vectors, X 0, X 1, X 2, X 3, and X 4, each with 32-D, and assign one label to this 160-D vector at the last time step. The error propagates from the last step only, and the outputs at intermediate time-steps are ignored. The number of parameters in the network is 8,386. In this case, the maximum mutual information is obtained when the number of samples is 32K, and the memorization accuracy is 99.52 %. Therefore, the per-parameter capacity of the model is 3.7 bits. The RNN shows higher per-parameter capacity than FCDNNs and CNNs. We have shown that FCDNNs, CNNs, and RNNs have different per-parameter capacities. According to the parameter-data ratio, a trained DNN can be an over-parameterized, max-capacity, or underparameterized model. Thus, we can assume that the DNN performance under quantization would depend on not only the network structure, such as FCDNN, CNN, or RNN, but also the parameter-data ratio. The experiments are divided into two cases. The first is to measure performance degradation via quantization precision when each model is in the maximum capacity region. The second analyzes performance when the models are in the over-parameterized region. When the FCDNN, CNN, and RNN are trained to have the maximum memorization capacity, the performances with parameter quantization are shown in Fig. 3(a). The FCDNN, CNN, and RNN models are shown. The fixed-point performances of two FCDNNs, two CNNs, and two RNNs are illustrated. With 6-bit parameter quantization, the FCDNN shows no accuracy drop. However, those for CNNs and RNNs are 5 % and 18 %, respectively. Because the RNN contains the largest amount of information at each parameter, the loss caused by parameter quantization seems to be the most severe. We also find that there is no decline in performance until the parameter precision is lowered to 6-bit for FCDNNs, 8-bit for CNNs, and 10-bit for RNNs, even when all models use full capacity. Next, we show the fixed-point performance of DNNs when they are trained to be in the overparameterized region. Note that the per-parameter capacity is lowered in the over-parameterized region. We conducted simulation with half size of the maximum number of data that can be memorized. For example, an FCDNN used for the measurement has 3 hidden layers with a hiddenlayer dimension of 128; the capacity of the corresponding model is about 2 17 bits. The network is FORMULA6 FC FORMULA0 FC FORMULA1 FC FC FORMULA0 FC FORMULA1 (a) LSTM FORMULA0 LSTM FORMULA1 LSTM FORMULA0 (c) over-parameterized when the number of memorized samples is 2 16. Fig. 3(b) shows that the FCDNN model memorizes all samples even with 4-bit parameter quantization when the model uses half of the capacity. Also, over-parmeterized model is less sensitive to bit-precision on CNNs and RNNs. The performances of fixed-point DNNs with the number of samples are shown in Fig. 4. The shows that DNNs are more robust when the networks are more over-parameterized.2 We have assessed the required precision of networks for performing memorization tasks. The memorization test only uses the training data that are artificially generated. However, most neural networks should conduct more than memorization because the test data are not seen during the training. In this section, we analyze the effects of network quantization for performing real tasks. We train two different sized CNN models with CIFAR-10 data. The structures of the two models are as follows: DISPLAYFORM0 The size of kernels of both models is (3×3), 16C represents a convolution layer with 16 channels and 128F C means a fully connected layer with 128-dimension. The number of parameters is 0.22M for the small model and 3.5M for the large one. Both models were trained with the same hyper-parameter setting. To analyze the impact of network quantization on the test performance, we plot the loss and accuracy surfaces of floating-point and quantized CNN models in Fig. 5. For simplicity, the of floatingpoint and 2-bit fixed-point CNN are given. Please refer to the Supplementary materials for other . When applying the training data that may have been memorized during the training phase, the large model shows indifferent performance surface regardless of the parameter precision. But, for the small model, the 2-bit model shows quite degraded performance when compared to the floating-point network. However, the test accuracy of small 2-bit model is not much lowered. We can notice that the loss surface of the 2-bit model shown in Fig. 5 is much wider than that of the floating-point model. This is consistent with recent studies on generalization BID18 BID22. This observation suggests that the quantized networks are more resilient when performing generalization tasks. Thus, the required precision of the network obtained with the memorization task can be considered a conservative estimate. Fig. 6 shows the training and test data based performance of fixed-point CNN and RNN on real data. T iny model has the same structure as the small model, but reducing the number of channels by half and the size of the fully connected layers by a quarter. The RNNs are trained for language modeling with Penn Treebank corpus BID25, and the models consists of two LSTM layers with the same dimension. Here, both for CNN and RNN models, we can confirm that large networks are more robust to quantization. Also, the networks need more parameter precision when conducting memorization tasks using the training set, rather than solving real problems using the test set. We have measured fixed-point DNN performance on real tasks and are shown in FIG7. FCDNN models are trained with MNIST dataset, ResNet BID15 models are trained using ILSVRC-2012 dataset BID29 and RNN based word-level language models (WLMs) are designed using PTB dataset. Experimented FCDNNs and RNNs are composed of two FC layers and two LSTM layers with the same dimension, respectively. 4-bit quantized FCDNNs shows almost same performance compared to the floating-point networks even when the number of neurons are only 8. Performances are preserved up to 6 bits on ResNets and RNN WLMs. Their resiliency to quantization increases as networks become larger. Quantization of parameters is a straightforward way of reducing the complexity of DNN implementations, especially when VLSI or special purpose neural processing engines are used. Our study employed simulations on varying sizes of generic forms of DNN models. Memorization tests using random binary input data were conducted to determine the total capacity by measuring the mutual information. Our simulation show that the per-parameter capacity is not sensitive to the model size, but is dependent on the structure of the network models, such as FCDNN, CNN, and RNN. The maximum per-parameter capacities of FCDNNs, CNNs, and RNNs are approximately 2.3 bits, 3.0 bits, and 3.7 bits per parameter. Thus, RNNs have the tendency of demanding more bits when compared to FCDNNs. We quantized DNNs under various capacity-utilization regions and showed that the capacity of parameters are preserved up to 6 bits, 8 bits, and 10 bits on FCDNNs, CNN, and RNNs, respectively. The performance of the quantized networks was also tested with image classification and language modeling tasks. The show that networks need more parameter precision when conducting memorization tasks, rather than inferencing with unseen data. Thus, the precision obtained through the memorization test can be considered a conservative estimate in implementing neural networks for solving real problems. This research not only gives valuable insights on the capacity of parameters but also provides practical strategies for training and optimizing DNN models. DISPLAYFORM0 In FCDNNs, each weight matrix, W l between two layers, demands |h l−1 | × |h l | weights, where |h| is the number of units for the layer, l. CNNs, which are popular for image processing, usually receive 2-dimensional (D) or 3-D input data, whose size is much larger than the filter or kernel size. The set of weights between layers is referred to as the'kernel' and the output is referred to as the'feature map'. Because the input size is usually much larger than the kernel size, the CNN parameters are reused many times. A kernel slides over the input feature map and produces an output feature map, and the sliding step is determined by the stride, s. The convolution weights is denoted as W l ∈ R k l,h ×k l,w ×n l−1 ×n l and feature map of the layer as C l ∈ R c l,h ×c l,w ×n l. k l,h and k l,w are height and width of each kernel and c l,h, c l,w are height and width of the feature map in layer l, respectively. n l is the number of feature map in the layer l. CNNs can have a variety of structures according to the number of channels, the size of the kernels, and the number of layers. We attempted to produce a general setting for CNNs. The experiments models are shown in Table. 1. We constructed CNNs to have twice the number of feature maps and half the height/width after pooling. Also, to minimize side-effects by fully connected layers, the output feature of the last convolution layer is flattened and directly propagated to the sof tmax layer. The capacity per parameter is measured only for parameters in convolution layers, by subtracting capacity of the fully connected layer. RNNs have a feedback structure that reflects the information in the previous steps when processing sequence data. RNNs are composed of one or multiple recurrent layers, and each layer computes the output, y t, and the hidden state, h t, using the previous hidden state, h t−1, and the input, x t.We use LSTM as the recurrent layers, showing stable performance in various applications. The mutual information equation of a network can be obtained as follows. We first re-write our random variables X, Y, andŶ. DISPLAYFORM0 DISPLAYFORM1 where f (θ, X i) is the predict of a network when the input is X i. Under our experimental setting, both X and Y have uniform random distribution. Note that X and Y are independent as well as Y i and Y j when i = j. Therefore, DISPLAYFORM2 And we use the network's average accuracy p as a probability of Y i =Ŷ i, so that DISPLAYFORM3 Finally, the equation is derived as: DISPLAYFORM4 The loss surfaces of fixed-point CNNs are shown in Fig. 8 and Fig. 9. The models are trained with CIFAR-10 dataset. The loss surfaces of RNN LMs for PTB dataset are also shown in FIG1 and FIG1. The performance degradation on test dataset is lower than the degradation on training dataset in all experiments.'

We find that irrationality from an expert demonstrator can help a learner infer their preferences.

represent the short summary for finding the corresponding scientific article (long).
------
E.g.:
Using the DSL grammar and reinforcement learning to improve synthesis of programs with complex control flow. == Program synthesis is the task of automatically generating a program consistent with a specification. Recent years have seen proposal of a number of neural approaches for program synthesis, many of which adopt a sequence generation paradigm similar to neural machine translation, in which sequence-to-sequence models are trained to maximize the likelihood of known reference programs. While achieving impressive , this strategy has two key limitations. First, it ignores Program Aliasing: the fact that many different programs may satisfy a given specification (especially with incomplete specifications such as a few input-output examples). By maximizing the likelihood of only a single reference program, it penalizes many semantically correct programs, which can adversely affect the synthesizer performance. Second, this strategy overlooks the fact that programs have a strict syntax that can be efficiently checked. To address the first limitation, we perform reinforcement learning on top of a supervised model with an objective that explicitly maximizes the likelihood of generating semantically correct programs. For addressing the second limitation, we introduce a training procedure that directly maximizes the probability of generating syntactically correct programs that fulfill the specification. We show that our contributions lead to improved accuracy of the models, especially in cases where the training data is limited. The task of program synthesis is to automatically generate a program that is consistent with a specification such as a set of input-output examples, and has been studied since the early days of Artificial Intelligence BID34. There has been a lot of recent progress made on neural program induction, where novel neural architectures inspired from computation modules such as RAM, stack, CPU, turing machines, and GPU BID10 BID17 BID20 BID11 BID31 BID18 have been proposed to train these architectures in an end-to-end fashion to mimic the behavior of the desired program. While these approaches have achieved impressive , they do not return explicit interpretable programs, tend not to generalize well on inputs of arbitrary length, and require a lot of examples and computation for learning each program. To mitigate some of these limitations, neural program synthesis approaches BID16 BID28 BID7 have been recently proposed that learn explicit programs in a Domain-specific language (DSL) from as few as five input-output examples. These approaches, instead of using a large number of input-output examples to learn a single program, learn a large number of different programs, each from just a few input-output examples. During training, the correct program is provided as reference, but at test time, the learnt model generates the program from only the input-output examples. While neural program synthesis techniques improve over program induction techniques in certain domains, they suffer from two key limitations. First, these approaches use supervised learning with reference programs and suffer from the problem of Program Aliasing: For a small number of input-output examples, there can be many programs that correctly transform inputs to outputs. The problem is the discrepancy between the single supervised reference program and the multitude of correct programs. FIG0 shows an example of this: if maximizing the probability of ground truth program, predicting Program B would be assigned a high loss even though the two programs are semantically equivalent for the input-output example. Maximum likelihood training forces the model to learn to predict ground truth programs, which is different from the true objective of program synthesis: predicting any consistent program. To address this problem, we alter the optimization objective: instead of maximum likelihood, we use policy gradient reinforcement learning to directly encourage generation of any program that is consistent with the given examples. The second limitation of neural program synthesis techniques based on sequence generation paradigm BID7 ) is that they often overlook the fact that programs have a strict syntax, which can be checked efficiently. Similarly to the work of BID28, we explore a method for leveraging the syntax of the programming language in order to aggressively prune the exponentially large search space of possible programs. In particular, not all sequences of tokens are valid programs and syntactically incorrect programs can be efficiently ignored both during training and at test time. A syntax checker is an additional form of supervision that may not always be present. To address this limitation, we introduce a neural architecture that retains the benefits of aggressive syntax pruning, even without assuming access to the definition of the grammar made in previous work BID28. This model is jointly conditioned on syntactic and program correctness, and can implicitly learn the syntax of the language while training. We demonstrate the efficacy of our approach by developing a neural program synthesis system for the Karel programming language BID29, an educational programming language, consiting of control flow constructs such as loops and conditionals, making it more complex than the domains tackled by previous neural program synthesis works. This paper makes the following key contributions:• We show that Reinforcement Learning can directly optimize for generating any consistent program and improves performance compared to pure supervised learning.• We introduce a method for pruning the space of possible programs using a syntax checker and show that explicit syntax checking helps generate better programs.• In the absence of a syntax checker, we introduce a model that jointly learns syntax and the production of correct programs. We demonstrate this model improves performance in instances with limited training data. Program synthesis is one of the fundamental problems in Artificial Intelligence. To the best of our knowledge, it can be traced back to the work of BID34 where a theorem prover was used to construct LISP programs based on a formal specification of the input-output relation. As formal specification is often as complex as writing the original program, many techniques were developed to achieve the same goal with simpler partial specifications in the form of input-output (IO) examples BID1 BID33. Rule-based synthesis approaches have recently been successful in delivering on the promise of Programming By Example BID24, the most widely known example being the FlashFill system BID13 in Excel. However, such systems are extremely complicated to extend and need significant development time from domain experts to provide the pruning rules for efficient search. As a , the use of Machine Learning methods have been proposed, based on Bayesian probabilistic models BID23 or Inductive Logic programming BID25 BID26 to automatically generate programs based on examples. Recently, inspired by the success of Neural Networks in other applications such as vision BID19 or speech recognition BID9 ) differentiable controllers were made to learn the behaviour of programs by using gradient descent over differentiable version of traditional programming concepts such as memory addressing BID10, manipulating stacks BID17 BID12, register machines BID20, and data manipulation BID27. These approaches to program induction however tend to struggle with generalization, especially when presented with inputs of a different dimension than the one they were trained with and require a very large amount of training data. Some exceptions to this include Neural Programmer Interpreters BID31 and its extensions BID5 ) that learn from program traces rather than only examples. However, they still learn a different model for each program and are computationally expensive, unlike our system that uses a single model for learning a large number of programs. A series of recent works aim to infer explicit program source code with the assumption that code structure provides an inductive bias for better generalization. In particular, explicitly modeling control flow statements such as conditionals and loops can often lead to programs capable of generalizing, regardless of input size BID8 BID4 BID32. One remaining drawback of these approaches is the need to restart learning from scratch for each new program. Thus they are largely unsuited for the situation of synthesizing a new program on-the-fly from very few examples. The latest developments use large datasets of artificially generated programs and learn to map embeddings of IO examples to information about the programs to generate. BID3 produce scores over attributes, to be used as heuristics to speed up search-based techniques. BID28 use their dataset to learn probability over the expansion rules of a predefined grammar, while BID7 directly predict the source code of the programs. These last two methods use supervised training to maximize the likelihood of a single reference program, while we directly optimize for the generation of any consistent program. Our approach to optimize program correctness is similar in spirit to advances in Neural Machine Translation BID37 BID30 that leverage reinforcement learning to optimize directly for evaluation metrics. Taking advantage of the fact that programs can be syntactically checked and unit tested against the specification examples, we show how to improve on those REINFORCE based methods. Recently, BID14 proposed a method similar to ours based on Maximum Marginal Likelihood to generate programs based on a description in natural language. From an application point of view, our target domain is more complex as our DSL includes control flow operations such as conditionals and loops. Moreover, natural language utterances fully describe the steps that the program needs to take, while learning from IO examples requires planning over potentially long executions. Their approach is more akin to inferring a formal specification based on a natural language description, as opposed to our generation of imperative programs. Incorporating knowledge of the grammar of the target domain to enforce syntactical correctness has already proven useful to model arithmetic expressions, molecules BID21, and programs BID28 BID38. These approaches define the model over the production rules of the grammar; we instead operate directly over the terminals of the grammar. This allows us to learn the grammar jointly with the model in the case where no formal grammar specification is available. Our approach is extremely general and can be applied to the very recently proposed methods for inferring and executing programs for visual reasoning BID16 that to the best of our knowledge does not directly explictly encourage grammar consistency of the ing program. Before describing our proposed methods, we establish the necessary notation, present the problem setting and describe the general paradigm of our approach. To avoid confusion with probabilities, we will use the letter λ to denote programs. I and O will be used to denote respectively input states and output states and we will use the shortcut IO to denote a pair of corresponding input/output examples. A state constitutes what the programs are going to be operating on, depending on the application domain. In FlashFill-type applications BID28 BID7, Input and Output states would be strings of characters, while in our Karel environment, states are grids describing the presence of objects. If we were to apply our method to actual programming languages, states would represent the content of the machine's registers and the memory. At training time, we assume to have access to N training samples, each training sample consisting of a set of K Input/Output states and a program implementing the mapping correctly: DISPLAYFORM0 where λ i (I k i) denotes the ing state of applying the program λ i to the input state I k i. Our goal is to learn a synthesizer σ that, given a set of input/output examples produces a program: DISPLAYFORM1 We evaluate the programs on a set of test cases for which we have both specification examples and held-out examples: DISPLAYFORM2 At test time, we evaluate the performance of our learned synthesizer by generating, for each sample in the test set, a programλ j. The metric we care about is Generalization: DISPLAYFORM3 3.2 NEURAL PROGRAM SYNTHESIS ARCHITECTURE Similar to BID7 we use a sequential LSTM-based BID15 ) language model, conditioned on an embedding of the input-output pairs. Each pair is encoded independently by a convolutional neural network (CNN) to generate a joint embedding. A succinct description of the architecture can be found in section 6.1 and the exact dimensions are available in the supplementary materials. where each token comes from an alphabet Σ. We model the program one token at a time using an LSTM. At each timestep, the input consists of the concatenation of the embedding of the IO pair and of the last predicted token. One such decoder LSTM is run for each of the IO pairs, all using the same weights. The probability of the next token is defined as the Softmax of a linear layer over the max-pooled hidden state of all the decoder LSTMs. A schema representing this architecture can be seen in FIG1.The form of the model that we are learning is: DISPLAYFORM0 At test time, the most likely programs are obtained by running a beam search. One of the advantages of program synthesis is the ability to execute hypothesized programs. Through execution, we remove syntactically incorrect programs and programs that are not consistent with the observed examples. Among the remaining programs, we return the most likely according to the model. Results of all the decoders are maxpooled and the prediction is modulated by the mask generated by the syntax model. The probability over the next token is then obtained by a Softmax transformation. To estimate the parameters θ of our model, the default solution is to perform supervised training, framing the problem as Maximum Likelihood estimation. BID7 follow this approach and use stochastic gradient descent to solve: DISPLAYFORM0 However, this training objective exhibits several drawbacks. First, at training time, the model is only exposed to the training data distribution, while at test time, it is fed back the token from its own previous predictions. This discrepancy in distribution of the inputs is well known in Natural Language Processing under the name of exposure bias BID30.Moreover, this loss does not represent the true objective of program synthesis. In practice, any equivalent program should be as valid a prediction as the reference one. This property, that we call program aliasing, is not taken into account by the MLE training. Ideally, we would like the model to learn to reason about the necessary steps needed to map the input to the output. As a , the loss shouldn't penalize correct programs, even if they do not correspond to the ground truth. The first modification that we propose is to change the target objective to bring it more in line with the goal of program synthesis. We replace the optimization problem of by DISPLAYFORM0 where R i (λ) is a reward function designed to encode the quality of the sampled programs. Note that this formulation is extremely generic and would allow to represent a wide range of different objective functions. If we assume that we have access to a simulator to run our programs, we can design R i so as to optimize for generalization on held-out examples, preventing the model to overfit on its inputs. Additional property such as program conciseness, or runtime efficiency could also be encoded into the reward. DISPLAYFORM1 Step Figure 3: Approximation using a beamsearch. All possibles next tokens are tried for each candidates, the S (here 3) most likely according to p θ are kept. When an End-Of-Sequence token (green) is reached, the candidate is held out. At the end, the most likely complete sequences are used to construct an approximate distribution, through rescaling. However, this expressiveness comes at a cost: the inner sum in FORMULA7 is over all possible programs and therefore is not tractable to compute. The standard method consists of approximating the objective by defining a Monte Carlo estimate of the expected reward, using S samples from the model. To perform optimization, an estimator of the gradient of the expected reward is built based on the REINFORCE trick BID35. DISPLAYFORM2 However, given that we sample from a unique model, there is a high chance that we will sample the same programs repeatedly when estimating the gradient. This is especially true when the model has been pre-trained in a supervised manner. A different approach is to approximate the distribution of the learned distribution by another one with a smaller support. To obtain this smaller distribution, one possible solution is to employ the S most likely samples as returned by a Beam Search. We generate the embedding of the IO grids and perform decoding, keeping at each step the S most likely candidates prefixes based on the probability p θ given by the model. At step t, we evaluate p θ (s 1 . . . s t, IO k i k=1..K) for all the possible next token s t and all the candidates (s 1 . . . s t−1) previously obtained. The S most likely sequences will be the candidates at the (t + 1) step. FIG2 represents this process. Based on the final samples obtained, we define a probability distribution to use as an approximation of p θ in. As opposed to, this approximation introduces a bias. It however has the advantage of aligning the training procedure more closely with the testing procedure where only likely samples are going to be decoded. Formally, this corresponds to performing the following approximation of the objective function, (BS(p θ, S) being the S samples returned by a beam search with beam size S): DISPLAYFORM3 where DISPLAYFORM4 With this approximation, the support of the distribution q θ is much smaller as it contains only the S elements returned by the beam search. As a , the sum become tractable and no estimator are needed. We can simply differentiate this new objective function to obtain the gradients of the loss with regards to p θ and use the chain-rule to subsequently obtain gradient with regards to θ necessary for the optimization. Note that if S is large enough to cover all possible programs, we recover the objective of.Based on this more tractable distribution, we can define more complex objective functions. In program synthesis, we have the possibility to prune out several predictions by using the specification. Therefore, we can choose to go beyond optimizing the expected reward when sampling a single program and optimize the expected reward when sampling a bag of C programs and keeping the best one. This in a new objective function: DISPLAYFORM5 where q θ is defined as previously. We argue that by optimizing this objective function, the model gets the capability of "hedging its bets" and assigning probability mass to several candidates programs, ing in a higher diversity of outputs. In the special case where the reward function only takes values in {0, 1}, as it is when we are using correctness as a reward, this can be more easily computed as: DISPLAYFORM6 The derivation leading to this formulation as well as a description on how to efficiently compute the more general loss can be found in appendix A. Note that although this formulation brings the training objective function closer to the testing procedure, it is still not ideal. It indeed makes the assumption that if we have a correct program in our bag of samples, we can identify it, ignoring the fact that it is possible to have some incorrect program consistent with the IO pairs partial specification (and therefore not prunable). In addition, this represents a probability where the C programs are sampled independently, which in practice we wouldn't do. One aspect of the program synthesis task is that syntactically incorrect programs can be trivially identified and pruned before making a prediction. As a , if we use stx to denote the event that the sampled program is syntactically correct, what we care about modeling correctly is p(λ | IO k i k=1..K, stx). Using Bayes rule, we can rewrite this: DISPLAYFORM0 We drop the conditional dependency on the IO pairs in the second line of as the syntactical correctness of the program is independent from the specification when conditioned on the program. We can do the same operation at the token level, denoting by stx 1...t the event that the sequence of the first t tokens s 1 · · · s t doesn't contain any syntax error and may therefore be a prefix to a valid program. DISPLAYFORM1 Given a grammar, it is possible to construct a checker to determine valid prefixes of programs. Example applications include compilers to signal syntax errors to the user and autocomplete features of Integrated Development Environments (IDEs) to restrict the list of suggested completions. The quantity p (stx 1...t | s 1 · · · s t) is therefore not a probability but can be implemented as a deterministic process for a given target programming language. In practice, this is implemented by getting at each timestep a mask M = {− inf, 0} | Σ | where M j = − inf if the j-th token in the alphabet is not a valid token in the current context, and 0 otherwise. This mask is added to the output of the network, just before the Softmax operation that normalizes the output to a probability over the tokens. Conditioning on the syntactical correctness of the programs provides several advantages: First, sampled programs become syntactically correct by construction. At test time, it allows the beam search to only explore useful candidates. It also ensures that when we are optimizing for correctness, the samples used to approximate the distribution are all going to be valid candidates. Restricting the dimension of the space on which our model is defined also makes the problem simpler to learn. It may not always be feasible to assume access to a syntax checker. In general, we wish to retain the syntax checker's ability to aggressively prune the search in program space without requiring access to the syntax itself. To this end, we propose to represent the syntax checker as a neural network module and learn it jointly. Similar to the base model, we implement learned syntax checking using an LSTM g φ. Comparing to the decoder LSTM, there are two major differences:• The syntaxLSTM is conditioned only on the program tokens, not on the IO pairs. This ensures that the learned checker models only the syntax of the language.• The output of the syntaxLSTM is passed through an elementwise x → − exp(x) activation function and added to the decoder LSTM's output. Similar to the mask in Section 5.1, the exponential activation function allows the syntaxLSTM to output high penalties to any tokens deemed syntactically incorrect. The addition of the syntaxLSTM doesn't necessitate any change to the training procedure as it is simply equivalent to a change of architecture. However, in the supervised setting, when we have access to syntactically correct programs, we have the possibility of adding an additional term to the loss to prevent the model from masking valid programs: DISPLAYFORM0 This loss penalizes the syntaxLSTM for giving negative scores to each token belonging to a known valid program. We use the reference programs as example of valid programs when we perform supervised training. The Karel programming language is an educational programming language , used for example in Stanford CS introductory classes (cs1) or in the Hour of Code initiative (hoc). It features an agent inside a gridworld (See FIG0, capable of moving (move, turn{Left,Right}), modifying world state ({pick,put}Marker), and querying the state of the nearby environment for its own markers (markerPresent, noMarkerPresent) or for natural obstacles (frontIsClear, leftIsClear, rightIsClear). Our goal is to learn to generate a program in the Karel DSL given a small set of input and output grids. The language supports for loops, while loops, and conditionals, but no variable assignment. Compared to the original Karel language, we only removed the possibility of defining subroutines. The specification for the DSL can be found in appendix B.To evaluate our method, we follow the standard practice BID7 BID28 BID27 BID3 and use a synthetic dataset generated by randomly sampling programs from the DSL. We perform a few simple heuristic checks to ensure generated programs have observable effect on the world and prune out programs performing spurious actions (e.g. executing a turnLeft just after a turnRight for example). For each program, we a set of IO pairs are generated by sampling random input grids and executing the program on them to obtain the corresponding output grids. A large number of them are sampled and 6 are kept for each program, ensuring that all conditionals in a program are hit by at least one of the examples. The first 5 samples serve as the specification, and the sixth one is kept as held-out test pair. 5000 programs are not used for training, and get split out between a validation set and a test set. We represent the input and output elements as grids where each cell in the grid is a vector with 16 channels, indicating the presence of elements (AgentFacingNorth, AgentFacingSouth, · · ·, Obstacle, OneMarkerPresent, TwoMarkersPresent, · · ·). The input and output grids are initially passed through independent convolution layers, before being concatenated and passed through two convolutional residual blocks and a fully connected layer, mapping them to a final 512-dimensional representation. We run one decoder per IO pair and perform a maxpooling operation over the output of all the decoders, out of which we perform the prediction of the next token. Our models are implemented using the Pytorch framework (pyt). Code and data will be made available. Table 1: RL_beam optimization of program correctness in consistent improvements in top-1 generalization accuracy over supervised learning MLE, even though the exact match of recovering the reference program drops. The improved objective function in further improvements. We trained a variety of models on the full Karel dataset containing 1-million examples as well as a reduced dataset containing only 10, 000 examples. In general, the small dataset serves to help understand the data efficiency of the program synthesis methods and is motivated by the expected difficulty of obtaining many labeled examples in real-world program synthesis domains. The Karel DSL was previously used by BID6 to study the relative perfomances of a range of methods depending on the available amount of data. The task considered was however different as they attempted to perform program induction as opposed to program synthesis. Rather than predicting a program implementing the desired transformation, they simply output a specification of the changes that would from applying the program so a direct number comparison wouldn't be meaningful. Models are grouped according to training objectives. As a baseline, we use MLE, which corresponds to the maximum likelihood objective (Eq.6), similar to the method proposed by BID7. Unless otherwise specified, the reward considered for our other methods is generalization: +1 if the program matches all samples, including the held out one and 0 otherwise. RL uses the expected reward objective (Eq.7), using REINFORCE to obtain a gradient estimate (Eq.8). RL_beam attempts to solve the proxy problem described by Equation FORMULA11 and RL_beam_div the richer loss function of Equation. RL_beam_div_opt also optimizes the loss of equation FORMULA12 but the reward additionally includes a term inversly proportional to the number of timesteps it takes for the program to finish executing. All RL models are initialized from pretrained supervised models. Optimizing for correctness (RL): Results in Table 1 show that optimizing for the expected program correctness consistently provides improvements in top-1 generalization accuracy. Top-1 Generalization Accuracy (Eq. 4) denotes the accuracy of the most likely program synthesized by beam search decoding having the correct behaviour across all input-output examples. We didn't perform any pruning of programs that were incorrect on the 5 specification examples. The improved performance of RL methods confirms our hypothesis that better loss functions can effectively combat the program aliasing problem. On the full dataset, when optimizing for correctness, Exact Match Accuracy decreases, indicating that the RL models no longer prioritize generating programs that exactly match the references. On the small dataset, RL_beam methods improves both exact match accuracy and generalization. Comparing RL_beam to standard RL, we note improvements across all levels of generalization. By better aligning the RL objective with the sampling that happens during beam search decoding, consistent improvements can be made in accuracy. Further improvements are made by encouraging diversity in the beam of solutions (RL_beam_div) and penalizing long running programs (RL_beam_div_opt).In the settings where little training data is available, RL methods show more dramatic improvements over MLE, indicating that data efficiency of program synthesis methods can be greatly improved by using a small number of samples first for supervised training and again for Reinforcement Learning. As a side note, we were unable to achieve high performance when training the RL methods from scratch. The necessity of extensive supervised pretraining to get benefits from Reinforcement Table 3: Grammar prunes the space of possible programs: On the full dataset, handwritten syntax checking MLE_handwritten improves accuracy over no grammar MLE, although MLE_large shows that simply adding more parameters in even greater gains. On the small dataset, learning the syntax MLE_learned outperforms handwritten grammar and larger models. Learning fine-tuning is well-known in the Neural Machine Translation literature BID30 BID37 BID36 BID2. TAB3 examines the top-1, top-5, and top-50 generalization accuracy of supervised and RL models. RL_beam methods performs best for top-1 but their advantage drops for higher-rank accuracy. Inspection of generated programs shows that the top predictions all become diverse variations of the same program, up to addition/removal of no-operations (turnLeft followed by turnRight, full circle obtained by a series of four turnLeft). The RL_beam_div objective helps alleviate this effect as does RL_beam_div_opt, which penalizes redundant programs. This is important as in the task of Program Synthesis, we may not necessarily need to return the most likely output if it can be pruned by our specification. We also compare models according to the use of syntax: MLE_handwritten denotes the use of a handwritten syntax checker (Sec 5.1), MLE_learned denotes a learned syntax (Sec 5.2), while no suffix denotes no syntax usage. Table 3 compares syntax models. On the full dataset, leveraging the handwritten syntax leads to marginally better accuracies than learning the syntax or using no syntax. Given access to enough data, the network seems to be capable of learning to model the syntax using the sheer volume of training examples. On the other hand, when the amount of training data is limited, learning the syntax produces significantly better performance. By incorporating syntactic structure in the model architecture and objective, more leverage is gained from small training data. Interestingly, the learned syntax model even outperforms the handwritten syntax model. We posit the syntaxLSTM is free to learn a richer syntax. For example, the syntaxLSTM could learn to model the distribution of programs and discourage the prediction of not only syntactically incorrect programs, but also the unlikely ones. To control for the extra parameters introduced by the syntaxLSTM, we compare against MLE_large, which uses no syntax but features a larger decoder LSTM, ing in the same number of parameters as MLE_learned. Results show that the larger number of parameters is not enough to explain the difference in performance, which again indicates the utility of jointly learning syntax. Analysis of learned syntax: Section 5.2 claimed that by decomposing our models into two separate decoders, we could decompose the learning so that one decoder would specialize in picking the likely tokens given the IO pairs, while the other would enforce the grammar of the language. We now provide experimental evidence that this decomposition happens in practice. TAB5 shows the percentage of syntactically correct programs among the most likely predictions of the MLE + learned model trained on the full dataset. Both columns correspond to the same set of parameters but the second column doesn't apply the syntaxLSTM's mask to constrain the decoding process. The precipitous drop in syntax accuracy indicates the extent to which the program decoder has learned to rely on the syntaxLSTM to produce syntactically correct programs. FIG2 compares the syntax masks generated by the learned and handwritten syntax models while decoding Program A in FIG0. FORMULA2 FIG2 analyzes the difference between the handwritten and learned syntax masks. White indicates similar output, which occupies the majority of the visualization. Blue cells correspond to instances where the syntaxLSTM labeled a token correct when it actually was syntactically incorrect. This type of error can be recovered if the program decoder predicts those tokens as unlikely. On the other hand, red cells indicate the syntaxLSTM predicted a valid token is syntactically incorrect. This type of error is more dangerous because the program decoder cannot recover the valid token once it is declared incorrect. The majority of red errors correspond to tokens which are rarely observed in the training dataset, indicating that the syntaxLSTM learns to model more than just syntaxit also captures the distribution over programs. Given a large enough dataset of real programs, the syntaxLSTM learns to consider non-sensical and unlikely programs as syntactically incorrect, ensuring that generated programs are both syntactically correct and likely. In this section, we describe how the objective function of Equation FORMULA12 can be computed. We have a distribution q θ over S programs, as obtained by performing a beam search over p θ and renormalizing. We are going to get C independent samples from this distribution and obtain a reward corresponding to the best performing of all of them. In the case where the reward function R i (λ r) is built on a boolean property, such as for example "correctness of the generated program", Equation can be simplified. As R i (λ r) can only take the values 0 or 1, the term max j∈1.. C R i (λ j) is going to be equal to 0 only if all of the C sampled programs give a reward of zero. For each sample, there is a probability of DISPLAYFORM0 ) of sampling a program with rewards zero. The probability of not sampling a single correct program out of the C samples is q C incorrect. From this, we can derive the form of Equation. Note that this can be computed without any additional sampling steps as we have a close form solution for this expectation. In the general case, a similar derivation can be obtained. Assume that the programs outputted by the beam search have associated rewards R 0, R 1,..., R S and assume without loss of generality that R 0 < R 1 <.. < R S. The probability of sampling a program with a reward smaller than R i is q ≤Ri = λr∈BS(p θ,S) [R i (λ r) ≤ R i ] q θ (λ r | IO k i k=1..K) so the probability of obtaining a final reward of less than R i when sampling C samples is q C ≤Ri. As a , the probability of obtaining a reward of exactly R i is q The state of the grid word are represented as a 16 × 18 × 18 tensor. For each cell of the grid, the 16-dimensional vector corresponds to the feature indicated in TAB7 The decoders are two-layer LSTM with a hidden size of 256. Tokens of the DSL are embedded to a 256 dimensional vector. The input of the LSTM is therefore of dimension 768 (256 dimensional of the token + 512 dimensional of the IO pair embedding). The LSTM used to model the syntax is similarly sized but doesn't take the embedding of the IO pairs as input so its input size is only 256.One decoder LSTM is run on the embedding of each IO pair. The topmost activation are passed through a MaxPooling operation to obtain a 256 dimensional vector representing all pairs. This is passed through a linear layer to obtain a score for each of the 52 possible tokens. We add the output of the model and of the eventual syntax model, whether learned or handwritten and pass it through a SoftMax layer to obtain a probability distribution over the next token. All training is performed using the Adam optimizer, with a learning rate of 10 − 4. Supervised training used a batch size of 128 and RL methods used a batch size of 16. We used 100 rollouts per samples

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.