Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We present a novel approach for training neural abstract architectures which in- corporates (partial) supervision over the machine’s interpretable components. To cleanly capture the set of neural architectures to which our method applies, we introduce the concept of a differential neural computational machine (∂NCM) and show that several existing architectures (e.g., NTMs, NRAMs) can be instantiated as a ∂NCM and can thus benefit from any amount of additional supervision over their interpretable components. Based on our method, we performed a detailed experimental evaluation with both, the NTM and NRAM architectures, and showed that the approach leads to significantly better convergence and generalization capabilities of the learning phase than when training using only input-output examples. Recently, there has been substantial interest in neural abstract machines that can induce programs from examples BID2; BID4;; BID7; BID11; BID14; BID18; BID20; BID23; BID24. While significant progress has been made towards learning interesting algorithms BID8, ensuring the training of these machines converges to the desired solution can be very challenging. Interestingly however, even though these machines differ architecturally, they tend to rely on components (e.g., neural memory) that are more interpretable than a typical neural network (e.g., an LSTM). A key question then is:Can we somehow provide additional amounts of supervision for these interpretable components during training so to bias the learning towards the desired solution?In this work we investigate this question in depth. We refer to the type of supervision mentioned above as partial trace supervision, capturing the intuition that more detailed information, beyond inputoutput examples, is provided during learning. To study the question systematically, we introduce the notion of a differential neural computational machine (∂NCM), a formalism which allows for clean characterization of the neural abstract machines that fall inside our class and that can benefit from any amount of partial trace information. We show that common architectures such as Neural Turing Machines (NTMs) and Neural Random Access Machines (NRAMs) can be phrased as ∂NCMs, useful also because these architectures form the basis for many recent extensions, e.g., BID8; BID9; BID11. We also explain why other machines such as the Neural Program Interpreter (NPI) BID18 or its recent extensions (e.g., the Neural Program Lattice BID15) cannot be instantiated as an ∂NCM and are thus restricted to require large (and potentially prohibitive) amounts of supervision. We believe the ∂NCM abstraction is a useful step in better understanding how different neural abstract machines compare when it comes to additional supervision. We then present ∂NCM loss functions which abstractly capture the concept of partial trace information and show how to instantiate these for both the NTM and the NRAM. We also performed an extensive evaluation for how partial trace information affects training in both architectures. Overall, our experimental indicate that the additional supervision can substantially improve convergence while leading to better generalization and interpretability. To provide an intuition for the problem we study in this work, consider the simple task of training an NTM to flip the third bit in a bit stream (called Flip3rd) -such bitstream tasks have been extensively studied in the area of program synthesis (e.g., BID10 ; BID17). An example input-output pair for this task could be examples, our goal is to train an NTM that solves this task. An example NTM that generalizes well and is understandable is shown in FIG0. Here, the y-axis is time (descending), the x-axis is the accessed memory location, the white squares represent the write head of the NTM, and the orange squares represent the read head. As we can see, the model writes the input sequence to the tape and then reads from the tape in the same order. However, in the absence of richer supervision, the NTM (and other neural architectures) can easily overfit to the training set -an example of an overfitting NTM is shown in FIG0. Here, the traces are chaotic and difficult to interpret. Further, even if the NTM generalizes, it can do so with erratic reads and writes, an example of which is shown in FIG0. Here, the NTM learns to read from the third bit (circled) with a smaller weight than from other locations, and also reads and writes erratically near the end of the sequence. This model is less interpretable than the one in FIG0 because it is unclear how the model knows which the third bit actually is, or why a different read weight would help flip that bit. In this work we will develop principled ways for guiding the training of a neural abstract machine towards the behavior shown in FIG0. For instance, for Flip3rd, providing partial trace information on the NTM's read heads for 10% of the input-output examples is sufficient to bias the learning towards the NTM shown in FIG0 100% of the time. To capture the essence of our method and illustrate its applicability, we now define the abstract notion of a neural computational machine (NCM). NCMs mimic classic computational machines with a controller and a memory, and generalize multiple existing architectures. Our approach for supervision with partial trace information applies to all neural architectures expressible as NCMs. A useful feature of the NCM abstraction is that it clearly delineates end-to-end differentiable architectures BID7's NTM, BID14's NRAM), which can train with little to no trace supervision, from architectures that are not end-to-end differentiable BID18's NPI) and hence require a certain minimum amount of trace information. In the follow-up section, we show how to phrase two existing neural architectures (NTMs and NRAMs) as an NCM.An NCM is a triple of functions: a processor, a controller, and a loss:Processor The processor π: W × C × M → B × M performs a pre-defined set of commands C, which might involve manipulating memories in M. The commands may produce additional feedback in B. Also, the processor's operation may depend on parameters in W.Controller The controller κ: W × B × Q × I → C × Q × O decides which operations the machine performs at each step. It receives external inputs from I and returns external outputs in O. It can also receive feedback from the processor and command it to do certain operations (e.g., memory read). The decisions the controller takes may depend on its internal state (from Q). The controller can also depend on parameters in W. For instance, if the controller is a neural network, then the network's weights will range over W. Loss Function The loss function L e: Trace × E → R indicates how close a trace τ ∈ Trace of an execution of the machine (defined below) is to a behavior from a set E. The loss function provides a criterion for training a machine to follow a prescribed set of behaviors, and hence we impose certain differentiability conditions. We require that the loss surface is continuous and piecewise differentiable with respect to the weights w ∈ W for all examples e and inputs x with traces τ (w, x): DISPLAYFORM0 Execution The execution of the machine begins with an input sequence x = {x t} n 1 and initial values of the controller state q 0, memory m 0, and processor feedback b 0. At each time step t = 1... n, controller and processor take turns executing according to the following equations: DISPLAYFORM1 A trace τ (w, x, b 0, m 0, q 0) = {(c t, b t, q t, y t, m t)} n 1 records these quantities' values at each time step. We will occasionally write τ C, τ B,... for the trace projected onto one of its components c, b,.... ∂NCMs Note that the differentiability conditions that we impose on the loss do not imply that any of the NCM functions π, κ and L e are continuous or differentiable. They indeed can be highly discontinuous as in NCMs like BID21's memory networks with a hard attention mechanism, or as in BID18's neural programmer-interpreters. In order to fix these discontinuities and recover a differentiable loss surface, these architectures train with strong supervision only: the training examples e ∈ E must provide a value for every traced quantity that comes from a discontinuous parameter. In contrast, what we call differentiable neural computational machines (∂NCM), have κ, π and L e continuous and piecewise differentiable. In this case, the loss surface is differentiable with respect to every parameter. Thus, there is no need to specify corresponding values in the examples, and so we can train with as much trace information as available. We now show how NTMs and NRAMs can be instantiated as ∂NCMs. NTM as ∂NCM An Neural Turing Machine (NTM) BID7 FIG1 ) has access to a memory M ∈ R c×n of c cells of n real numbers each. We suppose the machine has one read head and one write head, whose addresses are, respectively, the probability vectors r, w ∈ {1...c}. At every time step, the read head computes the expected value m ∈ R n of a random cell at index i ∼ r. This value together with the current input are fed into a controller neural network, which then decides on several commands. It decides what fraction e ∈ R n to erase and how much a ∈ R n to add to the cells underneath the write head. The write head stores the tape expected after a random modification at index i ∼ w. Then the controller indicates the head movement with two probability vectors ∆r, ∆w ∈ {−1,0,+1} which are convolved with the respective head addresses (the actual addressing mechanism is more involved, but we omit it for brevity) Finally, the controller produces the current output value. In terms of NCMs, the NTM's variables fall into the following classes: DISPLAYFORM0 Each of these variables change over time according to certain equations (see Appendix A for details).The processor π and the controller κ functions for each time step satisfy: DISPLAYFORM1 The standard loss function L e for the NTM simply includes a term, such as cross-entropy or L 2 distance, for the machine output at every time step. Each of these compare the machine output to the respective values contained in the examples e ∈ E.NRAM as ∂NCM A Neural Random Access Machine (NRAM) BID14 is a neural machine designed for ease of pointer (de-) referencing. An NRAM has a variable sized memory M ∈ R c×c whose size varies between runs. It also has access to a register file r ∈ R n×c with a constant number n of registers. Both the memory and the registers store probability vectors over {1 . . . c}. The controller receives no external inputs, but at each time step reads the probability that a register assigns to 0. It also produces no external output, except a probability f ∈ for termination at the current time step. The output of the run is considered to be the final memory state. Unlike the NTM, computation in the NRAM is performed by a fixed sequence of modules. Each module implements a simple integer operation/memory manipulation lifted to probability vectors. For example, addition lifts to convolution, while memory access is like that of the NTM. At every time step the controller organizes the sequence of modules into a circuit, which is then executed. The circuit is encoded by a pair of probability distributions per module, as shown in FIG1. These distributions specify respectively which previous modules or registers will provide a given module first/second arguments. The distributions are stacked in the matrices a and b. A similar matrix c is responsible for specifying what values should be written to the registers at the end of the time step. The NCM instantiation of an NRAM is the following: DISPLAYFORM2 The equations that determine these quantities can be found in Appendix B. The processor function π and the controller function κ expressed in terms of these quantities are: DISPLAYFORM3 The loss of the NRAM is more complex than the NTM loss: it is an expectation with respect to the probability distribution p of termination time, as determined by the termination probabilities f t (see Appendix B). For every t = 1... k, the loss considers the negative log likelihood that the i-th memory cell at that time step equals the value e i provided in the example, independently for each i: DISPLAYFORM4 Incorporating supervision during NCM training can be helpful with: (i) convergence: additional bias may steer the minimization of the NCM's loss function L e, as much as possible, away from local minima that do not correspond to good solutions, (ii) interpretability: the bias can also be useful in guiding the NCM towards learning a model that is more intuitive/explainable to a user (especially if the user already has an intuition on what it is that parts of the model should do), and (iii) generalization: the bias can steer the NCM towards solutions which minimize not just the loss on example of difficulties it has seen, but on significantly more difficult examples. The way we provide additional supervision to NCMs is, by encoding, for example, specific commands issued to the processor, into extra loss terms. Let us illustrate how we can bias the learning with an NTM. Consider the task of copying the first half of an input sequence {x t} 2l 1 into the second half of the machine's output {y t} 2l 1, where the last input x l from the first half is a special value indicating that the first half ended. Starting with both heads at position 1, the most direct solution is to consecutively store the input to the tape during the first half of the execution, and then recall the stored values during the second half. In such a solution, we expect the head positions to be: DISPLAYFORM0 To incorporate this information into the training, we add loss terms that measure the cross-entropy (H) between p(t) and w t as well as between q(t) and r t. Importantly, we need not add terms for every time-step, but instead we can consider only the corner cases where heads change direction: DISPLAYFORM1 H(p(t), w t ) + H(q(t), r t ). We now describe the general shape of the extra loss terms for arbitrary NCMs. Since, typically, we can interpret only the memory and the processor in terms of well-understood operations, we will consider loss terms only for the memory state and the communication flow between the controller and the processor. We leave the controller's hidden state unconstrained -this also permits us to use the same training procedure with different controllers. The generic loss is expressed with four loss functions for the different components of an NCM trace: DISPLAYFORM0 For each part α ∈ {C, B, O, M}, we provide hints (t, v, µ) ∈ σ α that indicate a time step t at which the hint applies, an example v ∈ E α for the relevant component, and a weight w ∈ R of the hint. The weight is included to account for hints having a different importance at different time-steps, but also to express our confidence in the hint, e.g., hints coming from noisy sources would get less weight. A subtrace σ is a collection of hints used for a particular input-output example e. We call it a subtrace because, typically, it contains hints for a proper subset of the states traced by the NCM during execution. The net loss for a given input-output example and subtrace equals the original loss L e added to the weighted losses for all the hints, scaled by a constant factor λ: DISPLAYFORM1 For NTMs, we allow hints on the output y, the addresses r and w, and the tape M. We include extra loss terms for the memory state only (all other loss terms are zero): DISPLAYFORM0 Unlike the output and addresses, values on the tape are interpreted according to an encoding internal to the controller (which emerges only during training). Forcing the controller to use a specific encoding for the tape, as we do with NTM output, can have a negative effect on training (in our experiments, training diverged consistently). To remedy this, we do not apply loss to the tape directly but to a decoded version of a cell on the tape. While a decoder might find multiple representations and overfit, we found that it forced just enough consistency to improve the convergence rate. The decoder itself is an auxiliary network φ trained together with the NTM, which takes a single cell from memory as input. The output of the decoder is compared against the expected value which should be in that cell: DISPLAYFORM1 For all subtraces we provide in our experiments with NTMs, the hints have the same unit weight. For NRAMs, we hint which connections should be present in the circuit the controller constructs at each step, including the ones for register updates. An example circuit is shown in FIG2. In terms of an NCM, this amounts to providing loss for commands and no loss for anything else. We set the loss to the negative log likelihood of the controller choosing specific connections revealed in the hint: DISPLAYFORM0 In our experiments, we observed that assigning higher weight to hints at earlier timesteps is crucial for convergence of the training process. For a hint at time-step t, we use the weight µ = (t + 1) −2. A possible reason for why this helps is that the machine's behavior at later time-steps is highly dependent on its behavior at the early time-steps. Thus, the machine cannot reach a later behavior that is right before it fixes its early behavior. Unless the behavior is correct early on, the loss feedback from later time-steps will be mostly noise, masking the feedback from early time-steps. Other Architectures The NCM can be instantiated to architectures as diverse as a common LSTM network or End-To-End Differentiable Memory Networks. Any programming inducing neural network with at least partially interpretable intermediate states for which the dataset contains additional hints could be considered a good candidate for application of this abstraction. We evaluated our NCM supervision method on the NTM and NRAM architectures. For each of the two architectures we implemented a variety of tasks and experimented with different setups of trace supervision. The main questions that we address are: (i) does trace supervision help convergence, interpretability, and generalization? (ii) how much supervision is needed to train such models? Below, we summarize our findings -further details are provided in the appendix. Figure 5: The number of initial runs which generalized for Flip3rd. The first dimension listed in the rows controls the execution details revealed in a subtrace, while the second dimension (the density column) controls the proportion of examples that receive extra subtrace supervision. We measured how often we successfully trained an NTM that achieves strong generalization. We consider a model to generalize if relative to the training size limit n, it achieves perfect accuracy on all of tests of size ≤ 1.5n, and perfect accuracy on 90% of the tests of size ≤ 2n. FIG3 reports the average improvement compared to a baseline using only I/O examples. We ran experiments with four different tasks and various types of hints (cf. Appendices C, E). Some of the hint types are: read and write specify respective head addresses for all time steps; address combines the previous two; corner reveals the head addresses, but only when the heads change direction; value gives value for a single cell. Except for three cases, trace supervision helped improve generalization. Here, RepeatFlip3d is most challenging, with baseline generalizing only 5% of the time (cf. Appendix I). Here we have the largest improvement with extra supervision: corner type of hints achieve eight-fold increase in success rate, reaching 40%. Another task with an even larger ratio is RepeatCopyTwice (cf. Appendix), where success increases from 15.5% to 100%.In addition to this experiment, we performed an extensive evaluation of different setups, varying the global λ parameter of the loss Eq. 8, and providing hints for just a fraction of the examples. The full are in Appendix I; here we provide those for RepeatFlip3d in Table 5. The table reveals that the efficacy of our method heavily depends on these two parameters. The best in this case are for the read/corner type of hints 1 2 / 1 10 of the time, with λ ∈ {0.1, 1}. The best for other tasks are achieved with different setups. Generally, our is that training with traces 50% of the time usually improves performance (or does not lower it much) when compared to the best method. This observation raises the interesting question of what the best type and amount of hints are for a given task. Finally, we observed that in all cases where training with trace supervision converged, it successfully learned the head movements/tape values we had intended. This show that trace supervision can bias the architecture towards more interpretable behaviors. In those cases, the NTM learned consistently sharper head positions/tape values than the baseline, as FIG4 shows for Flip3rd. BID16 reporting that ListK for example generalizes poorly, even when trained with noise in the gradient, curriculum learning, and an entropy bonus. We observed that when run on an indefinite number of examples with the correct number of timesteps and a correct module sequence, Swap and Increment would in fact occasionally generalize perfectly, but did not have the resources to run such indefinite tests with Permute, ListK, and Merge. FIG5 demonstrates that when training had finished, either because it had ended early or had reached 5000 training examples (our upper bound), generalization would in fact be on average significantly better than the baseline the more hints that were used for all tasks. Here, number of hints used seemed to be a sufficient predictor for the quality of the trained model. The effect of increasing supervision on the quality of the trained model was so strong that not even noise in the input was able to significantly hinder generalization. In FIG5, we corrupted a single character in the output examples for the Permute problem in 10% of the examples. We found that without any extra hints, no convergence was seen after training was complete, whereas with just corner subtraces, the generalization was nearly optimal. Furthermore, we found that noise in the trace does not seriously harm performance. We corrupted a single hint for 20% of the traces of the Increment task using otherwise full supervision, as can be seen in the NoisyFull line of FIG0. We presented a method for incorporating (any amount of) additional supervision into the training of neural abstract machines. The basic idea was to provide this supervision (called partial trace information) over the interpretable components of the machine and to thus more effectively guide the learning towards the desired solution. We introduced the ∂NCM architecture in order to precisely capture the neural abstract machines to which our method applies. We showed how to formulate partial trace information as abstract loss functions, how to instantiate common neural architectures such as NTMs and NRAMs as ∂NCMs and concretize the ∂NCM loss functions. Our experimental indicate that partial trace information is effective in biasing the learning of both NTM's and NRAM's towards better converge, generalization and interpretability of the ing models. The controller for the NTM consists of the networks ϕ, ψ y, ψ e, ψ a, χ r, χ w, which operate on the variables:x -in q -controller state r -read address ∆r -change in r e -erase M -tape y -out m -read value w -write address ∆w -change in w a -addThe equations that describe NTM executions are: DISPLAYFORM0 The controller of the NRAM consists of the networks ϕ, ψ a, ψ b, ψ c, ψ f, which operate on the variables:a -lhs circuit b -rhs circuit c -register inputs o -module outputs r -register state M -memory tape h -controller state f -stop probability. The equations that describe the NRAM execution are: DISPLAYFORM0 DISPLAYFORM1 For all of our NTM experiments we use a densely connected feed-forward controller. There are two architectural differences from the original NTM BID7 that helped our baseline performance: the feed-forward controller, the erase and the add gates use tanh activation; the output layer uses softmax. In the original architecture these are all logistic sigmoids. For the newly introduced tape decoder (active only during training) we used two alternative implementations: a tanh-softmax network, and a single affine transformation. We tested the NTM's learning ability on five different tasks for sequence manipulation, two of which have not been previously investigated in this domain. These tasks can be found in Appendix E.We performed experiments using several combination of losses as summarized in Appendix F. The observed training performance per task is shown in Appendix I, with rows corresponding to the different loss setups. The corner setup differs from the address setup in that the example subtraces were defined only for a few important corner cases. For example in RepeatCopyTwice, the write head was provided once at the beginning of the input sequence, and once at the end. Similarly, the read head was revealed at the beginning and at the end of every output repetition. In all other setups we provide full subtraces (defined for all time steps).The supervision amount can be tuned by adjusting the λ weight from Equation 8. Further, we can also control the fraction of examples which get extra subtrace supervision (the density row in Figure I). The performance metric we use is the percentage of runs that do generalize after 100k iterations for the given task and supervision type. By generalize we mean that the NTM has perfect accuracy on all testing examples up to 1.5× the size of the max training length, and also perfect accuracy on 90% of the testing examples up to 2× the maximum training length. We used a feed-forwad controller with 2 × 50 units, except for RepeatCopyTwice, which uses 2 × 100 units. For training we used the Adam optimizer BID12, a learning rate of 10 −3 for all tasks except RepeatFlip3d and Flip3rd which use 5 · 10 −4. The lengths of the training sequences for the first four tasks are from 1 to 5, whereas the generalization of the model was tested with sequences of lengths up to 20. For Flip3rd and RepeatFlip3d, the training sequence length was up to 16, whereas the testing sequences have maximum length of 32. Like in the NTM, we use a densely connected two layer feed forward controller for our experiments, and use ReLU as the activation function. We make no modifications to the original architecture, and use noise with the parameter η = 0.3 as suggested by BID16, and curriculum learning as described by BID22. We stop training once we get to a difficulty specified by the task, and increase the difficulty once 0 errors were found on a new testing batch of 10 samples. Each training iteration trains with 50 examples of the currently randomly sampled difficulty. Regardless of whether the model had converged, training is stopped after 5000 samples were used. Such a low number is used to replicate the potential conditions under which such a model might be used. As with the NTM, the Adam optimizer was used. The specific tasks we use are described in Appendix G, and the specific kinds of supervision we give are described in Appendix H. The λ we used here was 40. The system was implemented using PyTorch. Every input sequence ends with a special delimiter x E not occurring elsewhere in the sequence Copy -The input consists of generic elements, x 1... x n x E. The desired output is x 1... x n x E.RepeatCopyTwice -The input is again a sequence of generic elements, x 1... x n x E. The desired output is the input copied twice x 1... x n x 1... x n x E. Placing the delimiter only at the end of the output ensures that the machine learns to keep track of the number of copies. Otherwise, it could simply learn to cycle through the tape reproducing the given input indefinitely. We kept the number of repetitions fixed in order to increase baseline task performance for the benefit of comparison. DyckWords -The input is a sequence of open and closed parentheses, x 1... x n x E. The desired output is a sequence of bits y 1... y n x E such that y i = 1 iff the prefix x 1... x i is a balanced string of parentheses (a Dyck word). Both positive and negative examples were given. Flip3rd -The input is a sequence of bits, x 1 x 2 x 3... x n x E. The desired output is the same sequence of bits but with the 3rd bit flipped: x 1 x 2x3... x n x E. Such a task with a specific index to be updated (e.g., 3rd) still requires handling data dependence on the contents of the index (unlike say the Copy task).RepeatFlip3d -The input is a sequence of bits, x 1 x 2 x 3 x 4 x 5 x 5... x E. The desired output is the same sequence of bits but with every 3rd bit flipped: DISPLAYFORM0 F NTM SUBTRACES value traces provide hints for the memory at every timestep as explained in Equation FORMULA0.read -provides a hint for the address of the read head at every timestep.write -provides a hint for the address of the write head at every timestep.address -provides hints for the address of both the read and the write head at every timestep.addr+val -provides value, read and write hints for every timestep.corner -provides hints for the address of both the read and the write head at every "important" timestep -we decided what important means here depends on which task we are referring to. In general, we consider the first and last timesteps important, and also any timestep where a head should change direction. For example, in RepeatCopyTwice for an example of size n with e repeats, we'd provide the heads at timesteps 0, n, 2n, 3n..., en. Below we describe all the tasks we experimented with. We predominantly picked tasks that the NRAM is known to have trouble generalizing on. We did not introduce any new tasks, and more detailed descriptions of these tasks can be found in BID14.Swap -Provided two numbers, a and b and an array p, swap p[a] and p [b]. All elements but that in the last memory cell are not zero. Increment -Given an array p, return the array with one added to each element. All elements but that in the last cell for the input are not zero. Elements can be zero in the output. Permute -Given two arrays p and q return a new array s such that DISPLAYFORM0 The arrays p and q are preceded by a pointer, a, to array q. The output is expected to be a, DISPLAYFORM1 ListK -Given a linked list in array form, and an index k return the value at node k. Merge -given arrays p and q, and three pointers a, b, c to array p, q, and the output sequence (given as zeros initially), place the sorted combination of p and q into the output location. The following table describes the specific NRAM instantiation used for each task. The default sequence (def) is the one described by BID14. The number of timesteps is usually dependent on the length of the problem instance, M (equivalently the word size or difficulty), and in the case of ListKwas given with respect to the argument k. The difficulty (D) was simply the length of the sequence used. H NRAM SUBTRACES For each of the tasks listed Appendix G, we hand coded a complete circuit for every module and every timestep we would provide. The following subtraces types describe how we provide hints based on this circuit. None -provides no hints. Full -provides the entire circuit. SingleHint -provides a random hint at a random timestep. SingleTimestep -provides the entire circuit at a random timestep. Corners -provides the entire circuit at the first and last timesteps. Registers -provides hints for the registers at every timestep. Modules -provides hints for the modules at every timestep. Which Details to Reveal for NTM? The first dimension listed in the rows of the tables of Figure I controls the execution details revealed in a Subtrace. We use subtraces showing either the addresses without the tape values, only the read heads or the write heads, or even weaker supervision in a few corner cases. In tasks Copy FIG7 ), RepeatCopyTwice (FIG7) and DyckWords FIG7, it is frequently the case that when the NTM generalizes without supervision, it converges to an algorithm which we are able to interpret. For them, we designed the addr+val traces to match this algorithm, and saw increases in generalization frequency of at least 45%. It can be concluded that when additionally provided supervision reflects the interpretable "natural" behavior of the NTM, the learning becomes significantly more robust to changes in initial weights. Additionally, for tasks Flip3rd FIG7 ) and RepeatFlip3d FIG7 ), both the baseline and other supervision types are outperformed by training with read supervision. It is also notable that corner supervision in RepeatFlip3d achieves highest improvement over the baseline, 60% over 5%. In essence, this means that providing only a small part of the trace can diminish the occurrence of local minima in the loss function. How Often to Reveal for NTM? The second dimension controls the proportion of examples that receive extra subtrace supervision (the density columns in Figure I). For Flip3rd, RepeatCopyTwice and DyckWords we observed that having only a small number of examples with extra supervision leads to models which are more robust to initial weight changes than the baseline, although not necessarily always as robust as providing supervision all the time. A couple of interesting cases stand out. For Flip3rd with 10% corner subtraces and λ = 1, we find a surprisingly high rate of generalization. Providing address traces 10% of the time when training RepeatCopyTwice leads to better performance all the time. For RepeatFlip3d, write traces at 1% frequency and λ = 0.1 generalize 30% of the time vs. 5% for baseline. While the type of trace which works best varies per task, for each task there exists a trace which can be provided only 1% of the time and still greatly improve the performance over the baseline. This suggests that a small amount of extra supervision can improve performance significantly, but the kind of supervision may differ. It is an interesting research question to find out how the task at hand relates to the optimal kind of supervision. FIG0: The average number of errors on the test set for each task and subtrace type once trained. FORMULA0 with that of the NRAM using the full tracer for Merge. For this experiment, a maximum of 10000 samples were used for the DNGPU and 5000 for the NRAM. The DNGPU was run out of the box from the code supplied by the authors. 20 runs were averaged for the DNGPU and 38 runs for the NRAM. One can deduce that while neither is able to generalize this task perfectly, the simpler and easier to understand architecture, NRAM, does generalize better with fewer examples when those examples come with richer supervision. The NRAM is parametrized by one or more straight-line partial programs, i.e., programs with no branching and no loops, chosen by register states. The machine runs in a loop, repeatedly selecting the program for that register state then executing it. The programs are expressend in a simple single-assignment imperative language. Each program statement i invokes one of the modules of the architecture and assigns the of the invocation to a local variable x i. That variable cannot be changed later. The final program statement is a parallel-asignment that modifies the machine registers r 1... r k. The values that appear in assignments/invocations can be: variables in scope, machine registers, or holes?. These values are not used directly during execution: the actual values needs to be supplied by the NRAM controller. The values are only used as hints for the controller during training, with the whole? denoting no hint. We can describe the language in an EBNF-style grammar: P 1::= S 1 P i::= P i−1; S i P::= P 1; R 1 | P 2; R 2 |...An example program for the Increment task would be the following: DISPLAYFORM0 x 2 ← READ(r 1); x 3 ← ADD(x 2, x 1); x 4 ← WRITE(r 1, x 3); x 5 ← ADD(r 1, x 1); DISPLAYFORM1 Here, the controller is encouraged to read the memory at the location stored in the first register r 1, add one to it, then store it back, and then increment the the first register. An alternative to the trace-based approach is to make the controller produce values only for the holes, and use directly the specified variable/register arguments. This way, only the unspecified parts of the program are learned. This is, for example, the approach taken by ∂Forth BID0. There, programs are expressed in a suitably adapted variant of the Forth programming language, which is as expressive as the language discussed above, but less syntactically constrained. The drawback of this alternative is that whenever an argument other than a whole is specified, one must also specify the time steps to which it applies in all possible executions and not just the training ones. That is why, typically, these values are specified either for all or for none of the time steps. In the following examples, we will describe the register states using "0", "! 0" and "-" meaning respectively that a register has 0, that it contains anything but zero, or that it can contain anything. For any register pattern.x 1 ← READ(r 0); x 2 ← W RIT E(0, x 1); x 3 ← READ(r 1); x 4 ← ADD(x 3, x 1); x 5 ← READ(x 4); x 6 ← W RIT E(r 1, x 5); x 7 ← IN C(r 1); x 8 ← DEC(x 1); x 9 ← LT (x 7, x 8); r 0 ← 0; r 1 ← x 7; r 2 ← x 9; r 3 ← 0;

We increase the amount of trace supervision possible to utilize when training fully differentiable neural machine architectures.

represent the tldr for finding the corresponding scientific article:

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important. In this paper, we propose Bayesian quantized networks (BQNs), quantized neural networks (QNNs) for which we learn a posterior distribution over their discrete parameters. We provide a set of efficient algorithms for learning and prediction in BQNs without the need to sample from their parameters or activations, which not only allows for differentiable learning in quantized models but also reduces the variance in gradients estimation. We evaluate BQNs on MNIST, Fashion-MNIST and KMNIST classification datasets compared against bootstrap ensemble of QNNs (E-QNN). We demonstrate BQNs achieve both lower predictive errors and better-calibrated uncertainties than E-QNN (with less than 20% of the negative log-likelihood). A Bayesian approach to deep learning considers the network's parameters to be random variables and seeks to infer their posterior distribution given the training data. Models trained this way, called Bayesian neural networks (BNNs) , in principle have well-calibrated uncertainties when they make predictions, which is important in scenarios such as active learning and reinforcement learning . Furthermore, the posterior distribution over the model parameters provides valuable information for evaluation and compression of neural networks. There are three main challenges in using BNNs: Intractable posterior: Computing and storing the exact posterior distribution over the network weights is intractable due to the complexity and high-dimensionality of deep networks. Prediction: Performing a forward pass (a.k.a. as probabilistic propagation) in a BNN to compute a prediction for an input cannot be performed exactly, since the distribution of hidden activations at each layer is intractable to compute. Learning: The classic evidence lower bound (ELBO) learning objective for training BNNs is not amenable to backpropagation as the ELBO is not an explicit function of the output of probabilistic propagation. These challenges are typically addressed either by making simplifying assumptions about the distributions of the parameters and activations, or by using sampling-based approaches, which are expensive and unreliable (likely to overestimate the uncertainties in predictions). Our goal is to propose a sampling-free method which uses probabilistic propagation to deterministically learn BNNs. A seemingly unrelated area of deep learning research is that of quantized neural networks (QNNs), which offer advantages of computational and memory efficiency compared to continuous-valued models. QNNs, like BNNs, face challenges in training, though for different reasons: (4.1) The non-differentiable activation function is not amenable to backpropagation. (4.2) Gradient updates cease to be meaningful, since the model parameters in QNNs are coarsely quantized. In this work, we combine the ideas of BNNs and QNNs in a novel way that addresses the aforementioned challenges in training both models. We propose Bayesian quantized networks (BQNs), models that (like QNNs) have quantized parameters and activations over which they learn (like BNNs) categorical posterior distributions. BQNs have several appealing properties: • BQNs solve challenge due to their use of categorical distributions for their model parameters. • BQNs can be trained via sampling-free backpropagation and stochastic gradient ascent of a differentiable lower bound to ELBO, which addresses challenges, and above. • BQNs leverage efficient tensor operations for probabilistic propagation, further addressing challenge. We show the equivalence between probabilistic propagation in BQNs and tensor contractions , and introduce a rank-1 CP tensor decomposition (mean-field approximation) that speeds up the forward pass in BQNs. • BQNs provide a tunable trade-off between computational resource and model complexity: using a refined quantization allows for more complex distribution at the cost of more computation. • Sampling from a learned BQN provides an alternative way to obtain deterministic QNNs. In our experiments, we demonstrate the expressive power of BQNs. We show that BQNs trained using our sampling-free method have much better-calibrated uncertainty compared with the stateof-the-art Bootstrap ensemble of quantized neural networks (E-QNN) trained by. More impressively, our trained BQNs achieve comparable log-likelihood against Gaussian Bayesian neural network (BNN) trained with stochastic gradient variational Bayes (SGVB) (the performance of Gaussian BNNs are expected to be better than BQNs since they allows for continuous random variables). We further verify that BQNs can be easily used to compress (Bayesian) neural networks and obtain determinstic QNNs. Finally, we evaluate the effect of mean-field approximation in BQN, by comparing with its Monte-Carlo realizations, where no approximation is used. We show that our sampling-free probabilistic propagation achieves similar accuracy and log-likelihood -justifying the use of mean-field approximation in BQNs. In Appendix A, we survey different approaches for training Bayesian neural networks including sampling-free assumed density filtering (; ; Hernández-;), sampling-based variational inference (; ;), as well as sampling-free variational inference , probabilistic neural networks (; ;), quantized neural network (; ; ; ; ; ; ; ; ;), and tensor networks and tensorial neural networks (; Orús, 2014; ; 2017; ; ;). • We propose an alternative evidence lower bound (ELBO) for Bayesian neural networks such that optimization of the variational objective is compatible with the backpropagation algorithm. • We introduce Bayesian quantized networks (BQNs), establish a duality between BQNs and hierarchical tensor networks, and show prediction a BQN is equivalent to a series of tensor contractions. • We derive a sampling-free approach for both learning and inference in BQNs using probabilistic propagation (analytical inference), achieving better-calibrated uncertainty for the learned models. • We develop a set of fast algorithms to enable efficient learning and prediction for BQNs. Notation. We use bold letters such as θ to denote random variables, and non-bold letters such as θ to denote their realizations. We abbreviate Pr [θ = θ] of N data points, we aim to learn a neural network with model parameters θ that predict the output y ∈ Y based on the input x ∈ X. We first solve the learning problem to find an approximate posterior distribution Q(θ; φ) over θ with parameters φ such that Q(θ; φ) ≈ Pr[θ|D]. We then solve the prediction problem to compute the predictive distribution Pr[y|x, D] for arbitrary input x = x given Q(θ; φ). For notational simplicity, we will omit the conditioning on D and write Pr [y|x, D] as Pr [y|x] in what follows. In order to address the prediction and learning problems in BNNs, we analyze these models in their general form of probabilistic graphical models (shown in Figure 3b in Appendix B). Let h (l), θ and h (l+1) denote the inputs, model parameters, and (hidden) outputs of the l-th layer respectively. We assume that θ (l)'s are layer-wise independent, i.e. Q(θ; φ) = Computing the predictive distribution Pr[y|x, D] with a BNN requires marginalizing over the random variable θ. The hierarchical structure of BNNs allows this marginalization to be performed in multiple steps sequentially. In Appendix B, we show that the predictive distribution of h (l+1) given input x = x can be obtained from its preceding layer h (l) by This iterative process to compute the predictive distributions layer-by-layer sequentially is known as probabilistic propagation (; Hernández-;). With this approach, we need to explicitly compute and store each intermediate is a function of x). Therefore, probabilistic propagation is a deterministic process that computes ψ (l+1) as a function of ψ (l) and φ (l), which we denote as Challenge in Sampling-Free Probabilistic Propagation. If the hidden variables h (l)'s are continuous, Equation generally can not be evaluated in closed form as it is difficult to find a family of parameterized distributions P for h (l) such that h (l+1) remains in P under the operations of a neural network layer. Therefore most existing methods consider approximations at each layer of probabilistic propagation. In Section 4, we will show that this issue can be (partly) addressed if we consider the h (l)'s to be discrete random variables, as in a BQN. Objective Function. A standard approach to finding a good approximation Q(θ; φ) is variational inference, which finds φ such that the KL-divergence KL(Q(θ; φ)||Pr [θ|D] ) from Q(θ; φ) to Pr[θ|D] is minimized. In Appendix B, we prove that to minimizing the KL-divergence is equivalent to maximizing an objective function known as the evidence lower bound (ELBO), denoted as L(φ). where Probabilistic Backpropagation. Optimization in neural networks heavily relies on the gradientbased methods, where the partial derivatives ∂L(φ)/∂φ of the objective L(φ) w.r.t. the parameters φ are obtained by backpropagation. Formally, if the output produced by a neural network is given by a (sub-)differentiable function g(φ), and the objective L(g(φ)) is an explicit function of g(φ) (and not just an explicit function of φ), then the partial derivatives can be computed by chain rule: Published as a conference paper at ICLR 2020 The learning problem can then be (approximately) solved by first-order methods, typically stochastic gradient descent/ascent. Notice that For classification, the function g(φ) returns the probabilities after the softmax function, not the categorical label; An additional regularizer R(φ) on the parameters will not cause difficulty in backpropagation, given ∂R(φ)/∂φ is easily computed. Challenge in Sampling-Free Probabilistic Backpropagation. Learning BNNs is not amenable to standard backpropagation because the ELBO objective function L(φ) in (4b) is not an explicit (i.e. implicit) function of the predictive distribution g(φ) in (4a): Although L n (φ) is a function of φ, it is not an explicit function of g n (φ). Consequently, the chain rule in Equation on which backpropagation is based is not directly applicable. Alternative Evidence Lower Bound. We make learning in BNNs amenable to backpropagation by developing a lower bound is an explicit function of the from the forward pass.) With L n (φ) in hand, we can (approximately) find φ by maximizing the alternative objective via gradient-based method: In Appendix C.1, we proved one feasible L n (φ) which only depends on second last output h (L−1). ) is deterministic given input x and all parameters before the last layer θ Analytic Forms of L n (φ). While the lower bound in Theorem 3.1 applies to BNNs with arbitrary distributions P on hidden variables h, Q on model parameters θ, and any problem setting (e.g. classification or regression), in practice sampling-free probabilistic backpropagation requires that L n (φ) can be analytically evaluated (or further lower bounded) in terms of φ (L−1) and θ (L−1). This task is nontrivial since it requires redesign of the output layer, i.e. the function of Pr[y|h (L−1), θ (L−1) ]. In this paper, we develop two layers for classification and regression tasks, and present the classification case in this section due to space limit. Since L n (φ) involves the last layer only, we omit the superscripts/subsripts of, x n, y n, and denote them as h, ψ, φ, x, y. with K the number of classes) be the pre-activations of a softmax layer (a.k.a. logits), and φ = s ∈ R + be a scaling factor that adjusts its scale such that are pairwise independent (which holds under mean-field approximation) and The regression case and proofs for both layers are deferred to Appendix C. While Section 3 provides a general solution to learning in BNNs, the solution relies on the ability to perform probabilistic propagation efficiently. To address this, we introduce Bayesian quantized networks (BQNs) -BNNs where both hidden units h (l)'s and model parameters θ (l)'s take discrete values -along with a set of novel algorithms for efficient sampling-free probabilistic propagation in BQNs. For simplicity of exposition, we assume activations and model parameters take values from the same set Q, and denote the degree of quantization as D = |Q|, (e.g. Q = {−1, 1}, D = 2). Lemma 4.1 (Probabilistic Propagation in BQNs). After quantization, the iterative step of probabilistic propagation in Equation is computed with a finite sum instead of an integral: and a categorically distributed h (l) in h (l+1) being categorical as well. The equation holds without any assumption on the operation Notice all distributions in Equation are represented in high-order tensors: Suppose there are I input units, J output units, and K model parameters at the l-th layer, then and h (l+1) ∈ Q J, and their distributions are characterized by respectively. Therefore, each step in probabilistic propagation is a tensor contraction of three tensors, which establishes the duality between BQNs and hierarchical tensor networks . Since tensor contractions are differentiable w.r.t. all inputs, BQNs thus circumvent the difficulties in training QNNs , whose outputs are not differentiable w.r.t. the discrete parameters. This is not surprising: if we consider learning in QNNs as an integer programming (IP) problem, solving its Bayesian counterpart is equivalent to the approach to relaxing the problem into a continuous optimization problem . Complexity of Exact Propagation. The computational complexity to evaluate Equation is exponential in the number of random variables O(D IJK), which is intractable for quantized neural network of any reasonable size. We thus turn to approximations. We propose a principled approximation to reduce the computational complexity in probabilistic propagation in BQNs using tensor CP decomposition, which factors an intractable high-order probability tensor into tractable lower-order factors . In this paper, we consider the simplest rank-1 tensor CP decomposition, where the joint distributions of P and Q are fully factorized into products of their marginal distributions, thus equivalent to the mean-field approximation . With rank-1 CP decomposition on, the tensor contraction in reduces to a standard Tucker contraction where each term of ψ k parameterizes a single categorical variable. In our implementation, we store the parameters in their log-space, i.e. Q(θ Fan-in Number E. In a practical model, for the l-th layer, an output unit h k} according to the connectivity pattern in the layer. We denote the set of dependent input units and parameters for h, and define the fan-in number E for the layer as max j I Complexity of Approximate Propagation. The approximate propagation reduces the computational complexity from O(D IJK) to O(JD E), which is linear in the number of output units J if we assume the fan-in number E to be a constant (i.e. E is not proportional to I). Different types of network layers have different fan-in numbers E, and for those layers with E greater than a small constant, Equation is inefficient since the complexity grows exponential in E. Therefore in this part, we devise fast(er) algorithms to further lower the complexity. Small Fan-in Layers: Direct Tensor Contraction. If E is small, we implement the approximate propagation through tensor contraction in Equation. The computational complexity is O(JD E) as discussed previously. See Appendix D.1 for a detailed discussion. Medium Fan-in Layers: Discrete Fourier Transform. If E is medium, we implement approximate propagation through fast Fourier transform since summation of discrete random variables is equivalent to convolution between their probability mass function. See Appendix D.2 for details. With the fast Fourier transform, the computational complexity is reduced to O(JE 2 D log(ED)). Large Fan-in Layers: Lyapunov Central Limit Theorem. In a typical linear layer, the fan-in E is large, and a super-quadratic algorithm using fast Fourier transform is still computational expensive. Therefore, we derive a faster algorithm based on the Lyapunov central limit theorem (See App D.3) With CLT, the computational complexity is further reduced to O(JED). Remarks: Depending on the fan-in numbers E, we adopt CLT for linear layers with sufficiently large E such as fully connected layers and convolutional layers; DFT for those with medium E such as average pooling layers and depth-wise layers; and direct tensor contraction for those with small E such as shortcut layers and nonlinear layers. In this section, we demonstrate the effectiveness of BQNs on the MNIST, Fashion-MNIST, KM-NIST and CIFAR10 classification datasets. We evaluate our BQNs with both multi-layer perceptron (MLP) and convolutional neural network (CNN) models. In training, each image is augmented by a random shift within 2 pixels (with an additional random flipping for CIFAR10), and no augmentation is used in test. In the experiments, we consider a class of quantized neural networks, with both binary weights and activations (i.e. Q = {−1, 1}) with sign activations σ(·) = sign(·). For BQNs, the distribution parameters φ are initialized by Xavier's uniform initializer, and all models are trained by ADAM optimizer for 100 epochs (and 300 epochs for CIFAR10) with batch size 100 and initial learning rate 10 −2, which decays by 0.98 per epoch. Table 1: Comparison of performance of BQNs against the baseline E-QNN. Each E-QNN is an ensemble of 10 networks, which are trained individually and but make predictions jointly. We report both NLL (which accounts for prediction uncertainty) and 0-1 test error (which doesn't account for prediction uncertainty). All the numbers are averages over 10 runs with different seeds, the standard deviation are exhibited following the ± sign. Training Objective of BQNs. To allow for customized level of uncertainty in the learned Bayesian models, we introduce a regularization coefficient λ in the alternative ELBO proposed in Equation (i.e. a lower bound of the likelihood), and train the BQNs by maximizing the following objective: where λ controls the uncertainty level, i.e. the importance weight of the prior over the training set. Baselines. We compare our BQN against the baseline -Bootstrap ensemble of quantized neural networks (E-QNN). Each member in the ensemble is trained in a non-Bayesian way , and jointly make the prediction by averaging over the logits from all members. Note Evaluation of BQNs. While 0-1 test error is a popular metric to measure the predictive performance, it is too coarse a metric to assess the uncertainty in decision making (for example it does not account for how badly the wrong predictions are). Therefore, we will mainly use the negative log-likelihood (NLL) to measure the predictive performance in the experiments. Once a BQN is trained (i.e. an approximate posterior Q(θ) is learned), we consider three modes to evaluate the behavior of the model: analytic inference (AI), Monte Carlo (MC) sampling and Maximum a Posterior (MAP) estimation: 1. In analytic inference (AI, i.e. our proposed method), we analytically integrate over Q(θ) to obtain the predictive distribution as in the training phase. Notice that the exact NLL is not accessible with probabilistic propagation (which is why we propose an alternative ELBO in Equation), we will report an upper bound of the NLL in this mode. 2. In MC sampling, S sets of model parameters are drawn independently from the posterior posterior θ s ∼ Q(θ), ∀s ∈ [S], and the forward propagation is performed as in (non-Bayesian) quantized neural network for each set θ s, followed by an average over the model outputs. The difference between analytic inference and MC sampling will be used to evaluate (a) the effect of mean-field approximation and (b) the tightness of the our proposed alternative ELBO. 3. MAP estimation is similar to MC sampling, except that only one set of model parameters θ is obtained θ = arg max θ Q(θ). We will exhibit our model's ability to compress a Bayesian neural network by comparing MAP estimation of our BQN with non-Bayesian QNN. Expressive Power and Uncertainty Calibration in BQNs. We report the performance via all evaluations of our BQN models against the Ensemble-QNN in Table 1 and Figure 1. Compared to E-QNNs, our BQNs have significantly lower NLL and smaller predictive error (except for Fashion-MNIST with architecture CNN). As we can observe in Figure 1, BQNs impressively achieve comparable NLL to continuous-valued BNN, with slightly higher test error. As our model parameters only take values {−1, 1}, small degradation in predictive accuracy is expected. Evaluations of Mean-field Approximation and Tightness of the Alternative ELBO. If analytic inference (by probabilistic propagation) were computed exactly, the evaluation metrics would have been equal to the ones with MC sampling (with infinite samples). Therefore we can evaluate the approximations in probabilistic propagation, namely mean-field approximation in Equation and relaxation of the original ELBO in Equation, by measuring the gap between analytic inference and MC sampling. As shown in Figure 2, such gaps are small for all scenarios, which justifies the approximations we use in BQNs. To further decouple these two factors of mean-field approximation and relaxation of the original ELBO, we vary the regularization coefficient λ in the learning objective. For λ = 0 (where the prior term is removed), the models are forced to become deterministic during training. Since the deterministic models do not have mean-field approximation in the forward pass, the gap between analytic inference and MC-sampling reflects the tightness of our alternative ELBO. As λ increases, the gaps increases slightly as well, which shows that the mean-field approximation becomes slightly less accurate with higher learned uncertainty in the model. Table 3: Bayesian Model compression through direct training of Ensemble-QNN vs a Monte-Carlo sampling on our proposed BQN. Each ensemble consists of 5 quantized neural networks, and for fair comparison we use 5 samples for Monte-Carlo evaluation. All the numbers are averages over 10 runs with different seeds, the standard deviation are exhibited following the ± sign. interpreted as another approach to compress a BQN, which reduces the original size to its S/64 (with the same number of bits as an ensemble of S QNNs). In Tables 2 and 3, we compare the models by both approaches to their counterparts (a single QNN for MAP, and E-QNN for MC sampling) trained from scratch as in. For both approaches, our compressed models outperform their counterparts (in NLL). We attribute this to two factors: (a) QNNs are not trained in a Bayesian way, therefore the uncertainty is not well calibrated; and (b) Non-differentiable QNNs are unstable to train. Our compression approaches via BQNs simultaneously solve both problems. We present a sampling-free, backpropagation-compatible, variational-inference-based approach for learning Bayesian quantized neural networks (BQNs). We develop a suite of algorithms for efficient inference in BQNs such that our approach scales to large problems. We evaluate our BQNs by Monte-Carlo sampling, which proves that our approach is able to learn a proper posterior distribution on QNNs. Furthermore, we show that our approach can also be used to learn (ensemble) QNNs by taking maximum a posterior (or sampling from) the posterior distribution. assuming g n (φ) can be (approximately) computed by sampling-free probabilistic propagation as in Section 2. However, this approach has two major limitations: (a) the Bayes' rule needed to be derived case by case, and analytic rule for most common cases are not known yet. (b) it is not compatible to modern optimization methods (such as SGD or ADAM) as the optimization is solved analytically for each data point, therefore difficult to cope with large-scale models. Sampling-based Variational inference (SVI), formulates an optimization problem and solves it approximately via stochastic gradient descent (SGD). The most popular method among all is, Stochastic Gradient Variational Bayes (SGVB), which approximates L n (φ) by the average of multiple samples (; ;). Before each step of learning or prediction, a number of independent samples of the model parameters {θ s} S s=1 are drawn according to the current estimate of Q, i.e. θ s ∼ Q, by which the predictive function g n (φ) and the loss L n (φ) can be approximated by where f n (θ) = Pr[y n |x n, θ] denotes the predictive function given a specific realization θ of the model parameters. The gradients of L n (φ) can now be approximated as This approach has multiple drawbacks: (a) Repeated sampling suffers from high variance, besides being computationally expensive in both learning and prediction phases; (b) While g n (φ) is differentiable w.r.t. φ, f n (θ) may not be differentiable w.r.t. θ. One such example is quantized neural networks, whose backpropagation is approximated by straight through estimator (Our approach considers a wider scope of problem settings, where the model could be stochastic, i.e.] is an arbitrary function. considers the case that all parameters θ are Gaussian distributed, whose sampling-free probabilistic propagation requires complicated approximation . Quantized Neural Networks These models can be categorized into two classes: Partially quantized networks, where only weights are discretized ; Fully quantized networks, where both weights and hidden units are quantized (; ; ; ;). While both classes provide compact size, low-precision neural network models, fully quantized networks further enjoy fast computation provided by specialized bit-wise operations. In general, quantized neural networks are difficult to train due to their non-differentiability. Gradient descent by backpropagation is approximated by either straight-through estimators or probabilistic methods (; ;). Unlike these papers, we focus on Bayesian learning of fully quantized networks in this paper. Optimization of quantized neural networks typically requires dedicated loss function, learning scheduling and initialization. For example, considers pre-training of a continuous-valued neural network as the initialization. Since our approach considers learning from scratch (with an uniform initialization), the performance could be inferior to prior works in terms of absolute accuracy. Tensor Networks and Tensorial Neural Networks Tensor networks (TNs) are widely used in numerical analysis , quantum physiscs (Orús, 2014), and recently machine learning (; 2017) to model interactions among multi-dimensional random objects. Various tensorial neural networks (TNNs) have been proposed that reduce the size of neural networks by replacing the linear layers with TNs. Recently, points out the duality between probabilistic graphical models (PGMs) and TNs. I.e. there exists a bijection between PGMs and TNs. Our paper advances this line of thinking by connecting hierarchical Bayesian models (e.g. Bayesian neural networks) and hierarchical TNs. The problem settings of general Bayesian model and Bayesian neural networks for supervised learning are illustrated in Figures 3a and 3b using graphical models.... General Bayesian model Formally, the graphical model in Figure 3a implies the joint distribution of the model parameters θ, the observed dataset D = {(x n, y n)} N n=1 and any unseen data point (x, y) is factorized as follows: [y|x, θ]. In other words, we assume that the samples (x n, y n)'s (and unseen data point (x, y)) are are identical and independent distributed according to the same data distribution; and x n (or x) and θ together predict the output y n (or y) according to the same conditional distribution. Notice that the factorization above also implies the following equations: With these implications, the posterior predictive distribution Pr[y|x, D] can now expanded as: dθ where we approximate the posterior distribution Pr[θ|D] by a parameterized distribution Q(θ; φ). const. where L n (φ) is the expected log-likelihood, which reflects the predictive performance of the Bayesian model on the data point (x n, y n); and R(φ) is the KL-divergence between Q(θ; φ) and its prior Pr[θ], which reduces to entropy H(Q) if the prior of θ follows a uniform distribution. Hierarchical Bayesian Model A Bayesian neural network can be considered as a hierarchical Bayesian model depicted in Figure 3b, which further satisfies the following two assumptions: Assumption B.1 (Independence of Model Parameters θ (l) ). The approximate posterior Q(θ; φ) over the model parameters θ are partitioned into L disjoint and statistically independent layers {θ (l) } L−1 l=0 (where each φ (l) parameterizes θ (l) in the l-th layer) such that: satisfy the Markov property that h (l+1) depends on the input x only through its previous layer h (l): where we use short-hand notations h (: l) and θ (: l) to represent the sets of previous layers {h (k) } l k=0 and {θ (k) } l k=0. For consistency, we denote h = x and h (L) = y. Proof of probabilistic prorogation Based on the two assumptions above, we provide a proof for probabilistic propagation in Equation as follows: C ALTERNATIVE EVIDENCE LOWER BOUND AND ITS ANALYTIC FORMS C.1 ALTERNATIVE EVIDENCE LOWER BOUND (PROOF FOR THEOREM 3.1) The steps to prove the inequality almost follow the ones for probabilistic propagation above: where the key is the Jensen's inequality is not random variable (typical for an output layer), L n (φ) can be simplified as: where we write Pr[(L−1) can be obtained by differentiating over Equation, while other gradients ∂L n (φ)/φ (: L−2) further obtained by chain rule: which requires us to compute ∂L n (φ)/∂ψ (L−1) and ∂ψ can be derived from Equation, ∂ψ (L−1) /∂φ (: L−2) can be obtained by backpropagating outputs of the (L − 2) th layer obtained from probabilistic propagation in Equation. In other words: is a function of all parameters from previous layers φ (: L−2), and if each step can be obtained by iterative chain rule. In this part, we first prove the alternative evidence lower bound (ELBO) for Bayesian neural networks with softmax function as their last layers. Subsequently, we derive the corresponding backpropagation rule for the softmax layer. Finally, we show a method based on Taylor's expansion to approximately evaluate a softmax layer without Monte Carlo sampling. Theorem C.1 (Analytic Form of L n (φ) for Classification). Let h ∈ R K (with K the number of classes) be the pre-activations of a softmax layer (a.k.a. logits), and φ = s ∈ R + be a scaling factor that adjusts its scale such that are pairwise independent (which holds under mean-field approximation) and ) and s is a deterministic parameter. Then L n (φ) can be further upper bound by the following analytic form: Proof. The lower bound follows by plugging Pr [y|h, s] and Pr[h k |x] into Equation. where the last equation follows where the under-braced term is unity since it takes the form of Gaussian distribution. From Equation to, we use the Jensen's inequality to achieve a lower bound for integral of log-sum-exp. The bound can be tighten with advanced techniques in. Derivatives of L n (φ) in To use probabilistic backpropagation to obtain the gradients w.r.t. the parameters from previous layers, we first need to obtain the derivatives w.r.t. Furthermore, the scale s can be (optionally) updated along with other parameters using the gradient Prediction with Softmax Layer Once we learn the parameters for the Bayesian neural network, in principle we can compute the predictive distribution of y by evaluating the following equation: where we denote the softmax function as Unfortunately, the equation above can not be computed in closed form. The most straight-forward work-around is to approximate the integral by Monte Carlo sampling: for each h k we draw S samples {h independently and compute the prediction: Despite its conceptual simplicity, Monte Carlo method suffers from expensive computation and high variance in estimation. Instead, we propose an economical estimate based on Taylor's expansion. First, we expand the function c (h) by Taylor's series at the point µ (up to the second order): Before we derive the forms of these derivatives, we first show the terms of odd orders do not contribute to the expectation. For example, if c (h) is approximated by its first two terms (i.e. a linear function), Equation can be written as where the second term is zero by the symmetry of Pr[h k |x] around µ k (or simply the definition of µ k 's). Therefore, the first-order approximation exactly in a (deterministic) softmax function of the mean vector µ. In order to incorporate the variance into the approximation, we will need to derive the exact forms of the derivatives of c (h). Specifically, the first-order derivatives are obtained from the definition of c (h). and subsequently the second-order derivatives from the first ones: with these derivatives we can compute the second-order approximation as The equation above can be further written in vector form as: In this part, we develop an alternative evidence lower bound (ELBO) for Bayesian neural networks with Gaussian output layers, and derive the corresponding gradients for backpropagation. Despite the difficulty to obtain an analytical predictive distribution for the output, we show that its central moments can be easily computed given the learned parameters. Theorem C.2 (Analytic Form of L n (φ) for Regression). Let h ∈ R I be the output of last hidden layer (with I the number of hidden units), and φ = (w, s) ∈ R I × R + be the parameters that define the predictive distribution over output y as Suppose the hidden units {h k} K k=1 are pairwise independent (which holds under mean-field approximation), and each h i has mean µ i and variance ν i, then L n (φ) takes an analytic form: where (Proof. The Equation is obtained by plugging Pr[y|h; w, s] into Equation. where the long summation in the first term can be further simplified with notations of µ and ν: where w •2 denotes element-wise square, i.e. Derivatives of L n (φ) in Equation It is not difficult to show that the gradient of L n (φ) can be backpropagated through the last layer. by computing derivatives of L n (φ) w.r.t. µ and ν: Furthermore, the parameters {w, s} can be updated along with other parameters with their gradients: Prediction with Gaussian Layer Once we determine the parameters for the last layer, in principle we can compute the predictive distribution Pr[y|x] for the output y given the input x according to Unfortunately, exact computation of the equation above for arbitrary output value y is intractable in general. However, the central moments of the predictive distribution Pr[y|x] are easily evaluated. Consider we interpret the prediction as y = w h +, where ∼ N (0, s), its mean and variance can be easily computed as Furthermore, if we denote the (normalized) skewness and kurtosis of h i as γ i and κ i: Then the (normalized) skewness and kurtosis of the prediction y are also easily computed with the In this section, we present fast(er) algorithms for sampling-free probabilistic propagation (i.e. evaluating Equation). According to Section 4, we divide this section into three parts, each part for a specific range of fan-in numbers E. If E is small, tensor contraction in Equation is immediately applicable. Representative layers of small E are shortcut layer (a.k.a. skip-connection) and what we name as depth-wise layers. Shortcut Layer With a skip connection, the output h (l+1) is an addition of two previous layers h (l) and h (m). Therefore and the distribution of h (l+1) can be directly computed as Depth-wise Layers In a depth-wise layer, each output unit h Depth-wise layers include dropout layers (where θ (l) are dropout rates), nonlinear layers (where θ (l) are threshold values) or element-wise product layers (where θ (l) are the weights). For both shortcut and depth-wise layers, the time complexity is O(JD 2) since E <= 2. In neural networks, representative layers with medium fan-in number E are pooling layers, where each output unit depends on a medium number of input units. Typically, the special structure of pooling layers allows for faster algorithm than computing Equation directly. Max and Probabilistic Pooling For each output, a max pooling layer picks the maximum value from corresponding inputs, i.e. h i, while a probabilistic pooling layer selects the value the inputs following a categorical distribution, i.e. Pr[h For both cases, the predictive distribution of h (l+1) j can be computed as Prob: P (h where is the culminative mass function of P . Complexities for both layers are O(ID). Average Pooling and Depth-wise Convolutional Layer Both layers require additions of a medium number of inputs. We prove a convolution theorem for discrete random variables and show that discrete Fourier transform (DFT) (with fast Fourier transform (FFT)) can accelerate the additive computation. We also derive its backpropagation rule for compatibility of gradient-based learning. Then C v (f) is the element-wise product of all Fourier transforms C ui (f), i.e. Proof. We only prove the theorem for two discrete random variable, and the extension to multiple variables can be proved using induction. Now consider, where b = b 1 + b 2 and B = B 1 + B 2. Denote the probability vectors of u 1, u 2 and v as P 1 ∈ B1−b1, P 2 ∈ B2−b2 and P ∈ B−b respectively, then the entries in P are computed with P 1 and P 2 by standard convolution as follows: The relation above is usually denoted as P = P 1 * P 2, where * is the symbol for convolution. Now define the characteristic functions C, C 1, and C 2 as the discrete Fourier transform (DFT) of the probability vectors P, P 1 and P 2 respectively: where R controls the resolution of the Fourier transform (typically chosen as R = B − b + 1, i.e. the range of possible values). In this case, the characteristic functions are complex vectors of same length R, i.e. C, C 1, C 2 ∈ C R, and we denote the (functional) mappings as C = F(P) and C i = F i (P i). Given a characteristic function, its original probability vector can be recovered by inverse discrete Fourier transform (IDFT): which we denote the inverse mapping as P = F −1 (C) and P i = F −1 i (C i). Now we plug the convolution in Equation into the characteristic function C(f) in (92a) and rearrange accordingly: The equation above can therefore be written as C = C 1 •C 2, where we use • to denote element-wise product. Thus, we have shown summation of discrete random variables corresponds to element-wise product of their characteristic functions. With the theorem, addition of E discrete random variables can be computed efficiently as follows where F denotes the Fourier transforms in Equations (93a) and (93b). If FFT is used in computing all DFT, the computational complexity of Equation is O(ER log R) = O(E 2 D log(ED)) (since R = O(ED)), compared to O(D E) with direct tensor contraction. Backpropagation When fast Fourier transform is used to accelerate additions in Bayesian quantized network, we need to derive the corresponding backpropagation rule, i.e. equations that relate ∂L/∂P to {∂L/∂P i} I i=1. For this purpose, we break the computation in Equation into three steps, and compute the derivative for each of these steps. where in (100b) we use C/C i to denote element-wise division. Since P i lies into real domain, we need to project the gradients back to real number in (100c). Putting all steps together: In this part, we show that Lyapunov central limit approximation (Lyapunov CLT) accelerates probabilistic propagation in linear layers. For simplicity, we consider fully-connected layer in the derivations, but the can be easily extended to types of convolutional layers. We conclude this part by deriving the corresponding backpropagation rules for the algorithm. Linear Layers Linear layers (followed by a nonlinear transformations σ(·)) are the most important building blocks in neural networks, which include fully-connected and convolutional layers. A linear layer is parameterized by a set of vectors θ (l)'s, and maps where u. Let v j = σ(ṽ j) = σ(i∈I(j) θ ji u i ) be an activation of a linear layer followed by nonlinearity σ. Suppose both inputs {u i} i∈I and parameters {θ ji} i∈I(j) have bounded variance, then for sufficiently large |I(j)|, the distribution ofṽ j converges to a Gaussian distribution N (μ j,ν j) with mean and variance as Published as a conference paper at ICLR 2020 for CNN, we use a 4-layers network with two 5 × 5 convolutional layers with 64 channels followed by 2 × 2 average pooling, and two fully-connected layers with 1024 hidden units. For CIFAR10, we evaluate our models on a smaller version of VGG , which consists of 6 convolutional layers and 2 fully-connected layers: 2 x 128C3 -MP2 -2 x 256C3 -MP2 -2 x 512C3 -MP2 -1024FC -SM10. Table 4: Performance of different networks in terms of RMSE. The numbers for BQN are averages over 10 runs with different seeds, the standard deviation are exhibited following the ± sign. The for PBP, EBP are from , and the one for NPN is from . 10

Represent the TLDR for finding the corresponding scientific article

We propose Bayesian quantized networks, for which we learn a posterior distribution over their quantized parameters.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.