Question:
represent the text to find a very short summary (1-2 sentences max)

E.g.
The biological plausibility of the backpropagation algorithm has long been doubted by neuroscientists. Two major reasons are that neurons would need to send two different types of signal in the forward and backward phases, and that pairs of neurons would need to communicate through symmetric bidirectional connections. We present a simple two-phase learning procedure for fixed point recurrent networks that addresses both these issues. In our model, neurons perform leaky integration and synaptic weights are updated through a local mechanism. Our learning method extends the framework of Equilibrium Propagation to general dynamics, relaxing the requirement of an energy function. As a consequence of this generalization, the algorithm does not compute the true gradient of the objective function, but rather approximates it at a precision which is proven to be directly related to the degree of symmetry of the feedforward and feedback weights. We show experimentally that the intrinsic properties of the system lead to alignment of the feedforward and feedback weights, and that our algorithm optimizes the objective function. Deep learning BID18 is the de-facto standard in areas such as computer vision BID17, speech recognition and machine translation BID3. These applications deal with different types of data and share little in common at first glance. Remarkably, all these models typically rely on the same basic principle: optimization of objective functions using the backpropagation algorithm. Hence the question: does the cortex in the brain implement a mechanism similar to backpropagation, which optimizes objective functions?The backpropagation algorithm used to train neural networks requires a side network for the propagation of error derivatives, which is vastly seen as biologically implausible BID7. One hypothesis, first formulated by , is that error signals in biological networks could be encoded in the temporal derivatives of the neural activity and propagated through the network via the neuronal dynamics itself, without the need for a side network. Neural computation would correspond to both inference and error back-propagation. This work also explores this idea. The framework of Equilibrium Propagation BID29 requires the network dynamics to be derived from an energy function, enabling computation of an exact gradient of an objective function. However, in terms of biological realism, the requirement of symmetric weights between neurons arising from the energy function is not desirable. The work presented here extends this framework to general dynamics, without the need for energy functions, gradient dynamics, or symmetric connections. Our approach is the following. We start from classical models in neuroscience for the dynamics of the neuron's membrane voltage and for the synaptic plasticity (section 3). Unlike in the Hopfield model BID16, we do not assume pairs of neurons to have symmetric connections. We then describe an algorithm for supervised learning based on these models (section 4) with minimal extra assumptions. Our model is based on two phases: at prediction time, no synaptic changes occur, whereas a local update rule becomes effective when the targets are observed. The proposed update mechanism is compatible with spike-timing-dependent plasticity, which supposedly governs synaptic changes in biological neural systems. Finally, we show that the proposed algorithm has the desirable machine learning property of optimizing an objective function (section 5). We show this experimentally (Figure 3) and we provide the beginning for a theoretical explanation. Historically, models based on energy functions and/or gradient dynamics have represented a key subject of neural network research. Their mathematical properties often allow for a simplified analysis, in the sense that there often exists an elegant formula or algorithm for computing the gradient of the objective function BID0 BID24 BID29. However, we argue in this section that 1. due to the energy function, such models are very restrictive in terms of dynamics they can model -for instance the Hopfield model requires symmetric weights, 2. machine learning algorithms do not require computation of the gradient of the objective function, as shown in this work and the work of BID19.In this work, we propose a simple learning algorithm based on few assumptions. To this end, we relax the requirement of the energy function and, at the same time, we give up on computing the gradient of the objective function. We believe that, in order to make progress in biologically plausible machine learning, dynamics more general than gradient dynamics should be studied. As discussed in section 6, another motivation for studying more general dynamics is the possible implementation of machine learning algorithms, such as our model, on analog hardware: analog circuits implement differential equations, which do not generally correspond to gradient dynamics. Most dynamical systems observed in nature cannot be described by gradient dynamics. A gradient field is a very special kind of vector field, precisely because it derives from a primitive scalar function. The existence of a primitive function considerably limits the "number of degrees of freedom" of the vector field and implies important restrictions on the dynamics. In general, a vector field does not derive from a primitive function. In particular, the dynamics of the leaky integrator neuron model studied in this work (Eq. 1) is not a gradient dynamics, unless extra (biologically implausible) assumptions are made, such as exact symmetry of synaptic weights (W ij = W ji) in the case of the Hopfield model. Machine learning relies on the basic principle of optimizing objective functions. Most of the work done in deep learning has focused on optimizing objective functions by gradient descent in the weight space (thanks to backpropagation). Although it is very well known that following the gradient is not necessarily the best option -many optimization methods based on adaptive learning rates for individual parameters have been proposed such as and Adagrad BID9 -almost all proposed optimization methods rely on computing the gradient, even if they do not follow the gradient. In the field of deep learning, "computing the gradient" has almost become synonymous with "optimizing".In fact, in order to optimize a given objective function, not only following the gradient unnecessary, but one does not even need to compute the gradient of that objective function. A weaker sufficient condition is to compute a direction in the parameter space whose scalar product with the gradient is negative, without computing the gradient itself. A major step forward was achieved by BID19. One of the contributions of their work was to dispel the long-held assumption that a learning algorithm should compute the gradient of an objective function in order to be sound. Their algorithm computes a direction in the parameter space that has at first sight little to do with the gradient of the objective function. Yet, their algorithm "learns" in the sense that it optimizes the objective function. By giving up on the idea of computing the gradient of the objective function, a key aspect rendering backpropagation biologically implausible could be fixed, namely the weight transport problem. The work presented here is along the same lines. We give up on the idea of computing the gradient of the objective function, and by doing so, we get rid of the biologically implausible symmetric connections required in the Hopfield model. In this sense, the "weight transport" problem in the backpropagation algorithm appears to be similar, at a high level, to the requirement of symmetric connections in the Hopfield model. We suggest that in order to make progress in biologically plausible machine learning, it might be necessary to move away from computing the true gradients in the weight space. An important theoretical effort to be made is to understand and characterize the dynamics in the weight space that optimize objective functions. The set of such dynamics is of course much larger than the tiny subset of gradient dynamics. We denote by s i the averaged membrane voltage of neuron i across time, which is continuous-valued and plays the role of a state variable for neuron i. We also denote by ρ(s i) the firing rate of neuron i. We suppose that ρ is a deterministic function (nonlinear activation) that maps the averaged voltage s i to the firing rate ρ(s i). The synaptic strength from neuron j to neuron i is denoted by W ij. In biological neurons a classical model for the time evolution of the membrane voltage s i is the rate-based leaky integrator neuron model, in which neurons are seen as performing leaky temporal integration of their past inputs BID8: DISPLAYFORM0 Unlike energy-based models such as the Hopfield model BID16 that assume symmetric connections between neurons, in the model studied here the connections between neurons are not tied. Thus, our model is described by a directed graph, whereas the Hopfield model is best regarded as an undirected graph (Figure 1).(a) The network model studied here is best represented by a directed graph.(b) The Hopfield model is best represented by an undirected graph. Figure 1: From the point of view of biological plausibility, the symmetry of connections in the Hopfield model is a major drawback (1b). The model that we study here is, like a biological neural network, a directed graph (1a).3.2 SPIKE-TIMING DEPENDENT PLASTICITY Spike-Timing Dependent Plasticity (STDP) is considered a key mechanism of synaptic change in biological neurons BID21 BID11 BID22. STDP is often conceived of as a spike-based process which relates the change in the synaptic weight W ij to the timing difference between postsynaptic spikes (in neuron i) and presynaptic spikes (in neuron j) BID5. In fact, both experimental and computational work suggest that postsynaptic voltage, not postsynaptic spiking, is more important for driving LTP (Long Term Potentiation) and LTD (Long Term Depression) BID6 BID20.Similarly, have shown in simulations that a simplified Hebbian update rule based on pre-and post-synaptic activity can functionally reproduce STDP: DISPLAYFORM1 Throughout this paper we will refer to this update rule (Eq. 2) as "STDP-compatible weight change" and propose a machine learning justification for such an update rule. Let s = (s 1, s 2, . . .) be the global state variable and parameter W the matrix of connection weights W ij. We write µ(W, s) the vector whose components are defined as DISPLAYFORM0 defining a vector field over the neurons state space, indicating in which direction each neuron's activity changes: DISPLAYFORM1 Since ρ(s j) = ∂µi ∂Wij (W, s), the weight change Eq. 2 can also be expressed in terms of µ in the form dW ij ∝ ∂µi ∂Wij (W, s)ds i. Note that for all i = i we have ∂µ i ∂Wij = 0 since to each synapse W ij corresponds a unique post-synaptic neuron s i. Hence dW ij ∝ ∂µ ∂Wij (W, s) · ds. We rewrite the STDP-compatible weight change in the more concise form DISPLAYFORM2 The framework and the algorithm in their general forms are described in Appendix A.To illustrate our algorithm, we consider here the supervised setting in which we want to predict an output y given an input x. We describe a simple two-phase learning procedure based on the dynamics Eq. 4 and Eq. 5 for the state and the parameter variables. This algorithm is similar to the one proposed by BID29, but here we do not assume symmetric weights between neurons. Note that similar algorithms have also been proposed by; BID13 or more recently by BID23. Our contribution in this work are theoretical insights into why the proposed algorithm works. In the supervised setting studied here, the units of the network are split in two sets: the inputs x whose values are always clamped, and the dynamically evolving units h (the neurons activity, indicating the state of the network), which themselves include the hidden layers (h 1 and h 2 here) and an output layer (h 0 here), as in Figure 2. In this context the vector field µ is defined by its components µ 0, µ 1 and µ 2 on h 0, h 1 and h 2 respectively, as follows: DISPLAYFORM0 Here the scalar function ρ is applied elementwise to the components of the vectors. The neurons h follow the dynamics DISPLAYFORM1 In this section and the next we use the notation h rather than s for the state variable. The layer h 0 plays the role of the output layer where the prediction is read. The target outputs, denoted by y, have the same dimension as the output layer h 0. The discrepancy between the output units h 0 and the targets y is measured by the quadratic cost function DISPLAYFORM2 Unlike in the continuous Hopfield model, here the feed-forward and feedback weights are not tied, and in general the state dynamics Eq. 9 is not guaranteed to converge to a fixed point. However we observe experimentally that the dynamics almost always converges. We will see in section 5 that, for a whole set of values of the weight matrix W. the dynamics of the neurons h converges. Assuming this condition to hold, the dynamics of the neurons converge to a fixed point which we denote by h 0 (beware not to confuse with the notation for the output units h 0). The prediction h 0 0 is then read out on the output layer and compared to the actual target y. The objective function (for a single training case (x, y)) that we aim to minimize is the cost at the fixed point h 0, which we write DISPLAYFORM3 Note that this objective function is the same as the one proposed by BID1 BID28. Their method to optimize J is to compute the gradient of J thanks to an algorithm which they call "Recurrent Backpropagation". Other methods related to Recurrent Backpropagation exist to compute the gradient of J -in particular the "adjoint method", "implicit differentiation" and "Backprop Through Time". These methods are biologically implausible, as argued in Appendix B.Here our approach to optimize J is to give up on computing the true gradient of J and, instead, we propose a simple algorithm based only on the leaky integrator dynamics (Eq. 4) and the STDPcompatible weight change (Eq. 5). We will show in section 5 that our algorithm computes a proxy for the gradient of J. Also, note that in its general formulation, our algorithm applies to any vector field µ and cost function C (Appendix A) The idea of Equilibrium Propagation BID29 is to see the cost function C (Eq. 10) as an external potential energy for the output units h 0, which can drive them towards their target y. Following the same idea we define the "extended vector field" µ β as DISPLAYFORM0 and we redefine the dynamics of the state variable h as DISPLAYFORM1 The real-valued scalar β ≥ 0 controls whether the output h 0 is pushed towards the target y or not, and by how much. We call β the "influence parameter" or "clamping factor".The differential equation of motion Eq. 13 can be seen as a sum of two "forces" that act on the temporal derivative of the state variable h. Apart from the vector field µ that models the interactions between neurons within the network, an "external force" −β ∂C ∂h is induced by the external potential βC and acts on the output neurons: DISPLAYFORM2 DISPLAYFORM3 The form of Eq. 14 suggests that when β = 0, the output units h 0 are not sensitive to the targets y from the outside world. In this case we say that the network is in the free phase (or first phase). When β > 0, the "external force" drives the output units h 0 towards the target y. When β 0 (small positive value), we say that the network is in the weakly clamped phase (or second phase). Also, note that the case β → ∞, not studied here, would correspond to fully clamped outputs. We propose a simple two-phase learning procedure, similar to the one proposed by BID29. In the first phase of training, the inputs are set (clamped) to the input values. The state variable (all the other neurons) follows the dynamics Eq. 9 (or equivalently Eq. 13 with β = 0) and the output units are free. We call this phase the free phase, as the system relaxes freely towards the free fixed point h 0 without any external constraints on his output neurons. During this phase, the synaptic weights are unchanged. Figure 2: Input x is clamped. Neurons h include "hidden layers" h 2 and h 1, and "output layer" h 0 that corresponds to the layer where the prediction is read. Target y has the same dimension as h 0. The clamping factor β scales the "external force" −β ∂C ∂h that attracts the output h 0 towards the target y. In the second phase, the influence parameter β takes on a small positive value β 0. The state variable follows the dynamics Eq. 13 for that new value of β, and the synaptic weights follow the STDP-compatible weight change Eq. 5. This phase is referred to as the weakly clamped phase. The novel "external force" −β ∂C ∂h in the dynamics Eq. 13 acts on the output units and drives them towards their targets (Eq. 14). This force models the observation of y: it nudges the output units h 0 from their free fixed point value in the direction of their targets. Since this force only acts on the output layer h 0, the other hidden layers (h i with i > 0) are initially at equilibrium at the beginning of the weakly clamped phase. The perturbation caused at the output layer will then propagate backwards along the layers of the network, giving rise to "back-propagating" error signals. The network eventually settles to a new nearby fixed point, corresponding to the new value β 0, termed weakly clamped fixed point and denoted h β. Our model assumes that the STDP-compatible weight change (Eq. 5) occurs during the second phase of training (weakly clamped phase) when the network's state moves from the free fixed point h 0 to the weakly clamped fixed point h β. Normalizing by a factor β and letting β → 0, we get the update rule ∆W ∝ ν(W) for the weights, where ν(W) is the vector defined as DISPLAYFORM0 The vector ν(W) has the same dimension as W. Formally ν is a vector field in the weight space. It is shown in section 5 that ν(W) is a proxy to the gradient ∂J ∂W. The effectiveness of the proposed method is demonstrated through experimental studies (Figure 3). In this section, we attempt to understand why the proposed algorithm is experimentally found to optimize the objective function J (Figure 3). We say that W is a "good parameter" if:1. for any initial state for the neurons, the state dynamics dh dt = µ (W, x, h) converges to a fixed point -a condition required for the algorithm to be correctly defined, 2. the scalar product ∂J ∂W · ν(W) at the point W is negative -a desirable condition for the algorithm to optimize the objective function J.Experiments show that the dynamics of h (almost) always converges to a fixed point and that J consistently decreases (Figure 3). This means that, during training, as the parameter W follows the update rule ∆W ∝ ν(W), all values of W that the network takes are "good parameters". In this section we attempt to explain why. ∂J ∂W AND ν Theorem 1. The gradient of J can be expressed in terms of µ and C as DISPLAYFORM0 Similarly, the vector field ν (Eq. 16) is equal to DISPLAYFORM1 In these expressions, all terms are evaluated at the fixed point h 0. and that the angle between these two vectors is directly linked to the "degree of symmetry" of the Jacobian of µ.An important particular case is the setting of Equilibrium Propagation BID29, in which the vector field µ is a gradient field µ = − ∂E ∂h, meaning that it derives from an energy function E. In this case the Jacobian of µ is symmetric since it is the Hessian of E. Indeed DISPLAYFORM0. Therefore, Theorem 1 shows that ν is also a gradient field, namely the gradient of the objective function J, that is ν = − ∂J ∂W. Note that in this setting the set of "good parameters" is the entire weight space -for all W, the dynamics dh dt = − ∂E ∂h (W, h) converges to an energy minimum, and W converges to a minimum of J since ∆W ∝ − ∂J ∂W. We argue that the set of "good parameters" covers a large proportion of the weight space and that they contain the matrices W that present a form of symmetry or "alignment". In the next subsection, we discuss how this form of symmetry may arise from the learning procedure itself. Figure 3: Example system trained on the MNIST dataset, as described in Appendix C. The objective function is optimized: the training error decreases to 0.00% in around 70 epochs. The generalization error is about 2%. Right: A form of symmetry or alignment arises between feedforward and feedback weights W k,k+1 and W k+1,k in the sense that tr(W k,k+1 · W k+1,k) > 0. This architecture uses 3 hidden layers each of dimension 512. Experiments show that a form of symmetry between feedforward and feedback weights arises from the learning procedure itself (Figure 3). Although the causes for this phenomenon aren't understood very well yet, it is worth pointing out that similar observations have been made in previous work and different settings. A striking example is the following one. A major argument against the plausibility of backpropagation in feedforward nets is the weight transport problem: the signals sent forward in the network and those sent backward use the same connections. BID19 have observed that, in the backward pass, (back)propagating the error signals through fixed random feedback weights (rather than the transpose of the feedforward weights) does not harm learning. Moreover, the learned feedforward weights W k,k+1 tend to'align' with the fixed random feedback weights W k+1,k in the sense that the trace of W k,k+1 · W k+1,k is positive. Denoising autoencoders without tied weights constitute another example of learning algorithms where a form of symmetry in the weights has been observed as learning goes on BID31.The theoretical from BID2 also shows that, in a deep generative model, the transpose of the generative weights perform approximate inference. They show that the symmetric solution minimizes the autoencoder reconstruction error between two successive layers of rectifying linear units. Our approach provides a basis for implementing machine learning models in continuous-time systems, while requirements regarding the actual dynamics are reduced to a minimum. This means that the model applies to a large class of physical realizations of vectorfield dynamics, including analog electronic circuits. Implementations of recurrent networks based on analog electronics have been proposed in the past, e.g. BID13, however, these models typically required circuits and associated dynamics to adhere to an exact theoretical model. With our framework, we provide a way of implementing a learning system on a physical substrate without even knowing the exact dynamics or microscopic mechanisms that give rise to it. Thus, this approach can be used to analog electronic system end-to-end, without having to worry about exact device parameters and inaccuracies, which inevitably exist in any physical system. Instead of approximately implementing idealized computations, the actual analog circuit, with all its individual device variations, is trained to perform the task of interest. Thereby, the more direct implementation of the dynamics might in advantages in terms of speed, power, and scalability, as compared to digital approaches. Our model demonstrates that biologically plausible learning in neural networks can be achieved with relatively few assumptions. As a key contribution, in contrast to energy-based approaches such as the Hopfield model, we do not impose any symmetry constraints on the neural connections. Our algorithm assumes two phases, the difference between them being whether synaptic changes occur or not. Although this assumption begs for an explanation, neurophysiological findings suggest that phase-dependent mechanisms are involved in learning and memory consolidation in biological systems. Theta waves, for instance, generate neural oscillatory patterns that can modulate the learning rule or the computation carried out by the network BID26. Furthermore, synaptic plasticity, and neural dynamics in general, are known to be modulated by inhibitory neurons and dopamine release, depending on the presence or absence of a target signal. BID10; BID27.In its general formulation (Appendix A), the work presented in this paper is an extension of the framework of BID29 to general dynamics. This is achieved by relaxing the requirement of an energy function. This generalization comes at the cost of not being able to compute the (true) gradient of the objective function but, rather a direction in the weight space which is related to it. Thereby, precision of the approximation of the gradient is directly related to the "alignment" between feedforward and feedback weights. Even though the exact underlying mechanism is not fully understood yet, we observe experimentally that during training the weights symmetrize to some extent, as has been observed previously in a variety of other settings BID19 BID31 BID2. Our work shows that optimization of an objective function can be achieved without ever computing the (true) gradient. More thorough theoretical analysis needs to be carried out to understand and characterize the dynamics in the weight space that optimize objective functions. Naturally, the set of all such dynamics is much larger than the tiny subset of gradient-based dynamics. Our framework provides a means of implementing learning in a variety of physical substrates, whose precise dynamics might not even be known exactly, but which simply have to be in the set of sup-ported dynamics. In particular, this applies to analog electronic circuits, potentially leading to faster, more efficient, and more compact implementations. In this Appendix, we present the framework and the algorithm in their general formulations and we prove the theoretical . We consider a physical system specified by a state variable s and a parameter variable θ. The system is also influenced by an external input v, e.g. in the supervised setting v = (x, y) where y is the target that the system wants to predict given x. Let s → µ(θ, v, s) be a vector field in the state space and C(θ, v, s) a cost function. We assume that the state dynamics induced by µ converges to a stable fixed point s 0 θ,v, corresponding to the "prediction" from the model and characterized by DISPLAYFORM0 The objective function that we want to optimize is the cost at the fixed point DISPLAYFORM1 Note the distinction between J and C: the cost function is defined for any state s whereas the objective function is the cost at the fixed point. The training objective (for a single data sample v) is find arg min DISPLAYFORM2 Equivalently, the training objective can be reformulated as a constrained optimization problem: DISPLAYFORM3 where the constraint µ (θ, v, s) = 0 is the fixed point condition. All traditional methods to compute the gradient of J (adjoint method, implicit differentiation, Recurrent Backpropagation and Backpropagation Through Time or BPTT) are thought to be biologically implausible. Our approach is to give up on computing the gradient of J and let the parameter variable θ follow a vector field ν in the parameter space which is "close" to the gradient of J.Before defining ν we first introduce the "extended vector field" DISPLAYFORM4 where β is a real-valued scalar called "influence parameter". Then we extend the notion of fixed point for any value of β. For any β we define the β-fixed point s β θ,v such that DISPLAYFORM5 Under mild regularity conditions on µ and C, the implicit function theorem ensures that, for a fixed data sample v, the funtion (θ, β) → s β θ,v is differentiable. Now for every θ and v we define the vector ν(θ, v) in the parameter space as DISPLAYFORM6 As shown in section 4, the second term on the right hand side can be estimated in a biologically realistic way thanks to a two-phase training procedure. and ∂s DISPLAYFORM7 Proof of Lemma 2. First we differentiate the fixed point equation Eq. 27 with respect to θ: DISPLAYFORM8 Rearranging the terms we get Eq. 28. Similarly we differentiate the fixed point equation Eq. 27 with respect to β: DISPLAYFORM9 Rearranging the terms we get Eq. 29.Theorem 3. The gradient of the objective function is equal to DISPLAYFORM10 and the vector field ν is equal to DISPLAYFORM11 All the factors on the right-hand sides of Eq. 32 and Eq. 33 are evaluated at the fixed point s 0 θ.Proof of Theorem 3. Let us compute the gradient of the objective function with respect to θ. Using the chain rule of differentiation we get DISPLAYFORM12 Hence Eq. 32 follows from Eq. 28 evaluated at β = 0. Similarly, the expression for the vector field ν (Eq. 33) follows from its definition (Eq. 26), the identity Eq. 29 evaluated at β = 0 and the fact that We finish by stating and proving a last . Consider the setting introduced in section 4 with the quadratic cost function C = 1 2 y − h 0 2. In the weakly clamped phase, the "external influence" −β (y − h 0) added to the vector field µ (with β 0) slightly attracts the output state h 0 to the target y. It is intuitively clear that the weakly clamped fixed point is better than the free fixed point in terms of prediction error. Proposition 5 below generalizes this property to any vector field µ and any cost function C.Proposition 4. Let s 0 be a stable fixed point of the vector field s → µ(s), in the sense that s − s 0 · µ (s) < 0 for s in the neighborhood of s 0 (i.e. the vector field at s points towards s 0). Then the Jacobian of µ at the fixed point ∂µ ∂s s 0 is negative, in the sense that DISPLAYFORM13 Proof. Let v be a vector in the state space, α > 0 a positive scalar and let s:= s 0 + αv. For α small enough, the vector s is in the region of stability of s 0. Using a first order Taylor expansion and the fixed point condition µ s 0 = 0 we get DISPLAYFORM14 as α → 0. Hence the . The following proposition shows that, unless the free fixed point s 0 θ,v is already optimal in terms of cost value, for β > 0 small enough, the nudged fixed point s β θ,v achieves lower cost value than the free fixed point. Thus, a small perturbation due to a small increment of β nudges the network towards a configuration that reduces the cost value. The inequality holds because B ADJOINT METHOD AND RELATED ALGORITHMS Earlier work have proposed various methods to compute the gradient of the objective function J (Eq. 20). One common method is the "adjoint method". In the context of fixed point recurrent neural networks studied here, the adjoint method leads to Backpropagation Through Time (BPTT) and "Recurrent Backpropagation" BID1 BID28. BPTT is the method of choice today for deep learning but its biological implausibility is obvious -it requires the network to store all its past states for the propagation of error derivatives in the second phase. Recurrent Backpropagation corresponds to a special case of BPTT where the network is initialized at the fixed point. This algorithm does not need to store the past states of the network (the state at the fixed point suffices) but it still requires neurons to send a different kind of signals through a different computational path in the second phase, which seems therefore less biologically plausible than our algorithm. Our approach is to give up on the idea of computing the gradient of the objective function. Instead we show that the STDP-compatible weight change computes a proxy to the gradient in a more biologically plausible way. For completeness, we state and prove a continuous-time version of Backpropagation Through Time and Recurrent Backpropagation. The formulas for the propagation of error derivatives (Theorem 6 and Corollary 7) will make it obvious that our algorithm is more biologically plausible than both of these algorithms. To keep notations simple, we omit to write dependences in the data sample v. We consider the dynamics ds dt = µ(θ, s) and denote by s t the state of the system at time t ≥ 0 when it starts from an initial state s 0 at time t = 0. Note that s t converges to the fixed point s 0 θ as t → ∞. We then define a family of objective functions L(θ, s 0, T):= C (θ, s T) for every couple (s 0, T) of initial state s 0 and duration T ≥ 0. This is the cost of the state at time t = T when the network starts from the state s 0 at time t = 0. Note that L(θ, s 0, T) tends to J(θ) as T → ∞ (Eq. 20).We want to compute the gradient ∂L ∂θ (θ, s 0, T) as T → ∞. For that purpose, we fix T to a large value and we consider the following quantity ∂L ∂s T −t:= ∂L ∂s (θ, s T −t, t),which represents the "partial derivative of the cost with respect to the state at time T − t". In other words this is the "partial derivative of the cost with respect to the (T − t)-th hidden layer" if one regards the network as unfolded in time (though time is continuous here). The formulas in Theorem 6 below correspond to a continuous-time version of BPTT for the propagation of the partial derivatives Computing ∂L ∂s T −t h i cannot reach values above 1. As a consequence h i must remain in the domain 0 ≤ h i ≤ 1. Therefore, rather than the standard gradient descent (Eq. 58), we will use a slightly different update rule for the state variable h: DISPLAYFORM0 This little implementation detail turns out to be very important: if the i-th hidden unit was in some state h i < 0, then Eq. 58 would give the update rule h i ← (1 −)h i, which would imply again h i < 0 at the next time step (assuming < 1). As a consequence h i would remain in the negative range forever. We use different learning rates for the different layers in our experiments. We do not have a clear explanation for why this improves performance, but we believe that this is due to the finite precision with which we approach the fixed points. The hyperparameters chosen for each model are shown in Table 1 and the are shown in Figure 3. We initialize the weights according to the Glorot-Bengio initialization BID12. Table 1: Hyperparameters. for both the 2 and 3 layered MNIST. Example system trained on the MNIST dataset, as described in Appendix C. The objective function is optimized: the training error decreases to 0.00%. The generalization error lies between 2% and 3% depending on the architecture. The learning rate is used for iterative inference (Eq. 59). β is the value of the clamping factor in the second phase. α k is the learning rate for updating the parameters in layer k. We were also able to train on MNIST using a Convolutional Neural Network (CNN). We got around 2% generalization error. The hyperparameters chosen to train this Convolutional Neural Network are shown in TAB0. == We describe a biologically plausible learning algorithm for fixed point recurrent networks without tied weights != A theoretical analysis of a new class of RNNs, trained on neuroscience tasks, allows us to identify the role of dynamical dimensionality and cell classes in neural computations.

Question: Generative Adversarial Networks (GANs) have shown impressive in modeling distributions over complicated manifolds such as those of natural images. However, GANs often suffer from mode collapse, which means they are prone to characterize only a single or a few modes of the data distribution. In order to address this problem, we propose a novel framework called LDMGAN. We ﬁrst introduce Latent Distribution Matching (LDM) constraint which regularizes the generator by aligning distribution of generated samples with that of real samples in latent space. To make use of such latent space, we propose a regularized AutoEncoder (AE) that maps the data distribution to prior distribution in encoded space. Extensive experiments on synthetic data and real world datasets show that our proposed framework signiﬁcantly improves GAN’s stability and diversity. Generative models (; ; ; ; ; ;) provide powerful tools for unsupervised learning of probability distributions over difficult manifolds such as those of natural images. Among these models, instead of requiring explicit parametric specification of the model distribution and a likelihood function, Generative Adversarial Networks (GAN) only have a generating procedure. They generate samples that are sharp and compelling, which have gained great successes on image generation tasks (; ; ;) recently. GANs are composed of two types of deep neural networks that compete with each other: a generator and a discriminator. The generator tries to map noise sampled from simple prior distribution which is usually a multivariate gaussian to data space with the aim of fooling the discriminator, while the discriminator learns to determine whether a sample comes from the real dataset or generated samples. In practice, however, GANs are fragile and in general notoriously hard to train. On the one hand, they are sensitive to architectures and hyper-parameters . For example, the imbalance between discriminator and generator capacities often leads to convergence issues. On the other hand, there is a common failure issue in GANs called mode collapse. The generator tends to produce only a single sample or a few very similar samples in that case, which means GANs put large volumes of probability mass onto a few modes. We conjecture the mode missing issue in GANs is probably because GANs lack a regularization term that can lead the generator to produce diverse samples. To remedy this problem, in this work, we first propose a regularization constraint called Latent Distribution Matching. It suppresses the mode collapse issue in GANs by aligning the distributions between true data and generated data in encoded space. To obtain such encoded space, we introduce a regularized autoencoder which maps data distribution to a simple prior distribution, eg., a gaussian. As shown in Figure 1, we collapse the decoder of the regularized AE and generator of GAN into one and propose LDMGAN. Our framework can stabilize GAN's training and reduce mode collapse issue in GANs. Compared to other AE-based methods on 2D synthetic, MNIST, Stacked-MNIST, CIFAR-10 and CelebA datasets, our method obtains better stability, diversity and competitive standard scores. GANs were initially proposed by. They contain two neural networks: a generator and a discriminator. Let {x i} N i=1 denote the training data, where each x i ∈ R D is drawn from unknown data distribution p d (x). Generator is a neural network G γ that maps the noise vector z ∈ R K, typically drawn from a multi-variate gaussian, to data spacex ∈ R D. The discriminator D ω is a classification network that distinguishes real samples from generated samples. The parameters of these networks are optimized by solving the following minmax game: where E x indicates an expectation over the data distribution p d (x) and E z indicates an expectation over the prior noise distribution p(z). Given generator G, the optimal discriminator D is is the model distribution. With the help of the optimal discriminator, the GAN objective is actually minimizing the Jensen Shannon divergence between the model distribution and the data distribution. And the global minimum is achieved if and only if There are many works that try to stabilize GAN's training and alleviate mode collapse in GANs. DCGAN designed a class of deep convolutional architectures which has become standard architecture for training GANs. Improved GANs proposed several techniques like feature matching, mini-batch discrimination and historical averaging which stabilized GAN's training and reduced mode collapse. Unrolled GAN proposed unrolling optimization of the discriminator objective to train generator. TTUR introduced two time-scale update rule and proved its convergence to a local Nash equilibrium. analyzed the convergence properties of GANs and proposed WGAN which leveraged the Wasserstein distance and demonstrated its better convergence than Jensen Shannon divergence. However, WGAN required that the discriminator must lie on the space of 1-Lipschitz functions. It used a weight clipping trick to enforce that constraint. WGAN-GP stabilized WGAN by alternating the weight clipping by penalizing the gradient norm of the interpolated samples. SN-GAN proposed spectral normalization which controls the Lipschitz constraint of netowork layers. Some works make efforts towards integrating AEs into GANs. AEs can be used in discriminators. EBGAN employed an autoencoder structure in its discriminator and introduced repelling regularization to prevent mode collapse. BEGAN extended EBGAN by optimizing Wasserstein distance between AE loss distributions of real and generated samples. MDGAN contained a manifold step and a diffusion step that combined a plain autoencoder with GANs to suppress mode missing problem. However, MDGAN required two discriminators and it did not impose regularization on its AE. VAEGAN unified VAE and GAN into one model, and utilized feature-wise distance in intermediate features learned by discriminator to replace similarity metric in data space in VAE to avoid blur. AAE used adversarial learning in its encoded space to perform variational inference. BiGAN and Adversarial Learned Inference (ALI) jointly trained an inference model and a generative model through adversarial game. VEEGAN employed a reconstructor network to both map the true data distribution p d (x) to a gaussian and to approximately invert the generator network. AGE performed adversarial game between a generator and an encoder, and proposed to align the model distribution with data distribution in encoded space. They eliminated discriminator in their framework and used an encoder to extract statistics to align model distribution with data distribution. The most related works to ours are VAEGAN and VEEGAN. VAEGAN's motivation was to combine VAE with GAN, and they relied on an ELBO, namely While we perform marginal distribution matching, namely E q(z|x) [log p(x|z)] − D KL (q(z) p(z)), and view the second term as a regularization term. Therefore there is no need of re-parameterization trick in our framework. Also we compare the distributions of the real samples and generated samples in the encoded space to suppress mode collapse in GANs which is different from VAEGAN. VEEGAN had a reconstructor network which mapped the data distribution to a gaussian as well. However, VEEGAN autoencoded noise vectors rather than data items, and its training objective was formed as an adversarial game in joint space (x, z) like BiGAN and ALI. Our regularized autoencoder is similar to AAE. But instead of using adversarial learning in encoded space which also has possibility of falling into mode collapse as in data space, we use explicit divergence to align aggregated posterior with the prior distribution. GAN's training can be viewed as an adversarial game between two players, in which the discriminator tries to distinguish between real and generated samples, while the generator tries to fool the discriminator by pushing the generated samples towards the direction of higher discrimination values. However, there are many points in data manifold that have high discrimination values. When collapse to a single mode is imminent, the gradient of the discriminator may point in similar directions for many similar points . Since the discriminator iteself processes each sample independently, there is no mechanism to tell the outputs of the generator to become more dissimilar to each other. In this paper, we propose a framework called LDMGAN to address this problem. See Figure 1 for a glance of our model's architecture. Instead of exposing the dissimilarity of generated samples to the discriminator , we introduce a regularized autoencoder in our framework. We perform data dimensionality reduction on the training data to obtain a well-shaped manifold in encoded space, e.g., a factorized gaussian, by learning a regularized autoencoder. Then when sampling noise from a factorized gaussian to feed into the generator to produce generated samples, we encode the generated samples to latent space to enforce them to be coincident with the factorized gaussian. In this way, the generator receives a signal from that constraint to tell it to produce diverse samples. We call this constraint Latent Distribution Matching (LDM). In the following, we describe LDM and the regularized autoencoder in detail. Figure 1: Our LDMGAN's architecture. The thick lines denote added losses to GANs. The main idea of LDM is to introduce an encoder function F θ that can map the true data distribution p(x) to a prior Gaussian p(z). To understand why such an encoder helps alleviate address mode collapse, consider the example in Figure 2. The middle vertical panel represents the data space, where in this example the true data distribution p d (x) is a mixture of two Gaussians. The bottom panel depicts the input to the generator (latent code space), which is drawn from p(z), and the top panel depicts the of applying the encoder network to the generated data and true data. Assume that the encoder approximately maps the data distribution to the prior gaussian (see q(z) above the top panel in Figure 2 ). The purple arrows above the middle panel show the action of the encoder on the true data, whereas the green arrows show the same action on the generated data. If mode collapse occurs as it does in this example that the generator G γ captures only one of the two modes of p d (x), it would cause distribution mismatch between the two marginal distributions q(z) and q (z). The definitions of q(z) and q (z) go as follow: where. This mismatch is evaluated by a divergence measure ∆(q (z) q(z)) in the latent space. We only require this divergence to be nonnegative and zero if and only if the distributions are identical (∆(q (z) q(z)) = 0 ⇐⇒ q (z) = q(z)). Therefore, this mismatch value function is defined as: However, comparing two empirical (hence non-parametric) distributions q (z) and q(z) is difficult. We avoid this issue by introducing an intermediate reference distribution which is also the prior distribution p(z), ing in: As we have assumed that the encoder network can approximately map the data distribution to a gaussian, we can get rid of ∆(q(z) p(z)) with minor error. Finally, this mismatch term turns out to be It can be used to detect mode collapse in generator. We call this mismatch term as Latent Distribution Matching (LDM), and use this term to regularize the generator in GANs to combat mode collapse. AutoEncoders have long been used for data dimensionality reduction and feature learning. We impose the regularization that the distribution in the encoded space of real data is coincident with the prior distribution p(z) on the autoencoder. We call this autoencoder the regularized autoencoer in this paper. There are two ways of utilizing the regularized autoencoder. One is to pretrain a regularized autoencoder, then use the encoder of the regularized autoencoder only. The other is to combine the autoencoder training with GAN objective which has already been proposed in many GAN variants (; ;). We combine autoencoder with GANs in our model for three reasons: to guarantee the learning is grounded on all training samples; to regularize the generator in producing samples to resemble real ones; last but not least, to obtain an encoder that can approximately map the data distribution to a gaussian. In the following, we will explain the reasons for combining autoencoders in our model and describe our framework formally. One of the reasons that mode collapse occurs is probably that the areas near the missing modes are rarely visited by the generator, therefore providing very few samples to improve the generator around those areas. And only when G(z) is very close to the missing modes can the generator get gradients to push itself towards the missing modes. Consider a model combining GANs with reconstruction loss in data space d(x, G(E(x))). Given enough data examples x available in training data for minor mode M 0, we know G • E is a good autoencoder, then G • E(x) will be located very close to the minor mode M 0. Therefore the gradient from the discriminator ∇ log D(G • E(x)) might be able to push these points towards the minor mode M 0 and the learning is grounded on all training samples. For that purpose, the following loss is added to the generator: As the regularized autoencoder combined with GAN has imposed gaussian on the encoded space of real data samples, this guarantees such E(x) for minor modes can be sampled with fair possibility when sampling z from gaussian that are feeded to the generator. In addition, as lots of image synthesis works ) have used L 1 or L 2 loss in data space and achieved successes in producing realistic images. We use the autoencoder loss d(x, G • E(x)) in data space as well for the purpose of regularizing the generator in producing samples to resemble real ones. In this section, we formally demonstrate the regularized autoencoder we combine with GANs. Let x be the data sample vector,x be the reconstructed sample vector of x by the autoencoder and z be the latent code vector of the autoencoder. Let p(z) be the prior distribution we want to impose on the codes, q(z|x) be an encoding distribution and p(x|z) be the decoding distribution. Also let p d (x) be the data distribution, and p g (x) be the model distribution. We desire an autoencoder whose aggregated posterior q(z) is coincident with a prior distribution p(z). In this paper, we use a factorized gaussian as the prior distribution. The encoding function of the autoencoder q(z|x) defines the aggregated posterior in Equation 2. We choose a deterministic function form for q(z|x) which means the stochasticity of q(z) comes only from the the data distribution p d (x). We impose a regularization term on the encoded space of the autoencoder to minimize: ∆(q(z) p(z)). The autoencoder, meanwhile, attempts to minimize the reconstruction error d(x,x) (here d is some similarity metric, besides L 1 or L 2 distance in data space, it can also be distance metric in the feature-wise space like the intermediate layers of well-known VGG network or the discriminator of GANs). Therefore,the regularized autoencoder is trained in two phases: the reconstruction phase and the regularization phase. In the reconstruction phase, the regularized autoencoder uses SGD to update the encoder and decoder's parameters to reconstruct the input data vector. In the regularization phase, the autoencoder uses SGD to update the encoder's parameters to match the aggregated posterior with the prior distribution. Implemented by deep neural networks, with arbitrary capacity, the universal approximation theorem guarantees the regularized autoencoder is capable of approximately matching the aggregated posterior q(z) with the prior distribution p(z). Given the encoder E θ which is capable of approximately mapping the data distribution to a gaussian, consider a standard GAN training loss plus the LDM (Latent Distribution Matching) constraint on the generator: This training objective Equation 8 is what we apply in GAN framework when not concerning combining the regularized autoencoder into GAN. And after unifying the regularized autoencoder and GAN described above, the objective becomes: where λ 1 and λ 2 are hyper-parameters that control the loss weights of reconstruction error and LDM constraint respectively. Note that the LDM term ∆(q (z) p(z)) is minimized only with respect to G γ, which means the parameters of the encoder E are fixed during backpropagating the LDM term. The divergence measure between the empirical distribution and the prior distribution in the latent M -dimensional space that we use in this paper goes as follow: where µ i and σ i are the means and the standard deviations of the fitted empirical distribution Q along various dimensions. The training algorithm and architecture of our proposed model is depicted in Algorithm 1 and Figure 1 respectively. Algorithm 1 Our LDM-GAN's training algorithm 1: Dω, E θ, Gγ ← initialize network parameters respectively 2: repeat 3: X ← Random mini-batch of m data points from dataset 4: Xrec ← Gγ(Z) 8: Zrec ← E θ (X) 9: // Training discriminator D 10: // Training encoder E 12: // Training generator G 14: gγ ← ∇γ log(1 − Dω(X)) + log(1 − Dω(Xrec)) + λ1d(X, Xrec) + λ2DKL(q (Zrec) p(z)) 15: // Using SGD to update model's parameters 16: ω ← ω − ηgω; θ ← θ − ηg θ; γ ← γ − ηgγ 17: until convergence 18: return D, E, G In this section, we compare our model with four different models on 2D Synthetic, MNIST, Stacked-MNSIT, CIFAR-10 and CelebA datasets. We do not finetune the hyper-parameters λ 1 and λ 2 in our model, and by default set λ 1 = 1 and λ 2 = 1. Evaluation of mode collapse in GANs is very difficult especially when trained on natural images. However, the missing modes can be calculated precisely when using synthetic data for training. In this section we compare five different models on two synthetic datasets: a mixture of eight 2D Gaussian distributions arranged in a ring, and a mixture of twenty-five 2D Gaussian distributions arranged in a grid. The covariance matrices and centroids have been chosen such that the distributions exhibit lots of modes separated by large low-probability regions, which make them decently hard tasks despite the 2D nature of the dataset. All models share the same network architectures for fair comparison. The encoder and the generator network are consisted of two hidden layers, while the discriminator network only has one hidden layer. All networks are implemented by fully-connected layers and each layer has 64 neurons. The dimension of the prior input is 2. For VEEGAN and BiGAN, the discriminator's input is augumented by the size of the latent code. Note that the VAEGAN we used here is slightly different from the original paper, since the network architectures are shallow and the data is low-dimensional: we used L 2 loss on the data space for simplicity instead of on features learned by the discriminator. To quantify the mode collapse behavior we report two metrics: the number of modes that each model captures and the percentage of high quality samples. We draw 2048 samples from each model for evaluation. The high quality samples are defined as the points which are within three standard deviations of the nearest mode, and if there are more than twenty such samples registered to a mode, the mode is considered as captured. The are depicted in Table 1 and the numbers are averaged over eight runs. We also show the generated distribution of each model in Figure 3 As we can see from Table 1, our model captures the greatest number of modes on both synthetic datasets. The generator learned by our model is sharper and closer to the true distribution as shown in Figure 3. Though other models may obtain greater percentage of high quality samples, for example, vanilla GAN generates 99.96% high quality samples on 2D Ring dataset. However, it runs into severe mode missing problem. Vanilla GAN only captures one mode on 2D Ring dataset. 1. VEEGAN is able to capture most of the modes in some settings (LDMGAN also performs as well as VEEGAN under those settings), but it fails to generalize well under our settings. In particular, while our model resembles VAEGAN in some ways, VAEGAN suffers from some degree of mode missing problem as well and it is far from competitive compared to our proposed model. In order to systemically explore the effect of our proposed LDMGAN on alleviating mode collapse in GANs, we use a large scale grid search of different GAN hyper-parameters on the MNIST dataset. We use the same hyper-parameters settings for both vanilla GAN and our LDMGAN, and the encoder in our model is the "inverse" of the generator. The search range is listed in Table 2. Our grid search is similar to those proposed in;. Please refer to it for detailed explanations regarding these hyper-parameters. In order to estimate both the missing modes and the sample qualities, we first train a regular CNN classifier on the MNIST digits, and then apply it to compute the MODE Score which is proposed in. MODE Score is a variant of Inception Score (IS) and its definition goes as follow: where p(y|x) is the softmax output of a trained classifier of the labels, and p * (y) and p(y) are the marginal distributions of the label of the generated samples and training samples respectively. We train each architecture for 50 epochs and after that draw 10K samples for evaluation. The ing distribution of MODE Score is shown in Figure 4. It can be seen that our model clearly improves the sample quality and diversity compared to GANs. , we evaluate our method on the stacked MNIST dataset, a variant of the MNIST data specifically designed to increase the number of discrete modes. The data is synthesized by stacking three randomly sampled MNIST digits along the color channel ing in a 28×28×3 image. We now expect 1000 modes on this dataset, corresponding to the number of possible triples of digits. We generate a dataset of 50000 images for training. And for fair comparation we use the same architecture for all models. The generator and discriminator networks are implemented following DCGAN. For encoder network, we use a simple two layer MLP network without any regularization layers. For BiGAN and VEEGAN, we concatenate the last convolutional layer of the discriminator with the latent code which follows by two fully-connected layers in order to perform joint space discrimination. The dimension of the prior input is 64. Each model is trained for 50 epochs. As the true locations of the modes in this data are unknown, the number of modes are estimated using a trained classifier as described in subsection 5.2. We draw 26000 samples from each model for evaluation. As a measure of quality, following , we also report the KL divergence between the model distribution and the data distribution. The are averaged over 5 runs. As reported in Table 3, our model recovers greatest number of modes and better matches the data distribution than any other methods. Table 3: Degree of mode collapse, measured by modes captured and sample quality (as measured by KL) on Stacked-MNIST. Our model captures the most modes and also achieves the highest quality. For CIFAR-10 and CelebA datasets, we measure the performance with FID scores . FID can detect intra-class mode dropping, and measure the diversity as well as the quality of generated samples. FID is computed from 10K generated sampels and the pre-calculated statistics on all training data provided in the code repository 2. Our default parameters are used for all experiments. And the model architecture is the same as the standard DCGAN. We train each model for 50 epochs on CIFAR-10 and 30 epochs on CelebA. The dimension of the prior input is 100. All of our experiments are conducted using the unsupervised setting. We evaluate the FID score for each model per epoch during training, and leave the best one per run. As we can not reproduce VEEGAN and BiGAN on these datasets using the same architecture, we do not report FID scores on these two models. The reported are depicted in Table 4 and are averaged over 5 runs. It is important to mention that we do not pursue the state-of-the-art FID scores on both datasets as we only want to prove that our model improves GANs. It can be seen that our model has both the best FID scores and averaged FID scores on CIFAR-10 and CelebA. The improvement on CelebA is smaller compared to that on CIFAR-10. We suspect that this is due to the less complicated structure of face images, but anyway our model still consistently performs better than DCGAN and VAEGAN. The random samples generated from different models on both datasets are shown in Appendix A. Finally, to show that our proposed model are not just simply memorizing the training data, we interpolate between random vectors on generator of our model trained on CelebA dataset and the are depicted in Appendix A. CelebA Table 4: FID scores for different models, lower is better. Best FID is the best run in 5 trial. We propose a robust AE-based GAN model with novel Latent Distribution Matching constraint, called LDM-GAN, that can address the mode collapse effectively. Our model is different from previous works: We propose a novel regularization constraint called Latent Distribution Matching to align the distributions of real data and generated data in encoded space. We propose a new regularized autoencoder for approximately mapping data distribution to a gaussian. We designed a novel AE-based GAN that drastically alleviate mode collapse in GANs. Extensive experiments demonstrate that our method is stable and does not suffer from mode collapse problem in Synthetic 2D, MNIST, Stacked-MNIST datasets. Furthermore, we achieve better FID scores on CIFAR-10 and CelebA datasets compared to baseline models without any other tricks and techniques. These demonstrate that our proposed LDMGAN indeed improves GAN's stability and diversity. A APPENDIX

We propose an AE-based GAN that alleviates mode collapse in GANs.

Represent the TLDR for finding the corresponding scientific article

We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.


Question: Represent the article for finding a single-sentence TLDR

Question: Achieving faster execution with shorter compilation time can foster further diversity and innovation in neural networks. However, the current paradigm of executing neural networks either relies on hand-optimized libraries, traditional compilation heuristics, or very recently genetic algorithms and other stochastic methods. These methods suffer from frequent costly hardware measurements rendering them not only too time consuming but also suboptimal. As such, we devise a solution that can learn to quickly adapt to a previously unseen design space for code optimization, both accelerating the search and improving the output performance. This solution dubbed CHAMELEON leverages reinforcement learning whose solution takes fewer steps to converge, and develops an adaptive sampling algorithm that not only focuses on the costly samples (real hardware measurements) on representative points but also uses a domain knowledge inspired logic to improve the samples itself. Experimentation with real hardware shows that CHAMELEON provides 4.45×speed up in optimization time over AutoTVM, while also improving inference time of the modern deep networks by 5.6%. The enormous computational intensity of DNNs have ed in developing either hand-optimized kernels, such as NVIDIA cuDNN or Intel MKL that serve as backend for a variety of programming environment such as TensorFlow and PyTorch . However, the complexity of the tensor operations in DNNs and the volatility of algorithms, which has led to unprecedented rate of innovation , calls for developing automated compilation frameworks. To imitate or even surpass the success of hand-optimized libraries, recent research has developed stochastic optimization passes for general code: STOKE , and neural network code, TVM (a) and TensorComprehensions . TVM and TensorComprehensions are based on random or genetic algorithms to search the space of optimized code for neural networks. AutoTVM (b) builds on top of TVM and leverage boosted trees as part of the search cost model to avoid measuring the fitness of each solution (optimized candidate neural network code), and instead predict its fitness. However, even with these innovations the optimizing compilation time can be around 10 hours for ResNet-18 , and even more for deeper or wider networks. Since the general objective is to unleash new possibilities by developing automatic optimization passes, long compilation time hinders innovation and could put the current solutions in a position of questionable utility. To solve this problem, we first question the very statistical guarantees which the aforementioned optimization passes rely on. The current approaches are oblivious to the patterns in the design space of schedules that are available for exploitation, and causes inefficient search or even converges to solutions that may even be suboptimal. Also, we notice that current approaches rely on greedy sampling that neglects the distribution of the candidate solutions (configurations). While greedy sampling that passively filter samples based on the fitness estimations from the cost models work, many of their hardware measurements (required for optimization) tend to be redundant and wasteful. Moreover, we found that current solutions that rely on greedy sampling lead to significant fractions of the candidate configurations being redundant over iterations, and that any optimizing compiler are prone to invalid configurations which significantly prolongs the optimization time. As such, this work sets out to present an Adaptive approach to significantly reduce the compilation time and offer automation while avoiding dependence to hand-optimization, enabling far more diverse tensor operations in the next generation DNNs. We tackle this challenge from two fronts with the following contributions: Devising an Adaptive Exploration module that utilizes reinforcement learning to adapt to unseen design space of new networks to reduce search time yet achieve better performance. Proposing an Adaptive Sampling algorithm that utilizes clustering to adaptively reduce the number of costly hardware measurements, and devising a domain-knowledge inspired Sample Synthesis to find configurations that would potentially yield better performance. Real hardware experimentation with modern DNNs (AlexNet, VGG-16, and ResNet-18) on a highend GPU (Titan Xp), shows that the combination of these two innovations, dubbed CHAMELEON, yields 4.45×speedup over the leading framework, AutoTVM. CHAMELEON is anonymously available in https://github.com/anony-sub/chameleon, which will be made public. The general life-cycle of deep learning models from its birth to deployment comprises of two major steps. First stage is the designing and the training of a deep learning model by a research scientist, with the primary goal of achieving the highest feasible accuracy. Then, with a general demand to enable the intelligence on a wide range of devices (from mobile CPUs in the edge to cloud-scale GPUs), the second stage has emerged for the deployment of the pre-trained deep learning model to a target hardware by a deployment engineer. These stages are each iterative processes: research scientists iterate until it reaches the target performance in terms of accuracy whereas the deployment engineers iterate until the performance in terms of inference speed with a given hardware satisfies the given constraints. Importantly, these two stages are most often separate processes, and this paper mainly focuses on the second stage (deployment) of the cycle with an overarching goal of accelerating the overall deployment cycle by reducing the optimizing compilation time without compromising the performance of the output code. and TVM (a), using which we implement CHAMELEON that is available as a separate package for adoption in even other frameworks. The first phase of the workflow is the frontend compiler which performs the translation from the compiler and applies target-independent and white-box target-dependent optimizations that do not incorporate a measure of runtime. Target-independent passes transform the input DNN model without specificity to the target hardware. Operator fusion and data layout transformation in TVM are some examples of these passes, which lie in the same category as dead-code elimination or loop-invariant code motion in GCC or LLVM . Target-dependent passes, on the other hand, the compiler takes the hardware architecture (target) into account while optimizing the program; however, this also does not actively leverage runtime measures. The last stage is a black-box optimization pass, called optimizing compiler, that given a measure of performance at runtime from the hardware can further optimize the code. CHAMELEON falls in this class by offering an optimizing compiler that adapts to different design space to be more swift in optimizing deep neural networks compared to conventional approaches. Optimizing compilers usually take a black-box approach and use hardware measurements to configure the optimization based on a measure of fitness f of each solution. In Factors for tiling and binding # of filters height, and width of feature maps. tile rc, tile ry, tile rx Factors for tiling reduction axis such as # of channels, height, and width of filters. auto unroll max step Threshold of number of steps in the loop to be automatically unrolled. Explicitly unroll loop, this may let code generator to generate pragma unroll hint. order to make the problem tractable, the optimizing compilers for deep neural networks reduce the problem down to tuning the knobs θ for the output code template τ, and can be formulated as: A combination of assignment to the knobs is said to be a configuration Θ = (θ 1, θ 2, ..., θ n) while the dimensions of the design space D Θ is defined by the knobs. As such, in Equation 1, an optimizing compiler starts from a code template τ for each layer, and makes use of a search algorithm and real hardware measurements to efficiently find the best configuration Θ * ∈ D Θ. In this context, there are three variables that determine the effectiveness of the optimizing compiler: a large and diverse enough design space (knobs) that covers a variety of transformations, an effective search algorithm to adequately navigate this space, and a mechanism to cut down the number of costly hardware measurements that check the fitness of a solution. Table 1 lists the knobs for performing convolution on a GPU, where it is crucial that the code maximizes data reuse, uses the shared memory wisely, and minimizes bank conflicts. The knobs optimize various aspects of the execution, including tiling (e.g., tile x, tile y, . . .), unrolling (e.g., auto unroll max step and unroll explicit), and these knobs define a design space with 10 10 possibilities. Given the vastness of the design space, the remaining challenges are designing an effective search algorithm and designing a mechanism that reduces the cost of each step in the search (i.e. reducing the need to measure the hardware). As shown in Figure 2, optimizing compilation for DNNs can take an eon even with the advances from prior works (a; b;) With active research (; ; ; ; ;) that has been able to cut down the training time to only few hours and even minutes on big models (e.g., ResNet-50 ) for ImageNet, it renders the optimizing compilation time of the current solutions seem even more prominent. Especially, since the above-mentioned compilers have been integrated to the deep learning pipelines of major players in the industry; ), many users of these pipelines including the deployment engineers in these companies will be going through the compilation workflow depicted in Figure 1 numerous times. Therefore, current long compilation time can be a hindrance to deploying DNN in various hardware, hence a major bottleneck in enabling intelligence on wider range of target platforms. Furthermore, as we explore various neural topologies , deeper or wider networks , and new operations to achieve higher performance , we are pushed to optimize the networks more frequently. The long optimization times are multiplied with such trend, leaving the practical utility of the current compiler solutions to question. As such, the primary goal of this work is reducing the optimizing compilation time to meet the immediate needs of the industry for expedited DNN compilation to foster further diversity and innovation in designing DNNs. Such long optimization time from the inefficiency of simulated annealing which (while it stochastically guarantees a reasonable solution after huge number of iterations) fails to capture the patterns in the design space that can be exploited during the search. On the other hand, we can see in the figure that majority of the optimization time is spent on reaching for measurements on real hardware that is used as a feedback for the aforementioned search. Also, current approach even suffers from numerous invalid configurations that not only wastes the limited hardware measurement budget that the compiler starts with, but also incurs serious overhead to reset the target hardware for subsequent hardware measurements. As such, it is important that a sampling mechanism that selects potential configurations for hardware measurements to be smarter to ensure that each measurement is maximizing the chances of achieving a good solution and that it evades the invalid configurations. However, the current approaches rely on greedy sampling that passively sample based on the estimations from the cost models. This not only has a tendency to overfit but also neglect that solutions are distributed non-uniformly and that there are numerous invalid configurations. As discussed in Section 2, current solutions fall short of providing a swift optimization framework for optimizing emergent deep neural networks, because of the futility of the search in adapting to the design space from a random walk based search algorithm and the inefficiency of the physical hardware measurements from the greedy sampling. Therefore, developing a new framework that can overcome current challenges to unfetter neural network innovation from a prolonged optimization times can be boiled down to two problems: 1 improving the the search algorithm to better adapt to the design space, and 2 improving the sampling algorithm to both better adapt to the distribution of the solutions and decrease the possibility of running into invalid configurations. As such we make two innovations in the optimizing compiler for deep neural networks to develop CHAMELEON by applying reinforcement learning to the search that can adapt to new design spaces (Adaptive Exploration) and devising an Adaptive Sampling that replaces the current greedy sampling. Figure 3 outlines the overall design of our optimizing compiler, dubbed CHAMELEON 1, and gives an overview of the optimizing compilation process. CHAMELEON takes code template τ for each layer in the network and the corresponding design space D Θ as its input, and iteratively optimizes the code for configuration Θ to finally output τ (Θ *). The proposed Adaptive Exploration maneuvers the design space while using a cost model as a proxy for hardware measurements to the output set of candidate configurations S Θ. These configurations are then sampled with Adaptive Sampling so that the sampled configurations S Θ subsume the initial candidate configurations while reducing its number significantly. The sampled configurations S Θ are then passed to the code generator which combines the input template τ and the configurations S Θ to create a set of τ (Θ) that are sent to real hardware for runtime measurements. Runtimes from the hardware are used as the measure of fitness f and update the cost model to enhance the exploration of the subsequent iterations. After multiple iterations, τ (Θ *) with the best fitness f (shortest runtime) is selected as an output for the layer. As stated in Section 2, the current state-of-the-art approach (b) that leverages simulated annealing relies on the stochastic guarantees of its random walks. Therefore, the current approach requires numerous iterations of exploration to converge to a reasonable solution causing long compilation hours, thus insufficient to enable disruptive innovations in neural networks. We take an inspiring approach that avoids naive dependence on the stochastic guarantee of simulated annealing and leverage a technique that can learn to adapt to unseen design space to not only accelerate convergence but also bring some performance gains. As such, we develop Adaptive Exploration by leveraging Reinforcement Learning (RL), which is concerned with learning to maximize reward given an environment by making good exploration and exploitation tradeoffs, in our case maximizing fitness f of the explored configurations S Θ. Reinforcement learning formulation. Our RL-based Adaptive Exploration module uses an actor-critic style RL, where Policy Network learns to emit a set of directions (vector of increment/decrement/stay) for each knob in the design space that will increase f of the next configuration and the Value Network learns the design space D Θ to estimate the value of the action. The first layer of these networks that takes the current configuration Θ as input is shared to foster information sharing among the two networks, and its output is fed into the subsequent layers the networks. These networks not only learn the dependencies among the different knobs of the design space (which are interrelated) that helps our module navigate through the design space but also lean the potential gains of the modifications to the configurations. Learning procedure. Having formulated the RL-based Adaptive Exploration Module, an iteration of our optimization begins with a set of initial configurations and takes multiple search steps (episode) for each of the configurations. As shown in Figure 4, the agent makes an action and applies it to the configuration using configuration updater to get another configuration that potentially has better f. After finishing multiple search steps in the episode, all configurations S Θ are evaluated using a cost model, which its return values are used as a surrogate reward to update our agent, to reduce the number of costly hardware measurements. By taking this approach, f of S Θ improves as our module progresses through the episodes. In other words, by repeating multiple episodes and iterations, our Adaptive Exploration Module gradually learns to locate good configurations. Reducing number of costly hardware measurements. After the exploration step (regardless of the exploration method), we observe that the candidate configurations are clustered in subregions of the design space and these clusters are non-uniformly distributed (Figure 5). We also find that, while the design space's surface is discrete and un-smooth, a large fraction of configurations within each cluster achieve similar runtime (Figure 6). Utilizing these characteristics of the design space, we devise Adaptive Sampling that can sample a new set of candidates, by adapting to the shape of the design space and the non-uniformity of the distribution while leaving the performance of optimization intact. We first leverage Clustering algorithm to find configurations that are representative of each cluster; the sampling module uses centroids as the representative configurations. Our adaptive sampling iterates over a different number of clusters for their respective centroids and the L2 loss. In the context of optimizing compiler, selecting the number of centroids for clustering entails making the important tradeoff between selecting more centroids for better performance or fewer centroids for a reduced number of hardware measurements. As such, we must devise a method that would automatically make the tradeoff in a reasonable manner. We take advantage of the decreasing trend in the aforementioned L2 loss as we increase the number of centroids, and devise a Threshold-based Meta-Search to determine the number of clusters. By setting the threshold (hyperparameter) it allows the compiler to determine the point of diminishing return (Knee of the curve), inflection point beyond which fewer centroids may lead to performance degradation and more clusters would prolong the optimization substantially. Overall, our sampling curtails the number of hardware measurements so that it is just enough to subsume the entire subspace of the candidate configurations. Improving candidate configurations using sample synthesis. While the above sampling algorithm significantly reduces the number of hardware measurements compared to the conventional greedy sampling, without impacting the performance of the output code, we are still left with a critical issue of Redundancy among the candidate configurations. We find that the exploration algorithm (regardless of the type) combined with the greedy sampling frequently leads to redundancy among the candidate configurations over different iterations of optimization due to the overfitting of the cost model from the greediness of the sampling. Even though the exploration algorithm tries to explore unvisited regions of the design space, these explored (not exploited) configurations are discarded due to the greedy sampling which entirely depends on the cost model for its selections of the configurations. Therefore, the current greedy sampling algorithm has its limitation in focusing the hardware measurements to the same region over and over. On the other hand, we find that from a code optimization point of view, we know that many of the automated approaches for black-box optimization are prone to Invalid Configurations, which from too large a tile that goes over the input feature map boundary or errors during memory accesses (cannot be solved analytically). These invalid configurations not only blow the chances for better exploration but also leads to an extra optimization time overhead to reset the physical hardware for the subsequent hardware measurement. We try to overcome both of these limitations by devising Sample Synthesis. When our compiler runs into redundant samples, the proposed synthesis method analyzes the candidate samples to determine the most probable (most frequent = mode function) non-invalid choice for each knob to come up with a new configuration. This statistical combination of the most frequent knob settings yield configurations that combine the strengths of different knobs to converge to a better overall solution. In spirit, the recombination (crossover) operator in genetic algorithms also tries to combine the best features of the solutions with high fitness values. Algorithm 1 presents the integration of our Adaptive Sampling and the Sample Synthesis. Architecture exploration for the adaptive exploration. We use Proximal Policy Optimization (PPO) , a policy gradient that has been shown to adapt to various problems and have good sample complexity, as our reinforcement learning algorithm. Since reinforcement learning could incur computational overhead that could prolong the optimization time, we optimize the actor-critic networks through architecture exploration to find good tradeoff for size of these networks (that determines the computational overhead) and the optimization performance. return new candidates Feed to Code Generator to make measurements on hardware 12: end procedure Design choices for the adaptive sampling. We use a K-means Clustering to determine centroids of the configurations, because K-means has been shown effective in practice and it only requires K, over error or radius in other algorithms which are much more challenging to tune. For example, DBSCAN (Ester et al.) or mean-shift clustering are very sensitive to the above hyperparameters. On the other hand, K can be framed as a lever for trading performance and the speed of the optimization which abstracts away the aforementioned challenges enabling the thresholding-based swift meta-search for detecting the knee (optimal number of clusters). Hyperparameter tuning. Hyperparameter tuning is a very important task in machine learningbased tools and models. As such, we present the hyperparameters we used for the evaluation in Table 7 (in appendix), which its tuning took several days. For the hyperparameters in Table 8 (in appendix), we used the same set of values that were used in the AutoTVM paper (b) in order to conduct a fair comparison or CHAMELEON. Additionally, for parameters used in the Adaptive Exploration module, which is not present in AutoTVM, we have tuned the hyperparameters using the set of layers presented in Table 5 (in appendix). We emphasize, however, that the hyperparameters have been tuned offline before the deployment of CHAMELEON, and the hyperparameters are not changed during the use of the framework or the experimentation. So the tuning overhead is not part of the compilation after the Adaptive Exploration module is tuned once before releasing the compiler to the deployment practitioners. We integrate CHAMELEON into TVM (a) to perform component evaluation and compare with AutoTVM (b). We first evaluate components of CHAMELEON in Section 4.1 and Section 4.2 on set of convolution layers sampled from AlexNet , VGG-16 , and ResNet-18 . Then we provide end-to-end evaluation of CHAMELEON on both set of layers and end-to-end deep models, in Section 4.3. Due to space limitations, we present only the representative plots in the paper, and the complete set of and the details of the parameters are provided in the appendix. In the previous approach (b), authors have built a cost model to estimate fitness instead of performing costly measurements on real hardware, then used simulated annealing to find potentially optimal configurations. Figure 7 (a) compares the number of search steps taken per iteration to reach or converge to the solution in simulated annealing and reinforcement learning, respectively. Overall, observation is that CHAMELEON's reinforcement learning agent requires 2.88×less search steps compared to simulated annealing to find good solution. This comes from reinforcement learning agent's ability to quickly learn about the correlation between different dimensions, and start search on top of previous iterations, to reuse the information, over starting from scratch, relying on stochastic guarantees of the simulated annealing process. Figure 7(b) summarizes the effect of applying CHAMELEON's adaptive sampling module on simulated annealing and reinforcement learning based search. First, the show that using adaptive sampling helps the framework to make less hardware measurements regardless of the search algorithm used. The adaptive sampling algorithm reduces the number of measurements by 1.98×when used with simulated annealing and 2.33×with reinforcement learning One observation is that the adaptive sampling is more effective with reinforcement learning search. This comes from the reinforcement learning agent's capacity to better localize the search to meaningful samples (exploitation) while still aiming to find good solution by making diverse search (exploration). Diversity exploration of AutoTVM aims to spread out the candidate configurations with a regularizing effect that fosters uniform sampling. In contrast, our adaptive sampling uses a clustering algorithm to perform more measurements on the regions with higher likelihood of achieving better output performance, leading to a nonuniform sampling. While AutoTVM states that diversity-aware selection had no meaningful impact on most of the evaluated workloads, our adaptive sampling brings significant improvement as depicted in Figure 8. As shown, adaptive sampling brings an average of 13.5%and 19.0%improvement on simulated annealing and reinforcement learning, respectively. CHAMELEON integrates two components into the workflow: RL-based Adaptive Exploration and Adaptive Sampling. This section compares the performance of CHAMELEON with AutoTVM (b) on both set of layers and end-to-end deep neural networks. Layer evaluation. Figure 9 shows the trend of output code performance of ResNet-18's 11th layer over number of hardware measurements during optimization. The figure illustrates that our Adaptive Exploration finds better configurations than simulated annealing which in better output code performance, and the Adaptive Sampling reduces number of hardware measurements significantly during optimization. Also, CHAMELEON's Adaptive Exploration and Adaptive Sampling working in tandem emits better code with shorter optimization time than others. As such, Figure 10 (a) compares optimization time and the performance of the output code in CHAMELEON and AutoTVM to confirm the observation. CHAMELEON achieved 1.17×better performance with 4.82×shorter optimization time compared to AutoTVM. Overall, the suggest that our Adaptive Exploration effectively maneuvers the design space, and Adaptive Sampling reduces hardware measurements and the overall optimization time while even improving output performance. Table 3: End-to-end evaluation of the output performance for deep networks. End-to-end evaluation. Up until now, we have focused on evaluation with subset of layers. Now we continue our discussion to the applicability of CHAMELEON to optimization of end-to-end deep neural networks. Figure 10 (b) shows that CHAMELEON spends 3.59×, 5.73×, and 4.28×less time than AutoTVM to optimize AlexNet, VGG-16, and ResNet-18, respectively. On average, our work shows 4.45×optimization time speedup while achieving up to 6.4%improvement in terms of performance of output code. Inference time in Figure 10 (b) illustrates the speedup for optimized code. Raw numbers are available in Table 2 and Table 3. All in all, such improvements from efficient Adaptive Exploration and the reduced number of hardware measurements from Adaptive Sampling. CHAMELEON uniquely offers a solution that exclusively enables (i) Reinforcement Learning and (ii) Sampling in the context of (iii) Optimizing Compilers for neural networks. As such, we discuss the related work from each of the three independent research directions. Optimizing compilers. TensorComprehensions and TVM (a) use genetic algorithm and simulated annealing to choose parameters of polyhedral optimization for neural networks. In a more general context, some computing libraries make use of black box optimization and also profiling-based compilation passes utilize runtime information to generate optimized code. Later, AutoTVM (b) incorporates learning with boosted trees within the cost model for TVM to reduce the number of real hardware measurements. While CHAMELEON is inspired and builds on these prior works, unlike them, it is based on reinforcement learning for Adaptive Exploration and further reduces the number of measurements by focusing them through Adaptive Sampling that leverages clustering. Reinforcement learning for hyper-parameter optimization. There are a growing body of studies on using reinforcement learning to perform various optimizations (; ; ; ; ;) for a variety of objectives including hyper-parameter optimization for neural networks. For instance, DeepArchitect and NAS use reinforcement learning to automate the process of designing deep neural network models and their associated parameters. HAQ and ReLeQ use reinforcement learning to chose levels of quantization for the layers of a given deep neural network. AMC formulates neural network compression as a RL problem. Our work exclusively explores a different problem, that is optimizing compilers, using reinforcement learning. Sampling algorithms for learning. Active learning is a broad field (; ; ; ; ;) that uses a measure of the change in the model to decide which training data elements should be used to update the model. Passive learning is an alternative view that independent of the model, analyze the distribution of the training data set and selects a subset. The Adaptive Sampling algorithm for CHAMELEON shares similarities with Passive learning but it differs in its context. The sampling is designed to reduce the number of samples (configuration) for hardware measurement from the exploration of the design space whilst performing an optimization to accelerate the process. We present CHAMELEON to allow optimizing compilers to adapt to unseen design spaces of code schedules to reduce the optimization time. This paper is also an initial effort to bring Reinforcement Learning to the realm of optimizing compilers for neural networks, and we also develop an Adaptive Sampling with domain-knowledge inspired Sample Synthesis to not only reduce the number of samples required to navigate the design space but also augment its quality in terms of fitness. Experimentation with real-world deep models shows that CHAMELEON not only reduces the time for compilation significantly, but also improves the quality of the code. This encouraging suggests a significant potential for various learning techniques to optimizing deep learning models. A.1 DNN MODELS AND LAYERS A.3 HYPER-PARAMETERS . .

Represent the short summary for finding the corresponding scientific article (long)

Reinforcement learning and Adaptive Sampling for Optimized Compilation of Deep Neural Networks.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.