Question: Represent the article for finding a single-sentence TLDR

Question: Reservoir computing is a powerful tool to explain how the brain learns temporal sequences, such as movements, but existing learning schemes are either biologically implausible or too inefficient to explain animal performance. We show that a network can learn complicated sequences with a reward-modulated Hebbian learning rule if the network of reservoir neurons is combined with a second network that serves as a dynamic working memory and provides a spatio-temporal backbone signal to the reservoir. In combination with the working memory, reward-modulated Hebbian learning of the readout neurons performs as well as FORCE learning, but with the advantage of a biologically plausible interpretation of both the learning rule and the learning paradigm. Learning complex temporal sequences that extend over a few seconds -such as a movement to grab a bottle or to write a number on the blackboard -looks easy to us but is challenging for computational brain models. A common framework for learning temporal sequences is reservoir computing (alternatively called liquid computing or echo-state networks). It combines a reservoir, a recurrent network of rate units with strong, but random connections, with a linear readout that feeds back to the reservoir. Training of the readout weights with FORCE, a recursive least-squares estimator, leads to excellent performance on many tasks such as motor movements. The FORCE rule is, however, biologically implausible: update steps of synapses are rapid and large, and require an immediate and precisely timed feedback signal. A more realistic alternative to FORCE is the family of reward-modulated Hebbian learning rules, but plausibility comes at a price: when the feedback (reward minus expected reward) is given only after a long delay, reward-modulated Hebbian plasticity is not powerful enough to learn complex tasks. Here we combine the reservoir network with a second, more structured network that stores and updates a two-dimension continuous variable as a "bump" in an attractor. The activity of the attractor network acts as a dynamic working memory and serves as input to the reservoir network (fig. 1). Our approach is related to that of feeding an abstract oscillatory input or a "temporal backbone signal" into the reservoir in order to overcome structural weaknesses of reservoir computing that arise if large time spans need to be covered. In computational experiments, we show that a dynamic working memory that serves as an input to a reservoir network facilitates reward-modulated Hebbian learning in multiple ways: it makes a biologically plausible three-factor rule as efficient as FORCE; it admits a delay in the feedback signal; and it allows a single reservoir network to learn and perform multiple tasks. Our architecture is simple: the attractor network (the "memory") receives some task-specific input and produces a robust two-dimensional neural trajectory; the reservoir network (the "motor cortex") shapes its dynamics with this trajectory, and produces a potentially high-dimensional output (fig. 1). Figure 1: Model architecture: a moving 2D bump (left: activity bump (red) surrounded by inactive neurons (blue)) in an attractor network (left circle with two bump trajectories) projects to a reservoir (right circle); the output z(t) is read out from the reservoir and approximates the target function. Attractor network. Following, the bump attractor consists of 2500 neurons evolving as where x is the vector of firing rates evolving with time constant τ m, e is the task-specific external input, h is an adaptation variable with time constant τ a and s is the strength of adaptation. The weight matrix J = J s + J h has two parts. The symmetric part J s creates a two-dimensional translationinvariant structure ing in bump-like stable activity patterns, whereas J h represents structural noise. Due to the adaptation h, the bump moves across a path defined by the initial conditions and structural noise, creating long-lasting reliable activity patterns which also depend on the input e. Reservoir network. The reservoir learns to approximate a target function f (t) with the output z(t) by linearly combining the firing rate r with readout weights W ro: z = W ro r + η η η ≡ẑ + η η η with readout noise η η η. We use the same number of neurons and parameters as, where u is the membrane potential, ξ ξ ξ is the firing rate noise, W attr scales attractor input with coupling c, W rec and λ regulate chaotic activity, and W fb implements the feedback loop. Learning rule. We use the reward-modulated Hebbian rule of for the readout weights W ro, wherex denotes low-pass filtering of x, such that z(t) −z(t) ≈ η η η(t). The reward modulation M (t) tracks performance P (t) as The update rule is an example of a NeoHebbian three-factor learning rule and mimics gradient descent if we ignore the feedback loop. For model details, see appendix A. In fig. 2, the learning rules are compared on 50 target functions sampled from a Gaussian Process (GP) with exponential squared kernel (σ 2 = 10 4 to match the complexity of hand-picked functions from ). After each training period, we measure performance with normalized cross-correlation between the output and the target (ranging from -1 to 1, where 1 is a perfect match) on a single trial with frozen weights. Details are provided in appendix A; code: https://github.com/neuroai-workshopanon-1224113/working-memory-facilitating-reservoir-learning. When tested on one-second signals similar to those of (two insets in fig. 2), the full network with attractor input and reward-modulated Hebbian learning learns faster and more reliably than reward-modulated Hebbian learning without the input from the attractor network. After about 90 training trials, the full network achieves the performance of the FORCE rule (for which training error approaches one in the first trial,, while test error does so after 30-50 trials,; fig. 2A ). For target signals that extend over 10 seconds (same smoothness of the target functions, two insets in fig. 2B), the reward-modulated Hebbian rule achieves a performance of 1 after 200 trials if combined with input from the attractor network (fig. 2B) but fails completely without the attractor network (tuning of the hyperparameters on a logarithmic scale did not help; data not shown). Thus a threefactor learning rule succeeds to learn complex tasks if combined with a temporally structured input from the attractor network. FORCE learning needs a feedback signal at every time step. Standard reward-modulated Hebbian learning can support very small delays, but fails if updates are less frequent than every few ms. In our approach (fig. 2C), proposed updates are summed up in the , but applied only at the end of a one-second trial. We find that even with such a temporally sparse update, learning is still possible. The input from the dynamic working memory is necessary to achieve this task: when the strength of the input from the attractor network gradually decreases, performance drops; in the total absence of attractor input (c = 0.0; note that the reservoir still receives weak input noise) learning completely fails. Strikingly, delayed updates do not hurt performance, and the system achieves high (> 0.9) cross-correlation in fewer than 100 training trials if the input from the attractor network is strong enough. The transient drop in performance shortly after the start in fig. 2C is likely due to W ro = 0 in the beginning, meaning that the output is uncorrelated with the firing rates, and therefore the cumulative weight update does not approximate gradient information. It is well known that reservoir networks can learn multiple tasks given different initial conditions with both FORCE and the reward-modulated Hebbian rule. We want to check whether this also holds for our approach. We conjecture that different inputs to the attractor network generate unique neural trajectories that can be exploited by the reservoir network. To test this hypothesis, we train the network to produce hand-written digits. The static input to the attractor comes from the pre-processed MNIST dataset (network inputs are taken from one of the last layers of a deep network trained to classify MNIST) in order to provide a realistic input to the attractor network which transforms the static input into dynamic trajectories (noiseless, fig. 3B, and noisy, fig. 3D). We record 50 attractor trajectories used for training (used 4 times each, ing in 2000 training trials) and 50 for testing of each digit (1 second each), where each trajectory corresponds to a distinct input pattern. The reservoir learns a single drawing for each class. The variance of the structural noise in the attractor network is 3 times larger compared to the previous experiments in order to produce more robust bump trajectories (fig. 3D). The reward-modulated Hebbian rule masters 10 out of 10 digits when driven by a noiseless input from the attractor network (fig. 3A). In the presence of noise in the attractor network (fig. 3D), the performance is imperfect for "five" and "six" (fig. 3C). We checked that FORCE learning with the same noisy input did not improve the performance (data not shown). Note that a linear readout of the attractor (without the reservoir) would be insufficient: first, sometimes single digit trajectories are very dissimilar (e.g. the different zero's in fig. 3D); second, at points where trajectories cross each other, a delay-less linear readout must produce the same output, no matter what the digit is. We showed that a dynamic working memory can facilitate learning of complex tasks with biologically plausible three-factor learning rules. Our indicate that, when combined with a bump attractor, reservoir computing with reward-modulated learning can be as efficient as FORCE, a widely used but biologically unrealistic rule. The proposed network relies on a limited number of trajectories in the attractor network. To increase its capacity, a possible future direction would be to combine input from the attractor network with another, also input-specific, but transient input that would bring the reservoir into a different initial state. In this case the attractor network would work as a time variable (as in ), and the other input as the control signal (as in ). Apart from the biological relevance, the proposed method might be used for real-world applications of reservoir computing (e.g. wind forecasting ) as it is computationally less expensive than FORCE. It might also be an interesting alternative for learning in neuromorphic devices. Simulation details. Both networks were simulated with the Euler method with the step size dt = 1 ms. The attractor network dynamics was recorded after a 100 ms warm up period to allow creation of the bump solution, during which it received additional input from the images in section 3.3. Training was done consequently, without breaks in the dynamics between trials. For testing, the network started from the preceding training state and continued with frozen weights. After testing, the pre-training activity was restored. The code for experiments is available at https://github.com/neuroai-workshopanon-1224113/working-memory-facilitating-reservoir-learning. Test functions. Gaussian process test function were drawn from Forcing both ends of the function to be zero and denoting x = (0, T − 1), z = (1, . . ., T − 2), we sample test functions as where T is either 10 3 (short tasks) or 10 4 (long tasks). We chose σ 2 to roughly match the complexity of targets from (σ 2 = 10 4). 50 random functions were tested on 50 random reservoirs that nevertheless received the same attractor input (W attr was not resampled). In section 3.3, the same reservoir was used for all runs. The noisy input for section 3.3 was taken from an intermediate layer of a deep network trained to classify MNIST, and the noiseless input stimulated only a 5 by 5 square of neurons (unique for each digit). Attractor network parameters. The time constants were τ m = 30 ms, τ a = 400 ms. Adaptation strength was s = 1.5. The external input e was drawn independently for each neuron from a Gaussian distribution N (1, 0.0025 2). In section 3.3, the task-specific input was added to the noisy one. For the connectivity matrix J = J s + J h, the noisy part was drawn independently as (J h) ij ∼ N (0, σ 2 /N attr), with N attr = 2500 and σ = 2 in all experiments except for section 3.3, where we used σ = 6 for more robust trajectories. The symmetric part arranged the neurons on a 2D grid, such that every neuron i had its coordinates x i and y i ranging from 0 to 49. The connectivity led to mutual excitation of nearby neurons and inhibition of the distant ones, The bump center (used in fig. 3B and D) corresponded to the mean of the activity on the torus. Denoting activity of each neuron as r(x, y), the center on the x axis was calculated as where "angle" computes the counterclockwise angle of a complex variable (ranging from 0 to 2π). Reservoir network parameters. The time constant was τ = 50 ms, and total coupling strength was λ = 1.5. The readout weights W ro were initialized to zero. The feedback weights were drawn independently from a uniform distribution as (W fb) ij ∼ U(−1, 1). Both the recurrent connections and the weights from the attractor to the reservoir were drawn independently as (W rec) ij, (W attr) ij ∼ N (0, 1/pN res) · Be(p), with p = 0.1, N res = 1000, and Be being the Bernoulli distribution. A new reservoir, and thus W rec, was sampled for each new test function. The matrix W rec was the same for all tasks except the last one in section 3.3. State noise ξ ξ ξ and exploratory noise η η η were generated independently from the uniform distribution as ξ i ∼ U(−0.05, 0.05), η i ∼ U(−0.5, 0.5). When attractor was present, the reservoir neurons also received weak independent noise drawn from N (0, 0.0025). Learning rule. Low-pass filtering was done as x(t + dt) =x(t) + dt (x(t) −x(t))/τ f, τ f = 5 ms,x = 0. The learning rate η(t) was computed as η(t) = η 0 /(1 + t/τ l) (η 0 = 5 · 10 −4, τ l = 2 · 10 4 ms) and held at η 0 in section 3.2 to make independent of the decay.

Represent the short summary for finding the corresponding scientific article (long)

We show that a working memory input to a reservoir network makes a local reward-modulated Hebbian rule perform as well as recursive least-squares (aka FORCE)

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the article for finding a single-sentence TLDR

Question: Convolutional architectures have recently been shown to be competitive on many sequence modelling tasks when compared to the de-facto standard of recurrent neural networks (RNNs) while providing computational and modelling advantages due to inherent parallelism. However, currently, there remains a performance gap to more expressive stochastic RNN variants, especially those with several layers of dependent random variables. In this work, we propose stochastic temporal convolutional networks (STCNs), a novel architecture that combines the computational advantages of temporal convolutional networks (TCN) with the representational power and robustness of stochastic latent spaces. In particular, we propose a hierarchy of stochastic latent variables that captures temporal dependencies at different time-scales. The architecture is modular and flexible due to the decoupling of the deterministic and stochastic layers. We show that the proposed architecture achieves state of the art log-likelihoods across several tasks. Finally, the model is capable of predicting high-quality synthetic samples over a long-range temporal horizon in modelling of handwritten text. Generative modeling of sequence data requires capturing long-term dependencies and learning of correlations between output variables at the same time-step. Recurrent neural networks (RNNs) and its variants have been very successful in a vast number of problem domains which rely on sequential data. Recent work in audio synthesis, language modeling and machine translation tasks BID8 BID9 BID13 has demonstrated that temporal convolutional networks (TCNs) can also achieve at least competitive performance without relying on recurrence, and hence reducing the computational cost for training. Both RNNs and TCNs model the joint probability distribution over sequences by decomposing the distribution over discrete time-steps. In other words, such models are trained to predict the next step, given all previous time-steps. RNNs are able to model long-term dependencies by propagating information through their deterministic hidden state, acting as an internal memory. In contrast, TCNs leverage large receptive fields by stacking many dilated convolutions, allowing them to model even longer time scales up to the entire sequence length. It is noteworthy that there is no explicit temporal dependency between the model outputs and hence the computations can be performed in parallel. The TCN architecture also introduces a temporal hierarchy: the upper layers have access to longer input sub-sequences and learn representations at a larger time scale. The local information from the lower layers is propagated through the hierarchy by means of residual and skip connections BID2.However, while TCN architectures have been shown to perform similar or better than standard recurrent architectures on particular tasks BID2, there currently remains a performance gap to more recent stochastic RNN variants BID3 BID7 BID11 BID12 BID14 BID25. Following a similar approach to stochastic RNNs, BID21 present a significant improvement in the log-likelihood when a TCN model is coupled with latent variables, albeit at the cost of limited receptive field size. The computational graph of generative (left) and inference (right) models of STCN. The approximate posterior q is conditioned on dt and is updated by the prior p which is conditioned on the TCN representations of the previous time-step dt−1. The random latent variables at the upper layers have access to a long history while lower layers receive inputs from more recent time steps. In this work we propose a new approach for augmenting TCNs with random latent variables, that decouples deterministic and stochastic structures yet leverages the increased modeling capacity efficiently. Motivated by the simplicity and computational advantages of TCNs and the robustness and performance of stochastic RNNs, we introduce stochastic temporal convolutional networks (STCN) by incorporating a hierarchy of stochastic latent variables into TCNs which enables learning of representations at many timescales. However, due to the absence of an internal state in TCNs, introducing latent random variables analogously to stochastic RNNs is not feasible. Furthermore, defining conditional random variables across time-steps would in breaking the parallelism of TCNs and is hence undesirable. In STCN the latent random variables are arranged in correspondence to the temporal hierarchy of the TCN blocks, effectively distributing them over the various timescales (see FIG0 . Crucially, our hierarchical latent structure is designed to be a modular add-on for any temporal convolutional network architecture. Separating the deterministic and stochastic layers allows us to build STCNs without requiring modifications to the base TCN architecture, and hence retains the scalability of TCNs with respect to the receptive field. This conditioning of the latent random variables via different timescales is especially effective in the case of TCNs. We show this experimentally by replacing the TCN layers with stacked LSTM cells, leading to reduced performance compared to STCN.We propose two different inference networks. In the canonical configuration, samples from each latent variable are passed down from layer to layer and only one sample from the lowest layer is used to condition the prediction of the output. In the second configuration, called STCN-dense, we take inspiration from recent CNN architectures BID18 and utilize samples from all latent random variables via concatenation before computing the final prediction. Our contributions can thus be summarized as: 1) We present a modular and scalable approach to augment temporal convolutional network models with effective stochastic latent variables. 2) We empirically show that the STCN-dense design prevents the model from ignoring latent variables in the upper layers BID32. 3) We achieve state-of-the-art log-likelihood performance, measured by ELBO, on the IAM-OnDB, Deepwriting, TIMIT and the Blizzard datasets. 4) Finally we show that the quality of the synthetic samples matches the significant quantitative improvements. Auto-regressive models such as RNNs and TCNs factorize the joint probability of a variable-length sequence x = {x 1, . . ., x T} as a product of conditionals as follows: DISPLAYFORM0 where the joint distribution is parametrized by θ. The prediction at each time-step is conditioned on all previous observations. The observation model is frequently chosen to be a Gaussian or Gaussian mixture model (GMM) for real-valued data, and a categorical distribution for discrete-valued data. In TCNs the joint probabilities in Eq. are parametrized by a stack of convolutional layers. Causal convolutions are the central building block of such models and are designed to be asymmetric such that the model has no access to future information. In order to produce outputs of the same size as the input, zero-padding is applied at every layer. In the absence of a state transition function, a large receptive field is crucial in capturing long-range dependencies. To avoid the need for vast numbers of causal convolution layers, typically dilated convolutions are used. Exponentially increasing the dilation factor in an exponential growth of the receptive field size with depth BID31 BID2. In this work, without loss of generality, we use the building blocks of Wavenet as gated activation units have been reported to perform better. A deterministic TCN representation d l t at time-step t and layer l summarizes the input sequence x 1:t: DISPLAYFORM0 where the filter width is 2 and j denotes the dilation step. In our work, the stochastic variables z l, l = 1... L are conditioned on TCN representations d l that are constructed by stacking K Wavenet blocks over the previous d l−1 (for details see FIG6 in Appendix). VAEs BID20 BID24 introduce a latent random variable z to learn the variations in the observed non-sequential data where the generation of the sample x is conditioned on the latent variable z. The joint probability distribution is defined as: DISPLAYFORM0 and parametrized by θ. Optimizing the marginal likelihood is intractable due to the non-linear mappings between z and x and the integration over z. Instead the VAE framework introduces an approximate posterior q φ (z|x) and optimizes a lower-bound on the marginal likelihood: DISPLAYFORM1 where KL denotes the Kullback-Leibler divergence. Typically the prior p θ (z) and the approximate q φ (z|x) are chosen to be in simple parametric form, such as a Gaussian distribution with diagonal covariance, which allows for an analytical calculation of the KL-term in Eq.. An RNN captures temporal dependencies by recursively processing each input, while updating an internal state h t at each time-step via its state-transition function: DISPLAYFORM0 where f (h) is a deterministic transition function such as LSTM BID17 or GRU BID6 cells. The computation has to be sequential because h t depends on h t−1.The VAE framework has been extended for sequential data, where a latent variable z t augments the RNN state h t at each sequence step. The joint distribution p θ (x, z) is modeled via an auto-regressive model which in the following factorization: DISPLAYFORM1 In contrast to the fixed prior of VAEs, N (0, I), sequential variants define prior distributions conditioned on the RNN hidden state h and implicitly on the input sequence x BID7., and the inference model (right), which is shared by both variants. Diamonds represent the outputs of deterministic dilated convolution blocks where the dependence of dt on the past inputs is not shown for clarity (see Eq. FORMULA1). xt and zt are observable inputs and latent random variables, respectively. The generative task is to predict the next step in the sequence, given all past steps. Note that in the STCN-dense variant the next step is conditioned on all latent variables z l t for l = 1... L. The mechanics of STCNs are related to those of VRNNs and LVAEs. Intuitively, the RNN state h t is replaced by temporally independent TCN layers d l t. In the absence of an internal state, we define hierarchical latent variables z l t that are conditioned vertically, i.e., in the same time-step, but independent horizontally, i.e., across time-steps. We follow a similar approach to LVAEs in defining the hierarchy in a top-down fashion and in how we estimate the approximate posterior. The inference network first computes the approximate likelihood, and then this estimate is corrected by the prior, ing in the approximate posterior. The TCN layers d are shared between the inference and generator networks, analogous to VRNNs BID7. To preserve the parallelism of TCNs, we do not introduce an explicit dependency between different time-steps. However, we suggest that conditioning a latent variable z l−1 t on the preceding variable z l t implicitly introduces temporal dependencies. Importantly, the random latent variables in the upper layer have access to a larger receptive field due to its deterministic input d l t−1, whereas latent random variables in lower layers are updated with different, more local information. However, the latent variable z l−1 t may receive longer-range information from z l t. The generative and inference models are jointly trained by optimizing a step-wise variational lower bound on the log-likelihood BID20 BID24. In the following sections we describe these components and build up the lower-bound for a single time-step t. Each sequence step x t is generated from a set of latent variables z t, split into layers as follows: DISPLAYFORM0 where DISPLAYFORM1 Here the prior is modeled by a Gaussian distribution with diagonal covariance, as is common in the VAE framework. The subscript p denotes items of the generative distribution. For the inference distribution we use the subscript q. The distributions are parameterized by a neural network f We propose two variants of the observation model. In the non-sequential scenario, the observations are defined to be conditioned on only the last latent variable in the hierarchy, i.e., p θ (x t |z 1 t), following Sønderby et al. FORMULA0; BID16 and BID24 our STCN variant uses the same observation model, allowing for an efficient optimization. However, latent units are likely to become inactive during training in this configuration BID5 BID4 BID32 ing in a loss of representational power. The latent variables at different layers are conditioned on different contexts due to the inputs d l t. Hence, the latent variables are expected to capture complementary aspects of the temporal context. To propagate the information all the way to the final prediction and to ensure that gradients flow through all layers, we take inspiration from BID18 and directly condition the output probability on samples from all latent variables. We call this variant of our architecture STCN-dense. The final predictions are then computed by the respective observation functions: DISPLAYFORM0 where f (o) corresponds to the output layer constructed by stacking 1D convolutions or Wavenet blocks depending on the dataset. In the original VAE framework the inference model is defined as a bottom-up process, where the latent variables are conditioned on the stochastic layer below. Furthermore, the parameterization of the prior and approximate posterior distributions are computed separately BID5 BID24. In contrast, propose a top-down dependency structure shared across the generative and inference models. From a probabilistic point of view, the approximate Gaussian likelihood, computed bottom-up by the inference model, is combined with the Gaussian prior, computed top-down from the generative model. We follow a similar procedure in computing the approximate posterior. First, the parameters of the approximate likelihood are computed for each stochastic layer l: DISPLAYFORM0 followed by the downward pass, recursively computing the prior and approximate posterior by precision-weighted addition: DISPLAYFORM1 Finally, the approximate posterior has the same decomposition as the prior (see Eq. FORMULA6): DISPLAYFORM2 DISPLAYFORM3 Note that the inference and generative network share the parameters of dilated convolutions Conv (l). The variational lower-bound on the log-likelihood at time-step t can be defined as follows: DISPLAYFORM0 Using the decompositions from Eq. FORMULA6 and FORMULA0, the Kullback-Leibler divergence term becomes: DISPLAYFORM1 The KL term is the same for the STCN and STCN-dense variants. The reconstruction term L Recon t, however, is different. In STCN we only use samples from the lowest layer of the hierarchy, whereas in STCN-dense we use all latent samples in the observation model: DISPLAYFORM2 DISPLAYFORM3 In the dense variant, samples drawn from the latent variables z l t are carried over the dense connections. Similar to, the expectation over z One alternative option to use the latent samples could be to sum individual samples before feeding them into the observation model, i.e., sum([z . We empirically found that this does not work well in STCN-dense. Instead, we concatenate all samples [z DISPLAYFORM4 DISPLAYFORM5 analogously to DenseNet BID18 and BID19 . We evaluate the proposed variants STCN and STCN-dense both quantitatively and qualitatively on modeling of digital handwritten text and speech. We compare with vanilla TCNs, RNNs, VRNNs and state-of-the art models on the corresponding tasks. In our experiments we use two variants of the Wavenet model: the original model proposed in and a variant that we augment with skip connections analogously to STCN-dense. This additional baseline evaluates the benefit of learning multi-scale representations in the deterministic setting. Details of the experimental setup are provided in the Appendix. Our code is available at https://ait.ethz.ch/projects/2019/stcn/.Handwritten text: The IAM-OnDB and Deepwriting datasets consist of digital handwriting sequences where each time-step contains real-valued (x, y) pen coordinates and a binary pen-up event. The IAM-OnDB data is split and pre-processed as done in BID7. BID1 extend this dataset with additional samples and better pre-processing. BID7 26643 7413 1358 528 * VRNN (Normal) BID7 ≈ 30235 ≈ 9516 ≈ 1354 ≥ 495 * VRNN (GMM) BID7 ≈ 29604 ≈ 9392 ≈ 1384 ≥ 673 * SRNN (Normal) BID12 ≥ 60550 ≥ 11991 n/a n/a Z-forcing (Normal) BID14 ≥ 70469 ≥ 15430 n/a n/a Var. Bi-LSTM (Normal) BID25 ≥ 73976 ≥ 17319 n/a n/a SWaveNet (Normal) BID21 ≥ 72463 ≥ 15708 DISPLAYFORM0 ≥ 77438 ≥ 17670 n/a n/a the STCN-dense version. The same relative ordering is maintained on the Deepwriting dataset, indicating that the proposed architecture is robust across datasets. Fig. 3 compares generated handwriting samples. While all models produce consistent style, our model generates more natural looking samples. Note that the spacing between words is clearly visible and most of the letters are distinguishable. Speech modeling: TIMIT and Blizzard are standard benchmark dataset in speech modeling. The models are trained and tested on 200 dimensional real-valued amplitudes. We apply the same pre-processing as BID7. For this task we introduce STCN-dense-large, with increased model capacity. Here we use 512 instead of 256 convolution filters. Note that the total number of model parameters is comparable to SWaveNet and other SOA models. On TIMIT, STCN-dense TAB0 significantly outperforms the vanilla TCN and RNN, and stochastic models. On the Blizzard dataset, our model is marginally better than the Variational Bi-LSTM. Note that the inference models of SRNN BID12, Z-forcing BID14, and Variational Bi-LSTM BID25 receive future information by using backward RNN cells. Similarly, SWaveNet BID21 applies causal convolutions in the backward direction. Hence, the latent variable can be expected to model future dynamics of the sequence. In contrast, our models have only access to information up to the current time-step. These indicate that the STCN variants perform very well on the speech modeling task. Latent Space Analysis: BID32 observe that in hierarchical latent variable models the upper layers have a tendency to become inactive, indicated by a low KL loss BID10. TAB2 shows the KL loss per latent variable and the corresponding log-likelihood measured by ELBO in our models. Across the datasets it can be observed that our models make use of many of the latent variables which may explain the strong performance across tasks in terms of log-likelihoods. Note that STCN uses a standard hierarchical structure. However, individual latent variables have different information context due to the corresponding TCN block's receptive field. This observation suggests that the proposed combination of TCNs and stochastic variables is indeed effective. Furthermore, in STCN we see a similar utilization pattern of the z variables across tasks, whereas STCN-dense may have more flexibility in modeling the temporal dependencies within the data due to its dense connections to the output layer. Replacing TCN with RNN: To better understand potential symergies between dilated CNNs and the proposed latent variable hierarchy, we perform an ablation study, isolating the effect of TCNs and the latent space. To this end the deterministic TCN blocks are replaced with LSTM cells by keeping the latent structure intact. We dub this condition LadderRNN. We use the TIMIT and IAM-OnDB datasets for evaluation. TAB3 summarizes performance measured by the ELBO.The most direct translation of the the STCN architecture into an RNN counterpart has 25 stacked LSTM cells with 256 units each. Similar to STCN, we use 5 stochastic layers (see Appendix 7.1). Note that stacking this many LSTM cells is unusual and ed in instabilities during training. Hence, the performance is similar to vanilla RNNs. The second LadderRNN configuration uses 5 stacked LSTM cells with 512 units and a one-to-one mapping with the stochastic layers. On the TIMIT dataset, all LadderRNN configurations show a significant improvement. We also observe a pattern of improvement with densely connected latent variables. This experiments shows that the proposed modular latent variable design does allow for the usage of different building blocks. Even when attached to LSTM cells, it boosts the log-likelihood performance (see 5x512-LadderRNN), in particular when used with dense connections. However, the empirical suggest that the densely connected latent hierarchy interacts particularly well with dilated CNNs. We suggest this is due to the hierarchical nature on both sides of the architecture. On both datasets STCN models achieved the best performance and significantly improve with dense connections. This supports our contribution of a latent variable hierarchy, which models different aspects of information from the input time-series. DISPLAYFORM1 5 RELATED WORK BID24 propose Deep Latent Gaussian Models (DLGM) and propose the Ladder Variational Autoencoder (LVAE). In both models the latent variables are hierarchically defined and conditioned on the preceding stochastic layer. LVAEs improve upon DLGMs via implementation of a top-down hierarchy both in the generative and inference model. The approximate posterior is computed via a precisionweighted update of the approximate likelihood (i.e., the inference model) and prior (i.e., the generative model). Similarly, the PixelVAE BID16 incorporates a hierarchical latent space decomposition and uses an autoregressive decoder. BID32 show under mild conditions that straightforward stacking of latent variables (as is done e.g. in LVAE and PixelVAE) can be ineffective, because the latent variables that are not directly conditioned on the observation variable become inactive. Due to the nature of the sequential problem domain, our approach differs in the crucial aspects that STCNs use dynamic, i.e., conditional, priors BID7 at every level. Moreover, the hierarchy is not only implicitly defined by the network architecture but also explicitly defined by the information content, i.e., receptive field size. Dieng et al. FORMULA0 both theoretically and empirically show that using skip connections from the latent variable to every layer of the decoder increases mutual information between the latent and observation variables. Similar to BID10 in STCN-dense, we introduce skip connections from all latent variables to the output. In STCN the model is expected to encode and propagate the information through its hierarchy. BID30 suggest using autoregressive TCN decoders to remedy the posterior collapse problem observed in language modeling with LSTM decoders BID4. BID29 and BID9 use TCN decoders conditioned on discrete latent variables to model audio signals. Stochastic RNN architectures mostly vary in the way they employ the latent variable and parametrize the approximate posterior for variational inference. BID7 and BID3 use the latent random variable to capture high-level information causing the variability observed in sequential data. Particularly BID7 shows that using a conditional prior rather than a standard Gaussian distribution is very effective in sequence modeling. In BID12 BID14 BID25 ), the inference model, i.e., the approximate posterior, receives both the past and future summaries of the sequence from the hidden states of forward and backward RNN cells. The KL-divergence term in the objective enforces the model to learn predictive latent variables in order to capture the future states of the sequence. BID21's SWaveNet is most closely related to ours. SWaveNet also introduces latent variables into TCNs. However, in SWaveNet the deterministic and stochastic units are coupled which may prevent stacking of larger numbers of TCN blocks. Since the number of stacked dilated convolutions determines the receptive field size, this directly correlates with the model capacity. For example, the performance of SWaveNet on the IAM-OnDB dataset degrades after stacking more than 3 stochastic layers BID21, limiting the model to a small receptive field. In contrast, we aim to preserve the flexibility of stacking dilated convolutions in the base TCN. In STCNs, the deterministic TCN units do not have any dependency on the stochastic variables (see FIG0) and the ratio of stochastic to deterministic units can be adjusted, depending on the task. In this paper we proposed STCNs, a novel auto-regressive model, combining the computational benefits of convolutional architectures and expressiveness of hierarchical stochastic latent spaces. We have shown the effectivness of the approach across several sequence modelling tasks and datasets. The proposed models are trained via optimization of the ELBO objective. Tighter lower bounds such as IWAE BID5 or FIVO may further improve modeling performance. We leave this for future work. The network architecture of the proposed model is illustrated in FIG6. We make only a small modification to the vanilla Wavenet architecture. Instead of using skip connections from Wavenet blocks, we only use the latent sample zt in order to make a prediction of xt. In STCN-dense configuration, zt is the concatenation of all latent variables in the hierarchy, i.e., zt = [z Output layer f (o): For the IAM-OnDB and Deepwriting datasets we use 1D convolutions with ReLU nonlinearity. We stack 5 of these layers with 256 filters and filter size 1. DISPLAYFORM0 For TIMIT and Blizzard datasets Wavenet blocks in the output layer perform significantly better. We stack 5 Wavenet blocks with dilation size 1. For each convolution operation in the block we use 256 filters. The filter size of the dilated convolution is set to 2. The STCN-dense-large model is constructed by using 512 filters instead of 256. l t: The number of Wavenet blocks is usually determined by the desired receptive field size.• For the handwriting datasets K = 6 and L = 5. In total we have 30 Wavenet blocks where each convolution operation has 256 filters with size 2.• For speech datasets K = 5 and L = 5. In total we have 25 Wavenet blocks where each convolution operation has 256 filters with size 2. The large model configuration uses 512 filters. p and f q: The number of stochastic layers per task is given by L. We used dimensional latent variables for the handwriting tasks. It is for speech datasets. Note that the first entry of the list corresponds to z 1.The mean and sigma parameters of the Normal distributions modeling the latent variables are calculated by the f (l) p and f (l) q networks. We stack 2 1D convolutions with ReLU nonlinearity and filter size 1. The number of filters are the same as the number of Wavenet block filters for the corresponding task. Finally, we clamped the latent sigma predictions between 0.001 and 5. In all STCN experiments we applied KL annealing. In all tasks, the weight of the KL term is initialized with 0 and increased by 1 × e −4 at every step until it reaches 1.The batch size was 20 for all datasets except for Blizzard where it was 128.We use the ADAM optimizer with its default parameters and exponentially decay the learning rate. For the handwriting datasets the learning rate was initialized with 5 × e −4 and followed a decay rate of 0.94 over 1000 decay steps. On the speech datasets it was initialized with 1 × e −3 and decayed with a rate of 0.98. We applied early stopping by measuring the ELBO performance on the validation splits. We implement STCN models in Tensorflow BID0. Our code and models achieving the SOA are available at https://ait.ethz.ch/projects/2019/stcn/. Here we provide the extended table with Normal observation model entries for available models. Table 4: Average log-likelihood per sequence on TIMIT, Blizzard, IAM-OnDB and Deepwriting datasets. (Normal) and (GMM) stand for unimodal Gaussian or multi-modal Gaussian Mixture Model (GMM) as the observation model BID15 BID7. Asterisks * indicate that we used our re-implementation only for the Deepwriting dataset.

Represent the TLDR for finding the corresponding scientific article

We combine the computational advantages of temporal convolutional architectures with the expressiveness of stochastic latent variables.

We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The weak contraction mapping is a self mapping that the range is always a subset of the domain, which admits a unique fixed-point. The iteration of weak contraction mapping is a Cauchy sequence that yields the unique fixed-point. A gradient-free optimization method as an application of weak contraction mapping is proposed to achieve global minimum convergence. The optimization method is robust to local minima and initial point position. Many gradient-based optimization methods, such as gradient descent method, Newton's method and so on, face great challenges in finding the global minimum point of a function. As is known, searching for the global minimum of a function with many local minima is difficult. In principle, the information from the derivative of a single point is not sufficient for us to know the global geometry property of the function. For a successful minimum point convergence, the initial point is required to be sufficiently good and the derivative calculation need to be accurate enough. In the gradientbased methods, the domain of searching area will be divided into several subsets with regards to local minima. And eventually it will converge to one local minimum depends on where the initial point locates at. Let (X,d) be a metric space and let T:X → X be a mapping. For the inequality that, d(T (x), T (y)) ≤ qd(x, y), ∀x, y ∈ X.if q ∈, T is called contractive; if q ∈, T is called nonexpansive; if q < ∞, T is called Lipschitz continuous(1; 2). The gradient-based methods are usually nonexpansive mapping the solution exists but is not unique for general situation. For instance, if the gradient descent method is written as a mapping T and the objective function has many local minima, then there are many fixed points accordingly. From the perspective of spectra of bounded operator, for a nonexpansive mapping any minima of the objective function is an eigenvector of eigenvalue equation T (x) = λx,in which λ = 1. In the optimization problem, nonexpansive mapping sometimes works but their disadvantages are obvious. Because both the existence and uniqueness of solution are important so that the contractive mapping is more favored than the nonexpansive mapping(3; 4).Banach fixed-point theorem is a very powerful method to solve linear or nonlinear system. But for optimization problems, the condition of contraction mapping T: X → X that d(T (x), T (y)) ≤ qd(x, y) is usually too strict and luxury. In the paper, we are trying to extend the Banach fixedpoint theorem to an applicable method for optimization problem, which is called weak contraction mapping. In short, weak contraction mapping is a self mapping that always map to the subset of its domain. It is proven that weak contraction mapping admits a fixed-point in the following section. How to apply the weak contraction mapping to solve an optimization problem? Geometrically, given a point, we calculate the height of this point and utilize a hyperplane at the same height to cut the objective function, where the intersection between the hyperplane and the objective function will form a contour or contours. And then map to a point insider a contour, which the range of this mapping is always the subset of its domain. The iteration of the weak contraction mapping yields a fixed-point, which coincides with the global minimum of the objective function. In this section, the concept of weak contraction mapping and its fixed-point will be discussed in detail. Definition 1. Let (X, d and D) be a metric space. Both the metric measurement d and D are defined in the space. And the metric measurement D(X) refers to the maximum distance between two points in the vector space X: DISPLAYFORM0 Definition 2. Let (X, d and D) be a complete metric space. Then a mapping T: X → X is called weak contraction mapping on X if there exists DISPLAYFORM1 The weak contraction mapping is an extension of contraction map with a looser requirement that DISPLAYFORM2 and D) be a non-empty complete metric space with weak contraction mapping T: X → X. Then T admits a unique fixed-point x * in X when X 0 is decided. Let x 0 ∈ X be arbitrary and define a sequence {x n} be setting: x n = T (x n−1). The Theorem.1 is proven in the following lemmas. By definition, there exists q DISPLAYFORM3 Cauchy sequence in (X, d and D) and hence converges to a limit x * in X 0.Proof. Let m, n ∈ N such that m > n. DISPLAYFORM4 Let > 0 be arbitrary, since q ∈, we can find a large N ∈ N such that DISPLAYFORM5 Hence, by choosing m, n large enough: DISPLAYFORM6 Thus, {x n} is Cauchy and converges to a point x * ∈ X 0. DISPLAYFORM7 Proof. DISPLAYFORM8 ) with regards to a specific X 0.Proof. Suppose there exists another fixed-point y that T (y) = y, then choose the subspace X i that both the x * and y are the only elements in X i. By definition, X i+1 = R(T (X i)) so that, both the x * and y are elements in X i+1, namely, DISPLAYFORM9 Let a hyperplane L cut the objective function f(x), the intersection of L and f(x) forms a contour (or contours). Observing that the contour (or contours) will divide X into two subspaces the higher subspace X >:= {x | f (x) > h, ∀x ∈ X} and the lower subspace DISPLAYFORM10 Geometrically, the range of weak contraction mapping shrinks over iterates, such that, DISPLAYFORM11. Based on lemma.1.3, the D(X i) measurement converges to zero as i goes to infinity, namely, lim DISPLAYFORM12 And the sequence of iteration DISPLAYFORM13 is Cauchy sequence that converge to the global minimum of objective function f(x) if the f(x) has a unique global minimum point. Lemma 1.5. Provided there is a unique global minimum point of an objective function, then x * is the global minimum point of the function. Proof. The global minimum point must be insider the lower space {X ≤ i, ∀i ∈ N 0}. Similar to the proof of uniqueness of fixed-point, suppose the global minimum point x min of objective function is different from x *. By measuring the distance between fixed-point X * and the global minimum point DISPLAYFORM14 The inequality above indicates d(x *, x min) = 0, thus x * = x min.Compared with contraction map, the weak contraction map is much easier to implement in the optimization problem as the requirement DISPLAYFORM15 t require x i in sequence {x n} must move closer to each other for every step but confine the range of x i to be smaller and smaller. Therefore, the sequence {x n} can still be a Cauchy and has the asymptotic behavior to converge to the fixed-point. Given the objective function f (x) has a unique global minimum point, the task is to find a weak contraction mapping T: X → X such that the unique fixed-point of mapping is the global minimum point of the function. The weak contraction map for the optimization problem can be implemented in following way. First, provide one arbitrary initial point x 0 to the function and calculate the height L = f (x 0) of the point and this height is the corresponding contours' level; Second, given the initial point map to another point inside the contour. One practical way is to solve the equation f (x) = L and get n number of roots which locate on a contour(or contours) and then the average of these roots is the updated searching point. And then repeat these process until the iteration of searching point converge. This contour-based optimization algorithm utilizes the root-finding algorithm to solve the equation f (x) = L and get n number of roots. The starting point for the root-finding algorithm is generated by a random number generator. This stochastic process will help the roots to some extent widely distribute over the contour rather than concentrate on somewhere. The inequality d(x m, x n) ≤ q n D(X 0) indicates the rate of convergence, namely, the smaller q is the high rate of convergence will be achieved. Geometrically, the equation f (x) = L is the intersection of the objective function and a hyperplane whose height is L. We hope the hyperplane move downward in a big step during each iterate and the centroid of the contour refers to the most likely minimum position. Therefore, averaging the roots as an easy and effective way to map somewhere near the centroid. And there is a trade-off between the number of roots on the contour and the rate of convergence. The larger amount of roots on the contour, the more likely the average locates closer to the centroid of the contour, and then the less iterates are required for convergence. In another words, the more time spend on finding roots on a contour, the less time spend on the iteration, vice verse. The global minimum point x * is the fixed-point of the iteration x i+1 = T x i and solves the equation It is worth noting that the size of contour become smaller and smaller during the iterative process and eventually converge to a point, which is the minimum point of the function. DISPLAYFORM0 As shown in the previous examples, averaging the roots on the contour is an effective approach to map a point inside the interior of the lower space X ≤ when it is convex. However, in general situation, the space X ≤ is not guaranteed to be convex. In that case, it is important to decompose the lower space X ≤ into several convex subsets. In this study, the key intermediate step is to check whether two roots belong to the same convex subset and decompose all roots into several convex subsets accordingly. One practical way to achieve that is to pair each two roots and scan function's value along the segment between the two roots and check whether there exists a point higher than contour's level. Loosely speaking, if two roots belong to the same convex subset, the value of function along the segment is always lower than the contour's level. Otherwise, the value of function at somewhere along the segment will be higher than the contour's level. Traverse all the roots and apply this examination on them, then we can decompose the roots with regards to different convex subsets. This method is important to map a point insider interior of a contour and make hyperplane move downwards. To check whether two roots belong to the same convex subset, N number of random points along the segment between two roots are checked whether higher than the contour's level or not. When we want to check the function's value along the segment between r m and r n. The vector k = r m − r n is calculated so that the random point p i locate on the segment can be written as p i = r n + (r m − r n), ∈, where the is a uniform random number from 0 to 1. Then check whether the inequality holds for all random point such that f (p i) < f (r m), ∀i ≤ N. Obviously, the more random points on the segment are checked, the less likely the point higher than contour's level is missed(9; 10).After the set of roots are decomposed into several convex subsets, the averages of roots with regards to each subsets are calculated and the lowest one is returned as an update point from each iterate. Thereafter, the remaining calculation is repeat the iterate over and over until convergence and return the converged point as the global minimum. Nevertheless, the algorithm has been tested on Ackley function where the global minimum locates at that f = 0. And the first 6 iterates of roots and contours is shown FIG2 and the minimum point (-0.00000034,0.00000003) return by algorithm is shown in TABLE. 3. The test shows that the optimization algorithm is robust to local minima and able to achieve the global minimum convergence. The quest to find to the global minimum pays off handsomely. When the optimization method is tested on Ackley function, the average of roots and the level of contour for each iteration is shown above. In summary, the main procedure of the stochastic contour-based optimization method is decomposed into following steps: 1. Given the initial guess point x for the objective function and calculate the contour level L; 2. Solve the equation f (x) = L and get n number of roots. Decompose the set of roots into several convex subsets,return the lowest average of roots as an update point from each iterate; 3. Repeat the above iterate until convergence. The weak contraction mapping is a self mapping that always map to a subset of domain. Intriguingly, as an extension of Banach fixed-point theorem, the iteration of weak contraction mapping is a Cauchy and yields a unique fixed-point, which fit perfectly with the task of optimization. The global minimum convergence regardless of initial point position and local minima is very significant strength for optimization algorithm. We hope that the advanced optimization with the development of the weak contraction mapping can contribute to empower the modern calculation.

A gradient-free method is proposed for non-convex optimization problem

Represent the short summary for finding the corresponding scientific article (long)

In this paper, a new method we call Centered Initial Attack (CIA) is provided. It insures by construction the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process.


Question: Represent the scientific article to find a short TLDR

Question: Over the last decade, two competing control strategies have emerged for solving complex control tasks with high efficacy. Model-based control algorithms, such as model-predictive control (MPC) and trajectory optimization, peer into the gradients of underlying system dynamics in order to solve control tasks with high sample efficiency. However, like all gradient-based numerical optimization methods,model-based control methods are sensitive to intializations and are prone to becoming trapped in local minima. Deep reinforcement learning (DRL), on the other hand, can somewhat alleviate these issues by exploring the solution space through sampling — at the expense of computational cost. In this paper, we present a hybrid method that combines the best aspects of gradient-based methods and DRL. We base our algorithm on the deep deterministic policy gradients (DDPG) algorithm and propose a simple modification that uses true gradients from a differentiable physical simulator to increase the convergence rate of both the actor and the critic. We demonstrate our algorithm on seven 2D robot control tasks, with the most complex one being a differentiable half cheetah with hard contact constraints. Empirical show that our method boosts the performance of DDPGwithout sacrificing its robustness to local minima. In recent years, deep reinforcement learning (DRL) has emerged as a flexible and robust means of teaching simulated robots to complete complex tasks, from manipulation and locomotion (b), to navigating complex terrain . Compared with more direct optimization methods such as gradient descent or second-order optimization, DRL naturally incorporates exploration into its planning, allowing it to learn generalizable policies and robust state value estimations across simulated environments. Perhaps the most salient reason for DRL's surge in popularity is its ability to operate on black-box simulators where the underlying dynamics model is not available. DRL's model-free, Monte-Carlo-style methods have made it applicable to a wide range of physical (and non-physical) simulation environments, including those where a smooth, well-behaved dynamical model does not exist. This comes at two striking costs. First, such sampling procedures may be inefficient, requiring a large number of samples for adequate learning. Second, in order to be generally applicable to any model-free environment, underlying dynamical gradients are not used, even if they are available. In other words, valuable information that could greatly aid control tasks is not taken advantage of in these schemes. When an accurate model of robot dynamics is given, model-based methods such as model-predictive control (MPC) or trajectory optimization have historically been employed. These methods can solve tasks with higher sample efficiency than model-free DRL algorithms. Models provide access to ground-truth, analytical gradients of robot physics without the need for sample-based estimation. However, such methods don't incorporate exploration or learning into their procedures, and are especially prone to becoming trapped in poor local minima. While there has been a recent surge in fast and accurate differentiable simulators not previously available, most applications for control have relied on established local methods such as MPC (de), gradient descent , or trajectory optimization to solve control tasks. An ideal algorithm would exploit the efficiency of model-based methods while maintaining DRL's relative robustness to poor local minima. In this paper, we propose an actor-critic algorithm that leverages differentiable simulation and combines the benefits of model-based methods and DRL. We build our method upon standard actor-critic DRL algorithms and use true model gradients in order to improve the efficacy of learned critic models. Our main insights are twofold: First, gradients of critics play an important role in certain DRL algorithms, but optimization of these critics' gradients has not been explored by previous work. Second, the emergence of differentiable simulators enables computation of advantage estimation (AE) gradients with little additional computational overhead. Based on these observations, we present an algorithm that uses AE gradients in order to co-learn critic value and gradient estimation, demonstrably improving convergence of both actor and critic. In this paper, we contribute the following: 1) An efficient hybrid actor-critic method which builds upon deep deterministic policy gradients (DDPG, ), using gradient information in order to improve convergence in a simple way. 2) A principled mathematical framework for fitting critic gradients, providing a roadmap for applying our method to any deterministic policy gradient method, and 3) Demonstrations of our algorithm on seven control tasks, ranging from contact-free classic control problems to complex tasks with accurate, hard contact, such as the HalfCheetah, along with comparisons to both model-based control and DRL baselines. Model-based control methods peer into the underlying dynamics of a system in order to optimize control parameters. Such approaches have especially grown in popularity with the rise of differentiable simulators, whose application now spans rigid body simulation (de ;), soft body simulation , and fluid dynamics . Such simulators allow for backward propagating through an entire simulation, and even the simplest optimization algorithms (gradient descent) can be employed to a great effect. By treating robotic control tasks as a nonlinear numerical optimization problem, two more sophisticated strategies, trajectory optimization (; ; ; ;) and model-predictive control (MPC) , employ established numerical optimization techniques to efficiently solve complex, even high-dimensional control tasks. But, these methods are are prone to becoming trapped in poor local minima, especially in the presence of complex dynamics or terrain. Separate from model-based control methods, which assume the model of dynamics is known, modelbased learning methods attempt to learn a prior unknown model so as to directly optimize policies. In this paper, we do not focus on model-based learning, but rather the complementary problem, where we assume an accurate model of dynamics is given by a differentiable simulator and we use it to efficiently explore optimal control parameters. In contrast to the above local methods, stochastic global optimization methods, such as evolutionary algorithms (e.g., ) or simulated annealing, explore the solution space with a large number of samples in order to find global optima. Such methods, however, can be quite slow, due to the large number of samples they require. RL, while technically local, naturally incorporates sampling, exploration, and value function learning in a similar way that greatly mitigates optimization's propensity to becoming trapped in poor local minima. Our algorithm similarly incorporates exploration to mitigate the likelihood of poor local minima, while using gradient information to do so with high sample efficiency. Markov-Decision Problems (MDP) In this paper, we consider the rigid body simulation as a discrete-time Markov Decision Process (MDP) defined by (S, A, Γ, R), where S ⊆ R n is the state space, A ⊆ R m is the action space, Γ: S × A → S is the deterministic dynamic transition function, and R: S × A × S → R is the reward function. The optimal control policy π: S → A maximizes the γ-discounted sum of reward η(π) over a finite time horizon T, which is defined as: Here ρ 0 specifies the distribution of the initial state s 0 and r t = r(s t, a t, s t+1) represents the reward collected at time step t. We represent the policy function π as a neural network parameterized by θ and assume π θ is a deterministic policy that takes as input the current state s t and predicts the next action a t = π θ (s t). The action-value function Q π (s t, a t), which describes the expected return given the action a t and state s t at step t, is defined as in which a l ∼ π(s l) and s l ∼ Γ(s l−1, a l−1). The expectation can be removed in our case since both Γ and π are deterministic. This function is approximated by the critic network Q φ (s, a) parameterized with φ. Differentiable simulators In this paper, we focus on differentiable and deterministic rigid body simulators with fully observable states. At each time step t, the state s t fully characterizes the current status of the robot. The simulator takes current state s t and an action a t as input and returns the next state s t+1 by solving the governing physical equations. Additionally, we assume a differentiable simulator also returns a Jacobian matrix ∇ st,at s t+1. For all the differentiable rigid body simulators in this paper, the time and space complexities of computing the Jacobian matrix are the same as computing s t+1 in one forward step. Our hybrid algorithm takes inspiration from a subclass of modern actor-critic algorithms which seek to improve actor policies via critic gradients (such as DDPG). These algorithms iteratively improve the model of the critic, or estimated future return, using recorded data. Such algorithms implicitly assume that if the critic is well-fit to reality, then the gradients must also be well-fit. If we want to prioritize accurate critic gradients and better bootstrap our actor learning, it is best to ensure this directly. Specifically, we augment DDPG's critic gradient fitting step with a regularization term which prioritizes fitting accurate gradients in addition to accurate returns. Our choice of working in the domain of deterministic policy gradients is deliberate; as will be seen, this framework allows us to use our differentiable simulation engine to back propagate through the state-action pairs of our MDP, generating ground-truth reward gradients to fit. Algorithm 1 summarizes our method with our modifications to the original DDPG highlighted in color. Like actor-critic methods, our algorithm populates a replay buffer of state-action pairs from which to learn. At the same time, it records the dynamical Jacobian ∇ st,at s t+1 corresponding to those state-action pairs, without degrading its asymptotic time or space complexity. Like the state-action pairs, these Jacobians need only be computed during rollouts; they can be re-used during the embedded supervised training steps to efficiently improve the actor and critic networks. With additional Jacobian information about the dynamics at hand, we compute the gradients of the target network Q i. SinceQ i defines the target value that the critic Q φ attempts to fit to, ∇Q i provides a good estimation to the desired ∇Q as well. This motivates us to use it as a regularizer in the critic loss (Line 10 in Algorithm 1) with two additional weights w 1 and w 2 corresponding to the weights on the two partial derivatives. The definition of ∇ si,aiQi is provided by the following equation (For brevity, we use the network names to refer to their output values in the equation): which can be evaluated in constant time (Figure 1). It is important to note that this computation is only possible when a differentiable simulator is available due to the need for ∇ si,ai s i+1. Time and space complexity Our modifications require extra storage space for two Jacobian matrices of the size dim(s) × dim(s) and dim(s) × dim(a) associated with each s i in the replay buffer, which is tolerable as long as the dimensions of s and a are small constants. This is the case, for example, in rigid robot control tasks. In terms of the time cost, computing ∇Q i makes efficient use of the cached data. As a , its computation requires only a handful of matrix multiplications and back-propagation through Q and π. Simulate the agent using a t = π(s t) + N t at each time step t until termination 7: Update the critic parameter φ by minimizing the loss: Update the actor parameter θ using the following gradient: Update target networks:: end while We present a simple, analytical example with easily understandable dynamics in a low-dimensional space in order to illustrate the insights behind our algorithm. Consider a one dimensional kinematic mass point whose state s t and action a t are two scalars describing its position and velocity respectively. We define s 0 = −0.5 and a t ∈ [−1, 1]. The new state is computed by s t+1 = s t + a t ∆t where ∆t = 0.01 is a constant. The goal is to move the mass point to the origin s = 0 as quickly as possible. The simulation is terminated when |s t | < 0.01. The reward r(s, a, s) = (s 2 + 1) −1 − 1 encourages the mass point to converge to the origin in minimal number of steps. To see how the regularizer affects the training of the critic network, we compare from four sets of (w 1, w 2) parameters:, corresponding to the original DDPG algorithm;, which only regularizes ∇ siQi;, which only regularizes ∇ aiQi; and, which regularizes both ∇ siQi and ∇ aiQi. We plot the surfaces of intermediate critic network Q, their ground truth, and the return-timestep curves in Figure 2. In particular, the surfaces of ground truth Q for the optimal controller π are discounted returns directly measured from the forward simulation. Given these surfaces, the critic update steps in DDPG and the benefits of the regularizer can be explained visually: At each iteration, DDPG updates the critic network to fit the Q surface with points (s i, a i,Q i (s i, a i)), which are sampled from the replay buffer. Additionally, our hybrid method adds the regularizers to also fit the tangent directions of Q surface at the sample points. The tangent direction of the curve Q(·, a i) at s = s i is equivalent to ∇ aiQi. Since these tangent directions provide more information about the local shape of the Q surface, it is expected to boost the convergence of Q to its ground truth, which is reflected in Figure 2. We can also use this example to understand the effects of w 1 and w 2 on the convergence of Q: Figure 2 (c and d) demonstrate that w 2, or the weight for ∇ siQi, has a more evident effect on the convergence of Q. This can be partially explained by the fact that the ground truth Q surface has little variation in the a (action) direction, so knowing the partial derivative ∇ aiQi adds very little new information to the fitting problem. We test our hybrid algorithm on robot control tasks using a differentiable rigid body simulator, over three different types of settings related to the types of contact experienced: Contact-free simulation Many classical control examples are contact-free and useful for evaluating and understanding the performance of control algorithms (e.g. Pendulum, Acrobot, etc.). The equations of the motion of these problems can be compactly expressed in the form of the standard manipulator equations , allowing for the simple extraction of system gradients. Simulation with impulse-based collision responses Due to the significant changes in motion that occur during collisions, rigid body simulations with contacts possess much more complex dynamics than those which are contact-free. In this setting, we employ the Poisson collision model . During collision, an impulse is applied to the rigid body, instantaneously changing the velocity of the rigid body while conserving the momentum of the entire system. For simulating systems with such contacts, we follow (Popović et al., 2000). Simulation with constraint-based collision responses In order to simulate realistic motions for resting and collision contacts among rigid bodies, constraint-based methods formulate contacts as constrained systems, posing them as linear complementary problems (LCP) . Such simulation is accurate but much more complex than previous, simpler soft contact models. Inspired by the work of , we implemented a 2D LCP-based rigid body simulator and differentiate through contacts. Our hybrid control algorithm is implemented based on DDPG in OpenAI Baselines. Parameter space noise is applied to the actor network for exploration as proposed in. Slightly different from the mathematical definition, the critic network in this implementation takes normalized state and action as input, and outputs a normalized Q value in order to achieve task-irrelevant statistical stability. The mean value and standard deviation of state and Q value are empirically estimated from the replay buffer. Actions are scaled by a constant such that they fall in range [−1, 1]. Similarly, gradients in line 11 of Algorithm 1 are computed with normalized Q, states and actions. We present seven 2D control tasks implemented in our differentiable rigid body simulator. The tasks are of varying difficulty, including five classic control problems (CartPole, CartPoleSwingUp, Pendulum, MountainCar, and Acrobot), one problem with impulse-based collision response (RollingDie), and one complex control problem with LCP-based contact model (HalfCheetah). Classic control environments Our implementation of CartPole, CartPoleSwingUp, Pendulum, MountainCar, and Acrobot is based on the implementation of OpenAI Gym but with gradient information manually computed in the simulation. We ask readers to refer to for more details. In order to demonstrate applicability in the context of collisions, we designed a RollingDie task with impulse-based collision responses. The die is thrown from the air to the ground. Its initial state s 0 includes position in x and y, rotation angle, velocity in x and y, and angular velocity. The action a is the torque to the die, i.e. each step, the die can apply an internal torque so as to control its angular velocity (and thus its pose). The reward is defined based on the L2−norm of the distance between die center and the target position. Though for most of its trajectory the die is free falling and thus its position cannot be impacted by the torque, the angular velocity and torque do impact the angle with which the die impacts the ground and the ing bounce. This is a complex control task in which sparse, highly important moments (contact) can chaotically affect the trajectory of the die. Note that the die is allowed to bounce multiple times during the simulation, which increases the complexity of finding an optimized control. We designed a differentiable HalfCheetah example with LCP-based contacts. The HalfCheetah shares a similar structure as the one in MuJoCo but only has two links on each leg. The reward is defined as v x −0.1u T u, promoting forward velocity while penalizing highly actuated actions. Note that unlike MuJoCo's soft contact model, the hard contact model we implemented introduces an additional complexity and higher magnitude gradients, making it a more difficult control task. We compare our method with three baselines: DDPG, the RL algorithm that we base our method on, MPC with iterative Linear Quadratic Regulartor (iLQR) , a state-of-the-art model-based control algorithms employed in previous work on differentiable simulators , and gradient descent (GD), which employs a neural network actor and Adam to directly optimize a final return using the analytical gradients (as in and ). We repeat each algorithm 16 times on each task with different initializations. Initialization To ensure a fair comparison between all methods, we seed each algorithm with the same network models (if applicable). Since MPC does not require a neural network but directly solves for state and action variables, we initialize these variables by performing one rollout and collecting the state-action (s t, a t) pairs for initialization, using the same initial actor network. Metrics For methods using a neural network controller (ours, DDPG, and GD), we report the following metrics: return mean, which is the return from one episode generated by running the intermediate actor network every 800 timesteps, and return history, which is the average of the latest 100 return mean. For MPC, as it solves a t sequentially, return mean is stored whenever a t is returned. All returns are rescaled linearly so that the average return of a random controller is mapped to 0 and max(r) × T is mapped to 1. Figure 4 and Table 1 summarize the performance and averaged rewards of each method on the seven environments. Overall, our algorithm improves the sampling efficiency of DDPG and often discovers strategies with higher return. Compared with MPC and GD, our method suffers less from local minima, especially for more complex tasks such as HalfCheetah. For all environments, the experiment is terminated when return mean plateaus in GD, when MPC converges, or when the number of timesteps reaches pre-defined maximum number in DDPG and our algorithm. Unlike the other methods, MPC does not perform many sequential simulations, but rather performs one simulation optimized online. In order to perform as fair of a comparison as possible, we plot the predicted cumulative return (based on the previously simulated steps, current planned horizon, and remaining initialized action vector) as it evolves throughout the online planning procedure. As a , MPC may terminate without using the same number of function evaluations as RL algorithms or showing a sign of plateauing return-timestep curves. We do not compare MPC for the simpler control tasks (Pendulum, MountainCar, CartPole) that are trivial for the algorithm to solve. We also do not compare MPC for the Acrobot, for which the algorithm cannot be applied (discussed later). We discuss the performances of these methods in each environment below. Immediately obvious from our is the fact that DDPG and our algorithm are both competitive on all problems presented, regardless of problem difficulty. While MPC dominates on the simplest control tasks, it struggles on the more complicated tasks with hard contacts, and DRL approaches dominate. This underscores our thesis -that DRL's exploration properties make it better suited than model-based approaches for problems with a myriad of poor local minima. More naïve model-based approaches, such as GD, can succeed when they begin very close to a local minimum -as is the case with CartPole -but show slow or no improvement in dynamical environments with nontrivial control schemes. This is especially apparent in problems where the optimal strategy requires robots to make locally suboptimal motions in order to build up momentum to be used to escape local minima later. Examples include Pendulum, CartPoleSwingUp, and MountainCar, where the robot must learn to build up momentum through local oscillations before attempting to reach a goal. GD further fails on complex physical control tasks like HalfCheetah, where certain configurations, such as toppling, can be unrecoverable. Finally, we note that although MPC is able to tractably find a good solution for the RollingDie problem, the complex nonlinearities in the contact-heavy dynamics require long planning horizons (100 steps, chosen by running hyperparameter search) in order to find a good trajectory. Thus, although MPC eventually converges to a control sequence with very high reward, it requires abundant computation to converge. DRL-based control approaches are able to find success on all problems, and are especially competitive on those with contact. Compared with DDPG, our hybrid algorithm universally converges faster or to higher returns. The rolling die example presents a particularly interesting contrast. As the die is randomly initialized, it is more valuable to aim for higher return history rather than return mean due to the large variance in the initial state distribution. It can be seen from Figure 4 that our method managed to reach a higher average return history over 16 runs. Manually visualizing the controller from the best run in our method revealed that it discovered a novel two-bounce strategy for challenging initial poses (Figure 3), while most of the strategies in DDPG typically leveraged one bounce only. There are a few other reasons why our algorithm may be considered superior to MPC. First, our algorithm is applicable to a wider range of reward structures. While we had planned to demonstrate MPC on another classic control problem, namely the Acrobot, MPC is inapplicable to this robot's reward structure. The Acrobot's rewards penalize it with −1 point for every second it has not reached its target pose. MPC requires a differentiable reward, and this reward structure is not. Thus, our Hybrid DDPG algorithm applies to a wider range of problems than MPC. Second, closedloop network controllers are naturally more robust than MPC. Even as noise is added or initial conditions and tasks change, learned controllers can generalize. While MPC can recover from these scenarios, it requires expensive replanning. In these scenarios, MPC becomes especially unattractive to deploy on physical hardware, where power and computational resource constraints can render MPC inapplicable to realtime applications. Figure 3: Visualization of the twobounce strategy discovered by our algorithm. Solid red box: initial die. Dash cyan curve: trajectory of the die. Blue box: the target zone. Light red boxes: states of the die at collisions and about to enter the target. In this paper, we have presented an actor-critic algorithm that uses AE gradients to co-learn critic value and gradient estimation and improve convergence of both actor and critic. Our algorithm leverages differentiable simulation and combines the benefits of model-based methods and DRL. We designed seven 2D control tasks with three different contact scenarios and compared our method with several state-of-the-art baseline algorithms. We demonstrated our method boosts the performance of DDPG and is much less sensitive to local minima than model-based approaches. In the future, it would be interesting to see if our mathematical framework can be applied to improve the effectiveness of value functions used in other DRL algorithms. A APPENDIX In gradient-based optimization algorithms, the direction of gradients often plays a more crucial role than its length, especially when adaptive learning rates are applied. Here we experiment with the idea of penalizing the cosine angle between ∇Q φ and ∇Q instead of their L 2 distance in Algorithm 1. In particular, we replace line 10 with the following: In all examples, our hybrid algorithm achieves similar or better performance than pure DDPG. Figure 5 shows the experimental in our differentiable environments with the new loss function. For each environment, we started with the same parameters as in Figure 4 and decreased (w 1, w 2) ∈ {1, 0.1, 0.01} only if necessary. We ended up using the same parameters as before in MountainCar and Acrobot and switched weights in other environments. Comparing Figure 4 with Figure 5, we see the cosine angle regularizer is more effective in some examples (particularly MountainCar and Acrobot) than the original L2 norm. The L 2 regularizer outperforms the cosine angle for CartPoleSwingUp. The two regularizers perform similarly on the remaining examples. Apart from DDPG, we have also implemented our method with the original L 2 -norm loss in the Soft Actor-Critic (SAC) (a) algorithm. Figure 6 reports our experimental in five examples, among which our method improves the performance of the original SAC in three of them (CartPoleSwingUp, Pendulum, and MountainCar) and performs similarly in the other two. These have demonstrated that our proposed modification can be generalized to other actorcritic methods other than DDPG. In all examples, our hybrid algorithm performs similar to or better than pure SAC.

represent the tldr for finding the corresponding scientific article

We propose a novel method that leverages the gradients from differentiable simulators to improve the performance of RL for robotics control

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.