Question: Represent the scientific article to find a short TLDR

Question: Dynamical system models (including RNNs) often lack the ability to adapt the sequence generation or prediction to a given context, limiting their real-world application. In this paper we show that hierarchical multi-task dynamical systems (MTDSs) provide direct user control over sequence generation, via use of a latent code z that specifies the customization to the individual data sequence. This enables style transfer, interpolation and morphing within generated sequences. We show the MTDS can improve predictions via latent code interpolation, and avoid the long-term performance degradation of standard RNN approaches. Time series data often arise as a related'family' of sequences, where certain characteristic differences exist between the sequences in a dataset. Examples include the style of handwritten text , the response of a patient to an anaesthetic , or the style of locomotion in motion capture (mocap) data . In this paper, we will consider how such variation may be modelled, and effectively controlled by an end user. Such related data is often pooled to train a single dynamical system, despite the internal variation. For a simple model, such as a linear dynamical system (LDS), this will in learning only an average effect. In contrast, a recurrent neural network (RNN) may model this variation, but in an implicit and opaque manner. Such a'black-box' approach prohibits end-user control, and may suffer from mode drift, such as in , where a generated mocap sequence performs an unprompted transition from walking to drinking. Some of these problems may be alleviated by appending'context labels' to the inputs (see e.g. , §10.2.4) which describe the required customization. However, such labels are often unavailable, and the approach may fail to model the variation adequately even when they are. To move beyond these approaches, we consider latent variable models, where a latent variable z characterizes each sequence. This may be seen as a form of multi-task learning (MTL, see), from which we derive the name multi-task dynamical system (MTDS), with each sequence treated as a task. A straightforward approach is to append the latent z to the inputs of the model, similarly to the'context label' approach, thereby providing customization of the various bias (or offset) parameters of the model. A number of examples of this have been proposed recently, e.g. in and Miladinović et al.. Nevertheless, this'bias customization' has limited expressiveness and is often unsuitable for customizing simple models. In this paper we investigate a more powerful form of customization which modulates all the system and emission parameters. In this approach, the parameters of each task are constrained to lie on a learned low dimensional manifold, indexed by the latent z. Our experiments show that this approach in improved performance and/or greater data efficiency than existing approaches, as well as greater robustness to unfamiliar test inputs. Further, varying z can generate a continuum of models, allowing interpolation between sequence predictions (see Figure 1b for an example), and potentially morphing of sequence characteristics over time. Contributions In this paper we propose the MTDS, which goes beyond existing work by allowing full adaptation of all parameters of general dynamical systems via use of a learned nonlinear manifold. We show how the approach may be applied to various popular models, and provide general purpose...... learning and inference algorithms. Our experimental studies use synthetic data (sum of two damped harmonic oscillators) and real-world human locomotion mocap data. We illuminate various properties of the MTDS formulation in our experiments, such as data efficiency, user control, and robustness to dataset shift, and show how these go beyond existing approaches to time series modelling. We finally utilize the increased user control in the context of mocap data to demonstrate style morphing. To this end, we introduce the model in Section 2, giving examples and discussing the particular challenges in learning and inference. We discuss the relation to existing work in Section 3. Experimental setup and are given in Section 4 with a in Section 5. Consider a collection of input-output sequences D = {Y Ti}, i = 1,..., N, where T i denotes the length of sequence i. Each sequence i is described by a different dynamical system, whose parameter θ (i) depends on the hierarchical latent variable z (i) ∈ Z: for t = 1,..., T i. The state variables X (i) = {x Ti}, x t ∈ X follow the latent dynamics starting from x 0:= 0 (other choices of initial state are possible). See Figure 1a for a graphical model. In this paper we assume Z = R k which the vector-valued function h φ (·) transforms to conformable model parameters θ ∈ R d, d k. Note that h φ may keep some dimensions of θ constant with respect to z. We call this a Multi-Task Dynamical System, going beyond the usage in. In order to make the framework more concrete we will describe two general choices of the base model. In what follows we will write each parameter with a subscript z to denote dependence on z (e.g. A z := A(z)) to reduce notational clutter. The choice of p(z) and h φ will depend on the application, but a fairly general choice is a deep latent Gaussian model . See section A.1.1 in the supplementary material for further discussion. For a given z, a multi-task linear dynamical system (MTLDS) can be described by: w t ∼ N (0, R z), t ∼ N (0, S z), with θ z = {A z, B z, b z, C z, D z, d z, R z, S z} = h φ (z). The parameterization of θ z must satisfy the constraints of positive definite R z and S z and stable A z (i.e. A z 2 ≤ 1) for all z, hence projection methods such as in are not applicable. We choose an alternative formulation of the LDS, replacing the latent dynamics in eq. by: where Σ z is a diagonal matrix and Q z orthogonal with no loss of generality (proof in supp. mat.). Since Σ z Q z ≤ Σ z Q z = Σ z, stability can be enforced e.g. by Σ = diag{tanh (υ)} for some vector υ. For more details see section A.1.2 in the supplementary material. Due to the nonlinearity of an RNN, enforcing stability of A z is not strictly required (see e.g. , §4.4), although bounding the spectral radius may be useful for learning (e.g.). The dynamics of a multi-task RNN (MT-RNN) are described by: Combined with the emission model of eq., we have If long-term dependencies are important we may consider an orthogonal transition matrix (parameterized as for the MTLDS) to create a multi-task version of the Orthogonal RNN . The parameters φ of an MTDS can be learned from a dataset The first term in the integrand,. For clarity of exposition we only consider models with deterministic state, which extends to the MTRNN and MTLDS above (in the case w t = 0 for all t). This also avoids the interaction effect with the choice of approximate marginalization over X. Equation also cannot be computed in closed form in general, and so we resort to approximate learning. A natural choice is via the ELBO (see an alternative MCO approach for unsupervised tasks in Section A.1.3, supp. mat.). We write the ELBO of eq. as: where D KL is the Kullback-Leibler divergence and q λ (z|Y, U) an approximate posterior for z. We can now optimize a lower bound of the marginal likelihood via arg max φ,λ where low variance unbiased gradients of eq. are available via reparameterization , where µ λ, s λ are inference networks (e.g. Fabius & van). It can be difficult to learn a sensible latent representation if the base model is a powerful RNN. When each output can be identified unambiguously via the inputs preceding it, a larger ELBO can be obtained by the RNN learning the relationship without using the latent variable (see e.g.). A useful heuristic for avoiding such optima is KL annealing (e.g.). In our experiments we perform an initial optimization without the KL penalty (second term in eq. 9), initializing s λ (Y, U) to a small constant value. For an unseen test sequence {Y, U}, the posterior predictive distribution is p(y t+1:T | y 1:t, u 1:T) = Z p(y t+1:T | u 1:T, z) p(z | y 1:t, u 1:t) dz, usually estimated via Monte Carlo. The key quantity is the posterior over z, which may be approximated by the inference networks µ λ, s λ. However, for novel test sequences, the inference networks may perform poorly and standard approximate inference techniques may be preferred. For further discussion and a description of our inference approach, see sections A.1.4, A.1.5 in the supp. mat. We note that z may not require inference, for instance by using the posterior of a sequence in the the training set. This may be useful for artistic control, style transfer, embedding domain knowledge or overriding misleading observations. We also note that the latent code can be varied during the state rollout to simulate a task which varies over time. A number of dynamical models following the'bias customization' approach have been proposed recently. Miladinović et al. and propose models where the biases of an LSTM cell depend on a (hierarchical) latent variable. propose a dynamical system where the latent dynamics are concatenated with a time-constant latent variable. In contrast, our MTDS model performs full parameter customization, and this on both the dynamics and emission distributions. A number of other proposals may be considered specialized applications of the MTDS. use a small deterministic nonlinear dynamical system for the base model whose parameters depend on z using a nonlinear factor analysis structure. use a small RNN as the base model, where the transition matrix depends on z via multilinear decomposition. use a small stochastic nonlinear dynamical system with h φ a set of parameter vectors chosen discretely (or in convex combination) via z. Controlling and customizing sequence prediction has received much attention in the case of video data. As in the MTDS, these approaches learn features that are constant (or slowly varying) within a subsequence. and propose methods for disentangling time-varying and static features, but do not provide a useful density over the latter, nor an obvious way to customize the underlying dynamics. use a GAN architecture where z factorizes into content and motion components. force a parts-based decomposition of the scene before inferring the latent content z. However, as before, the dynamic evolution cannot be easily customized with these methods. Hierarchical approaches for dynamical systems with time-varying parameters are proposed in (corresponding to non-stationary assumptions) and in (for the purposes of local LDS approximation). These models, like the MTDS can adapt all the parameters, but are linear and correspond to single task problems. predict the parameters of simple time-varying LDS models directly via an RNN. While this is a multi-task problem, it is assumed that all necessary variation can be inferred from the inputs U. Multi-task GPs are commonly used for sequence prediction. Examples include those in; Titsias & Lázaro-; Álvarez et al.;. MTGPs however can only be linear combinations of (a small number of) latent functions, further, predictions depend critically upon often unknown mean functions, and inputs are not easily integrated. Note that an MTDS with no inputs, an LDS base model, a linear-Gaussian prior over the emission parameters We investigate the performance of the MTDS on two datasets. The first experiment investigates the performance of the MTDS on synthetic data generated by linear superposition of damped harmonic oscillation (DHO). The second experiment considers real-world mocap data for human locomotion. Data The generative model for J oscillators with constant amplitudes γ and variable frequency and decay factors,.., 80 for tasks i = 1, 2,..., N. The emission noise is distributed iid as Model We model the DHO data using an MTLDS with deterministic state X = R 4 and a k = 4 latent variable z. All LDS parameters were adapted via the latent z except D:= 0 and the emission variance s 2, which was learned. For optimization, we use the MCO algorithm of section A.1.3. This can obtain a tighter bound than the ELBO, and is useful to investigate convergence to the true model over increasing N. We contrast this with a bias customization approach (e.g. Miladinović et al., 2019), implemented similarly, but such that only the parameter b (eq. 6) depends on z. We also train a Pooled LDS, which is the standard approach, using the same parameters for all tasks, and a single-task (STL) LDS which is learned from scratch using Bayesian inference over all parameters for each task. The Pooled-LDS was initialized using spectral methods (see Van Evaluation We assess how quickly and effectively the models can adapt to novel test sequences with a training set size of N = 2 1, 2 2, . . ., 2 7 . (The STL approach effectively uses N = 0.) The test set comprises 20 additional sequences drawn from the generating distribution. For an initial subsequence y 1:t, we estimate the predictive posterior p(y t+1:T |y 1:t) for various t and assess the predictions via root mean squared error (RMSE) and negative log likelihood (NLL). For MTL we use the Monte Carlo inference method described in supp. mat. A.1.5 and for STL we use Hamiltonian Monte Carlo . Each experiment is repeated 10 times to estimate sampling variance. The , shown in Table 1 and supp. mat. section A.2.2, show substantial advantage of using the MTLDS ('MT Full') over single-task or pooled approaches. The MTLDS consistently outperforms the Pooled-LDS for all training sizes N ≥ 4. Merely performing bias customization ('MT Bias') is insufficient to perform much better than a pooled approach. An example of MTLDS test time prediction is shown in Figure 2, with Figures 2c and 2d demonstrating effective generalization from the N = 4 training examples (Figure 2a). Even after 40 observations, the STL approach (which is capable of fitting each sequence exactly) does not significantly outperform the N = 4 MTLDS. Furthermore, the runtime was approx. 1000 times longer since STL inference is higher dimensional and poorly conditioned, and requires a more expensive algorithm. Note that with a larger training set size of N = 128, the MLTDS approaches the likelihood of the true model (Figure 7, supp. mat.). Data The dataset consists of 31 sequences from (ca. 2000 frames average at 30fps) in 8 styles: angry, childlike, depressed, neutral, old, proud, sexy, strutting. In this case the family of possible sequences corresponds to differing walking styles. Each observation represents a 21-joint skeleton in a Lagrangian frame, y t ∈ R 64 where the root movement is represented by a smoothed component and its remainder. we represent joints by their spatial position rather than their rotation. We also provide inputs that an animator may wish to control: the root trajectory over the next second, the gait cycle and a boolean value determining whether the skeleton turns around the inside or outside of a corner. See section A.3.1 in the supplementary materials for more details. Model We use a recurrent 2-layer base model where the first hidden layer is a 1024 unit GRU and the second hidden layer is a 128 unit standard RNN, follwed by a linear decoding layer. The first-layer GRU does not vary with z, i.e. it learns a shared representation of the input sequence across all i. Explicitly, omitting index i, the model for a given z is: for t = 1,..., T. The parameters are θ = {ψ 1, ψ 2, H, C, d} where ψ 1 and H are constant wrt. z. The matrix H ∈ R ×1024 (< 1024) induces a bottleneck between layers, forcing z to explain more of the variance. For our experiments, a small can be used (we use = 24). The first layer GRU uses 1024 units since it was observed experimentally to produce smoother animations than smaller networks. The second layer does not use a gated architecture, as gates appear to learn style inference more easily, and in less use of z. For learning, each sequence was broken into overlapping segments of length 64 (approx. two second intervals), which allows z to vary across a sequence. We learn the model using an open-loop objective, i.e. the y t are not appended to the inputs. This forces the model to recover from its mistakes as in , although unlike these approaches, we do not append predictions to the inputs either. Our rationale is that the state captures the same information as the predictions, and while previous approaches required observations y 1:τ as inputs to seed the state, we can use the latent z. The model was optimized using the variational procedure in section 2.2, where a slower learning rate (by a factor of 10-50) for the first layer parameters (i.e. ψ 1, H) usually ed in a more descriptive z. We also found that standard variational inference for each z (i) worked better in general than using amortized inference. For comparison, we implement a bias customization model ('MTBias') via a deterministic state version of Miladinović et al., 2019, which follows eqs.- but only the RNN bias in eq. is a function of z. We also implement a 1-layer and 2-layer GRU without the multi-task apparatus, which serves both as an ablation test and a competitor model on the new dataset. Style inference is performed with the same network given an initial seed sequence y 1:τ. We train these in closed-loop (i.e. traditional next step 'teacher forcing' criterion) and open-loop settings. For baselines, we use constant predictions of (i) the training set mean and (ii) the last observed frame of the seed sequence ('zero-velocity' prediction). We test the data efficiency of the MTDS by training the models on subsets of the original dataset. Besides the models described above, 8'single-task' versions of the GRU models are trained which only see data for a single style. We use six training sets of approximate size 2 8, 2 9, 2 10, 2 11, 2 12, 2 13 frames per style, where sampling is stratified carefully across all styles, and major variations thereof. For all experiments, the model fit (MSE) is calculated from the same 32 held out sequences (each of length 64). The are shown in Figure 3a. As expected, the MTDS, MTBias and Pooled models obtain'multi-task' gains over STL approaches for small datasets. However, the MTDS demonstrates much greater data efficiency, achieving close to the minimum error with only 7% of the dataset. The MTBias model requires more than twice this amount to obtain the same performance, and the Pooled model requires more than four times this amount. More details, as with all mocap experiments, can be found in supp. mat. section A.2.3. We investigate how well the MTDS can generalize to novel sequence styles via use of a leave-one-out (LOO) setup, similar to transfer learning. For each test style, a model is trained on the other 7 styles in the training set, and hence encounters novel sequence characteristics at test time. We average the test error over the LOO folds as well as 32 different starting locations on each test sequence. The are given in Figure 3b. We see that while the competitor (pooled) models perform well initially, they usually degrade quickly (worse for closed-loop models). In contrast, the multi-task models finds a better customization which evidences no obvious worsening over the predictive interval. Unlike pooled-RNNs, the MTDS and MTBias models can firstly perform correct inference of their customization, and secondly can'remember' it over long intervals. We note that all models struggle to customize the arms effectively, since their test motions are often entirely novel. Customization to the legs and trunk is easier since less extrapolation is required (see animation videos linked in section A.4.1). We investigate the control available in the latent z by performing style transfer. For various inputs U (s1) from each source style s 1, we generate predictions from the model using target style s 2, encoded by z (s2). We use a classifier with multinomial outputs, trained on the 8 styles of the training set, to test whether the target style s 2 can be recognized from the data generated by the MTDS. Figure 3c gives the classifier'probability' for each target style s 2, averaged over all the inputs {U (s1): s 1 = s 2 }. Successful style transfer should in a the classifier assigning a high probability to the target style. These suggest that the prediction style can be well controlled by z (s2) in the case of the full MTDS, but the MTBias demonstrates reduced control for some (source, target) pairs. See the videos linked in section A.4.1 for examples, and sec. A.2.3 for more details. Qualitative investigation Qualitatively, the MTDS appears to learn a sensible manifold of walking styles, which we assess through visualization of the latent space. A k = 2 latent embedding can be seen in Figure 4 where the z (i) for each training segment i is coloured by the true style label. Some example motions are plotted in the figure. The MTDS embedding broadly respects the style label, but learns a more nuanced representation, splitting some labels into multiple clusters and coalescing others. These appear broadly valid, e.g. the'proud' style contains both marching and arm-waving, with the latter similar to an arm-waving motion in the'childlike' style. This highlights the limitation of relying on task labels. Visualizations such as Fig. 1b indicate that smooth style interpolation is available via interpolation in latent space. We take advantage of this in the animations (linked from sec. A.4.1) by morphing styles dynamically. In this work we have shown how to extend dynamical systems with a general-purpose hierarchical structure for multi-task learning. Our MTDS framework performs customization at the level of all parameters, not just the biases, and adapts all parameters for general classes of dynamical systems. We have seen that the latent code can learn a fine-grained embedding of sequence variation and can be used to modulate predictions. Clearly good predictive performance for sequences requires task inference, whether implicit or explicit. There are three advantages of making this inference explicit. Firstly, it enhances control over predictions. This might be used by animators to control the style of predictions for mocap models, or to express domain knowledge, such as ensuring certain sequences evolve similarly. Secondly, it can improve generalization from small datasets since task interpolation is available out-of-the-box. Thirdly, it can be more robust against changes in distribution at test time than a pooled model: is a unit Gaussian p(z) = N (0, I). This choice allows simple sampling schemes, and straight-forward posterior approximations. It is also a useful choice for interpolation, since it allows continuous deformation of its outputs. An alternative choice might be a uniform distribution over a compact set, however posterior approximation is more challenging, see Svénsen for one approach. Sensible default choices for h φ include affine operators and multilayer perceptrons (MLPs). However, when the parameter space R d is large, it may be infeasible to predict d outputs from an MLP. Consider an RNN with 100k parameters. If an MLP has m L−1 = 300 units in the final hidden layer, the expansion to the RNN parameters in the final layer will require 30×10 6 parameters alone. A practical approach is to use a low rank matrix for this transformation, equivalent to adding an extra linear layer of size m L where we must have m L m L−1 to reduce the parameterization sufficiently. Since we will typically need m L to be O, we are restricting the parameter manifold of θ to lie in a low dimensional subspace. Since MLP approaches with a large base model will then usually have a restricted final layer, are there any advantages over a simple linear-Gaussian model for the prior p(z) and h φ? There may indeed be many situations where this simpler model is reasonable. However, we note some advantages of the MLP approach: 1. The MLP parameterization can shift the density in parameter space to more appropriate regions via nonlinear transformation. 2. A linear space of recurrent model parameters can yield highly non-linear changes even to simple dynamical systems (see e.g. the bifurcations in §8 of). We speculate it might be advantageous to curve the manifold to avoid such phenomena. 3. More expressive choices may help utilization of the latent space (e.g.). This may in fact motivate moving beyond a simple MLP for the h φ. The matrices A, B, R, S of the MTLDS can benefit from specific parameterizations, which we will discuss in turn. Degeneracy of LDS. It will be useful to begin with the well-known over-parameterization of linear dynamical systems. The hidden dynamics of a LDS can be transformed by any invertible matrix G while retaining the same distribution over the emissions Y. This follows essentially because the basis used to represent X is arbitrary. The distribution over Y is unchanged under the following parameter transformations: Parameterization of A. The stability constraint, is equivalent to ensuring that the singular values of A lie within the unit hypercube (since singular values are non-negative). Let A = U ΣV T be the singular value decomposition (SVD) of A. Now we have from the previous that if an LDS has latent dynamics with transition parameter A, we may replace the dynamics under the similarity transform G −1 AG. Choose G = U, i.e. the left singular values of A, and hence A = ΣV T U =: ΣQ for some orthogonal matrix Q. This follows from the closure of the orthogonal group under multiplication, which is easily verified. Note that in choosing this transformation, no additional constraints are placed on the other parameters in the LDS. Orthogonal matrices can be parameterized in a number of ways (see e.g.). A straight-forward choice is the Cayley transform.: "if Q is an orthogonal matrix that does not have the eigenvalue -1, then it may be written in Cayley's form: where S is skew-symmetric". In order to permit negative eigenvalues, we can pre-multiply by a diagonal matrix E with elements in {+1, −1}. Since we then have A = ΣEQ, E can be absorbed into Σ, and so the stability constraint can be satisfied with the parameterization A = ΣQ where Σ is a diagonal matrix with elements in [−1, +1] and Q is a Cayley-transform of a skew-symmetric matrix. This follows from the overparameterization of the LDS, and we emphasise that the system equations and are not equivalent, but any LDS distribution over Y can be written with latent dynamics of the form. Choose G = κ −1 I in eq.. It may be observed that the scale κ of the latent system can be chosen arbitrarily without affecting A. We wish to avoid such degeneracies in a hierachical model, since we may otherwise waste statistical strength and computation on learning equivalent representations. We can remove this by fixing the scale of B. An indirect but straightforward approach is to upper bound the magnitude of each element of B. ForB predicted by h φ (z) we might choose the transformation B = tanh(B) where tanh acts element-wise. If a sparse B is desired, one can use an over-parameterization of two matricesB 1,B 2, and choose B = σ(B 1) • tanh(B 2), where • is element-wise multiplication, and σ a logistic sigmoid. The former parameterization is unlikely to find a sparse representation since the gradient of tanh is greatest at 0. Parameterization of R, S. The covariance matrices R, S must be in the positive definite cone. Where a diagonal covariance will suffice, any parameterization for enforcing positivity can be used, such as exponentiation, squaring or softplus. A number of parameterizations are available for full covariance matrices (see). A simple choice is to decompose the matrix, say R = LL T, where L is a lower triangular Cholseky factor. As before, it is useful to enforce uniqueness, which can be done by ensuring the diagonal is positive. We provide an alternative learning algorithm to the VB approach in section 2.2 which obtains a tighter lower bound. This was important for the DHO experiments in order to monitor convergence to the true model. The below is perhaps a novel approach for learning in unsupervised cases (i.e. where U = ∅), but cannot be performed efficiently for supervised problems without modification. Monte Carlo Objectives construct a lower bound for marginal likelihoods via a transformation of an appropriate Monte Carlo estimator. Specifically we consider the logarithmic transformation of: m = 1,..., M; an importance sampling estimator for p(Y). Using Jensen's inequality, we show that the following is a lower bound on the log marginal likelihood: where p(z 1:M):= p(z 1)...p(z M). The tightness of the bound can be increased by increasing the number of samples M . Assuming p(z) has been re-parameterized to be parameter-free, we can easily calculate the gradient (if not, see). By exchanging integration and differentiation, we can calculate the gradient as: Note that eq. is an importance sampled version of the Fisher identity. We might expect this estimator to suffer from high variance, since the prior is a poor proposal for the posterior. However, the prior should not be a poor proposal for the aggregate posterior, i.e. ). In fact, importance sampling from the prior may serve as a useful bias in this case, attracting the posterior distributions which have a large D KL (p(z | Y (i) )||p(z)) towards the prior. Our observation is that sampling from the prior can be amortized over each sequence Y (i), i = 1,..., N. Specifically, for each particle z m, the dynamics, can be run forward once to calculatê Y m, from which the likelihood Y (i), for all tasks i = 1,..., N can be calculated inexpensively. The amortized cost of taking M samples (e.g. M ∈ O(10 3)) now becomes M/N, which may be relatively small. We can also take advantage of low-discrepancy random variates such as Sobol sequences to reduce variance. We propose that each sequence i resamples a small number M rsmp ≤ 5 of particles from the importance weights for each i to reduce the cost of backpropagation (a similar resampling scheme is suggested in). See Algorithm 1. In the supervised case (i.e. where each observation Y (i) has a different input U (i) ), running the dynamics forward from a particle z m can no longer be amortized over all {Y (i) } since the prediction We can therefore only amortize the parameter generation θ = h φ (z), which is often less expensive than running the dynamics forward. For this reason Algorithm 1 is primarily restricted to unsupervised problems. A hybrid approach would essentially in the importance weighted autoencoder (IWAE) of. Inference at test time can be performed by any number of variational or Monte Carlo approaches. As in the main text, our focus here is on deterministic state dynamical systems. For stochastic state models, additional reasoning similar to Miladinović et al. will be required. A gold standard of inference over z may be the No U-Turn Sampler (NUTS) of (a form of Hamiltonian Monte Carlo), provided k is not too large and efficiency is not a concern. However, given the sequential nature of the model, it is natural to consider exploiting the posterior at time t for calculating the posterior at time t + 1. Bayes' rule suggests an update of the end end Optimize(optimizer, φ, g); end end following form: following the conditional independence assumptions of the MTDS. This update (in principle) incorporates the information learned at time t in an optimal way, and further suggests a constant time update wrt t. However, evaluation of p(y t+1 | u 1:t+1, h φ (z)) usually scales linearly with t, since the state x t+1 must be calculated recursively from x 0 given z and u 1:t+1. Nevertheless, sequential incorporation of previous information will perform a kind of annealing which reduces the difficulty, and hopefully the runtime of inference at each stage. We first provide some of the difficulties of such an approach, looking first at Monte Carlo (MC) methods. Naïve application of Sequential Monte Carlo (SMC) will in severe particle depletion over time. To see this, let the posterior after time t be p(z | y 1:t, u 1:t) = 1 M M m=1 w m δ(z− z m). Then the updated posterior at time t + 1 will be:, simply a re-weighting of existing particles. Over time, the number of particles with significant weights w m will substantially reduce. But since the model is static with respect to z (see), there is no dynamic process to'jitter' the {z m} as in a typical particle filter, and hence a resampling step cannot improve diversity. discusses two related solutions: firstly using'rejuvenation steps' (cf.) which applies a Markov transition kernel to each particle. The downside to this approach is the requirement to run until convergence; and the diagnosis thereof, which can in substantial extra computation. One might instead sample from a fixed proposal distribution (accepting a move with the usual Metropolis-Hastings probability) for which convergence is more easily monitored. A Sequential Monte Carlo sampler approach may be preferred, which permits local moves, and can reduce sample impoverishment via resampling (similar to SMC). However, the approach requires careful choices of both forward and backward Markov kernels which substantially reduces its ease of use. A well-known variational approach to problems with the structure of eq. is assumed density filtering (ADF, see e.g.). For each t, ADF performs the Bayesian update and the projects the posterior into a parametric family Q. The projection is done with respect to the reverse KL Divergence, i.e. q t+1 = arg min q∈Q D KL p(z | y 1:t+1, u 1:t+1) || q. Intuitively, the projection finds an'outer approximation' of the true posterior, avoiding the'mode seeking' behaviour of the forward KL, which is particularly problematic if it attaches to the wrong mode. Clearly the performance of ADF depends crucially on the choice of Q. Unfortunately, where Q is expressive enough to capture a good approximation, the optimization problem will usually be challenging, and must resort to stochastic gradient approaches, ing in an expensive inner loop. Furthermore, when the changes from q t to q t+1 are relatively small, the gradient signal will be weak, ing perhaps in misdiagnosed convergence and hence accumulation of error over increasing t. A recent suggestion of is to improve efficiency via re-use of previous (stale) gradient evaluations. Standard variance reduction techniques may also be considered to improve convergence in the inner loop. In our experiments, we found sampling approaches faster and more reliable for each update, as well as providing diagnostic information, and so we eschew variational approaches. (Our experiments used a fairly small k (≤ 10); variational approaches may be preferred in higher dimensional problems.) Specifically we use iterated importance sampling (IS) to update the posterior at each t. The key quantity for IS is the proposal distribution q prop: we need a proposal that is well-matched to the target distribution. Our observation is that the natural annealing properties of the filtering distributions (eq. 23) allow a slow and reliable adaptation of q prop. In order to capture complex multimodal posteriors, we parameterize q prop by a mixture of Gaussians (MoG). For each t, the proposal distribution is improved over N AIS iterations using adaptive importance sampling (AdaIS), described for mixture models in Cappé et al.. We briefly review the methodology for a target distribution p *. Let the AdaIS procedure at the nth iteration use the proposal: α j ∈ R + s.t. For our experiments, this approach worked robustly and efficiently, and appears superior to the alternatives discussed. Unlike SMC, we obtain a q prop which is a good parameteric approximation of the true posterior. We therefore avoid the sample impoverishment problem discussed above (eq. 25). Due to the small number of iterations of AdaIS required (usually ≤ 5 for our problems), it is substantially faster than MCMC moves, and since stochastic gradients are avoided, convergence is much faster than variational approaches. The scheme benefits from the observed fast initial convergence rates of the EM algorithm (see e.g.), particularly since early stopping can be used for the initial iterates. In practice, one may not wish to calculate a posterior at every t, but instead intervals of length τ. In our DHO experiments (k = 4) we use τ = 5, and usually have ESS > 0.6M after n = 4 inner iterations, with total computation per q t requiring 250-300ms on a laptop. We observe in our experiments that posteriors are often multimodal for t ≤ 20 and sometimes beyond, motivating the MoG parameterization. In these experiments, the MoG appears to capture the salient characteristics of the target distribution well. Note as in section A.1.3, Sobol or other low-discrepancy sequences may be used to reduce sampling variance from q prop. Under review as a conference paper at ICLR 2020 for t = 1,..., T, z ∈ R 4, x t ∈ R 4, suppressing task index i for clarity. Define x 0:= 0, and for all tasks u = [1, 0, 0, 0, . . .]. A is parameterized as discussed in section A.1.2 using a product of a diagonal and orthogonal matrix ΣQ. The diagonal of Σ is constrained to lie in [−1, +1] using the tanh function, and Q is parameterized by the Cayley transform of a skew symmetric matrix S. Using an upper triangular matrix Γ, we have S = Γ − Γ T, and Q = (I − S)(I + S) −1. We parameterize B via the product of logistic sigmoid and tanh functions as in section A.1.2 in order to learn a sparse parameterization. C is unconstrained, and the parameter s is optimized as a constant wrt. z. The STLDS is parameterized in the same way. The prior p(z) is a unit Gaussian distribution, and h φ is a 2 hidden-layer neural network. We use a fixed feature extractor T in the first layer in order to help encode a rectangular support within a spherically symmetric distribution. The second layer is a fully-connected 300 unit layer with sigmoid activations. Learning. The output of an MTLDS is very sensitive to the parameter A, and care must be taken to avoid divergence during optimization. The diagonal-orthogonal parameterization greatly helped to stabilize the optimization over a more naïve orthogonal-diagonal-orthogonal SVD parameterization. We also reduced the learning rate by a factor of 10 for A. It proved useful to artificially elevate the estimate of s during training using a prior log s ∼ N (−1.5, 0.05) (derived from preliminary experiments) since the MTDS can otherwise overfit small datasets (see also discussion in §3, Svénsen, 1998), with associated instability in optimization. The learning rate schedule is given in Table 2, for which the prior over log s ∼ N (m, 0.05) was annealed from m = −1.0 to m = −1.5. The "momentum" parameter β 1 (c.f.) is also reduced at the end of optimization. The latter was motivated by oscillation and moderate deviations observed near optima, apparently caused (upon investigation) by strong curvature of the loss surface. Inference. The latent z are inferred online using the adaptive IS scheme of section A.1.5. We also perform inference over log s since it is held artificially high for optimization, and its true optimal value is not known. An informative prior close to the learned value log s ∼ N −2.0, 0.1 2, was Epoch η β 1 log s mean M 1 8e-4 0.9 -1.0 1 000 200 8e-4 0.9 -1.3 1 000 600 4e-4 0.9 -1.5 2 000 1000 2e-4 0.8 -1.5 4 000 nevertheless used since the posterior was sometimes approximately singular, causing high condition numbers in the estimated covariance matrix of the proposal. The hyperparameters are given in Table 3. These parameters did not require tuning as for optimization, but were sensible defaults. These also seem to work well without tuning for other experiments such as the Mocap data. Each posterior for a given time t took on average approx. 0.3 seconds. We used the No U-Turn Sampler for the STL experiments due to poor conditioning and the higher complexity and dimensionality of the posterior (19 dimensions). Tuning is performed using ideas from Hoffman & Gelman (2014, Algorithm 4, 5), and the mass matrix is estimated from the warmup phase. 2 The warmup stage lasted 1000 samples and the subsequent 600 samples were used for inference. Each sampler was initialized from a MAP value, obtained via optimization with 10 random restarts. For both MAP optimization and sampling, we found it essential to enforce a low standard deviation (we used log s = −2 and log s ∼ N −2, 0.2 2 respectively) similarly to the MTL experiments. The autocorrelation-based effective sample size (, ch. 11.5) typically exceeds 100 for each parameter. Each posterior for a given time t took on average approx. 300 seconds. Note that as discussed in section A.1.4, unlike our AdaIS procedure, we cannot make much re-use of previous computation here. The average (over the 10 repetitions) are given in Table 4, which extends Table 1 in the main text with the NLL . The distribution of these can be seen in the violin plots of Figure 8. The RMSE of the MTLDS are all significantly better than both the pooled and single-task models according to a Welch's t-test and Mann-Whitney U-test, except for MTLDS-4 at t = 40. The latter is significantly better than the pooled model, but is indistinguishable from the STLDS at the level α = 0.05. We also consider the convergence of the MTLDS to the true model with increasing N. For each experiment, we average the log marginal likelihood of the test sequences estimated via 10 000 (Sobol) samples from the prior. As before, the prior should be a good proposal for the aggregate posterior, and we amortize the same samples over all test sequences. In order to interpret the difference to the true distribution log p * (Y test) − log p(Y test | φ), we use the Bayes Factor interpretations given by. For instance a difference of 1.0 is'barely worth mentioning', but a difference of 4.0 is'strong evidence' that the distributions are different. We average over 10 000 test examples to avoid sampling variation of the test set. Figure 7 show boxplots of the log marginal likelihood for each model over increasing N, where the boxes show the interquartile range (IQR) over the 10 repetitions. We see convergence towards the true value with increasing N, with the difference of the MTLDS-128'barely worth mentioning'. Figure 9a, which is a subset of the CMU skeleton. Representation in observation space. We choose a Lagrangian representation (Figure 9c) where the coordinate frame is centered at the root joint of the skeleton (joint 1 in Fig. 9a, the pelvis), projected onto the ground. The frame is rotated such that the z-axis points in the "forward" direction, roughly normal to the body. This is in contrast to the Eulerian frame (Figure 9b) which has an absolute fixed position for all t. In the Lagrangian frame, the joint positions are always relative to the root joint, which avoids confusing the overall trajectory of the skeleton (typified by the root joint), and the overall rotation of the skeleton, with the local motions of the joints. The relative joint positions can be represented by spatial position or by joint angle. For the latter, the spatial positions of all joints can be recovered from the angle made with their parent joint via use of forward kinematics (FK). This construction ensures the constant bone length of the skeleton over time, which is a desirable property. However, it also substantially increases the sensitivity of internal joints. For instance, the rotation of the trunk will disproportionately affect the error of the joints in both arms. For this reason, we have chosen to model the spatial position of joints, which may in violations of bone length, but avoids these sensitivity issues. See also §2.1. One can further encode the joint positions via velocity (i.e. differencing) which may in smoother predictions. We avoid this encoding for the local joint motion (joints 2 to 21) since it can suffer from accumulated errors, but we do use it to predict the co-ordinate frame as is standard in mocap models. Hence our per-frame representation consists of the velocityẋ,ż,ω of the co-ordinate frame, the relative vertical position of the root joint, and 3-d position of the remaining 20 joints, which gives y t ∈ R 64. Choice of inputs. Our choice of inputs will reflect controls that an animator may wish to manipulate. The first input will be the trajectory that the skeleton is to follow. As in Holden et al. (2017b), we provide the trajectory over the next second (30 frames), sampled uniformly every 5 frames. Unlike previous work, there is no trajectory history in the inputs since this can be kept in the recurrent state. The (2-d) trajectory co-ordinates are given wrt. the current co-ordinate frame, and hence can rotate rapidly during a tight corner. In order to provide some continuity in the inputs, we also provide a first difference of the trajectory in Eulerian co-ordinates. Table 5: Hyper-parameters of mocap models. η denotes the learning rate. The velocity implied by the differenced trajectory does not disambiguate the gait frequency vs. stride length. The same motion might be achieved with fast short steps, or slower long strides. We therefore provide the gait frequency via a phasor (as in b), whose frequency may be externally controlled. This is provided by sine and cosine components to avoid the discontinuity at 2π. A final ambiguity exists from the trajectory at tight corners: the skeleton can rotate either towards the focus of the corner, or towards the outside. Figure 9d demonstrates the latter, which appears not infrequently in the data. We provide a boolean indicator alongside the trajectory which identifies corners for which this happens. Altogether we have u t ∈ R 32: 12 inputs for the Lagrangian trajectory, 12 inputs for the differenced Eulerian trajectory, 2 inputs for the gait phase and 6 inputs for the turning indicators. Extracting the root trajectory. The root trajectory is computed by projecting the root joint onto the ground. However, this projection may still contain information about the style of locomotion, for instance via swaying. We wish to remove all such information, since a model can otherwise learn the style without reference to a latent z. Our goal is to find an appropriately smoothed version of the extracted trajectory T. We use a cubic B-spline fit to control points fitted to the'corners' of the trajectory. These control points are selected using a polygonal approximation to T using the Ramer-Douglas-Peucker algorithm (RDP, e.g.). Briefly, the RDP algorithm uses a divide-and-conquer approach which greedily chooses points that minimize the Hausdorff distance of T to the polygonal approximation. Some per-style tuning of the RDP parameter, and a small number of manually added control points rendered this a semi-automatic process. Extracting the gait phase. Foot contacts are calculated via the code used by Holden et al. (2017b), which is based on thresholding of vertical position and velocity of each foot. As in we check visually for outliers and correct misclassified foot contacts manually. The leading edge of each foot contact is taken to represent 0 (left) and π (right), and the gait phase is calculated by interpolation. In this section we discuss elements of the experimental setup, learning and inference common to all experiments. Details particular to each experiment can be found in the following section. Further Model Details The MTDS architecture is described in section A.3, aside from the choice of prior. We tested both linear and nonlinear h φ in preliminary experiments and the performance was often similar. The nonlinear version used a one hidden layer MLP with 300 hidden units with tanh activations. For the final affine layer, we used a rank 30 matrix which, chosen pragmatically as a trade-off between flexibility and parameter count (see discussion in section A.1.1). Both choices often performed similarly, however the linear approach was chosen, since optimization of the latent z on new data was faster, and apparently more robust to choice of initialization. A nonlinear h φ may be more important when the base model is simpler. The benchmark models use an encoding length of τ = 64 frames. The encoder shares parameters with the decoder, i.e. the RNN is simply'warm started' for 64 frames before prediction. The benchmark models, unlike the MTDS, predict the difference from the previous frame (or 'velocity') via a residual architecture, as this performs better in. Further Learning Details Our primary goal was qualitative: to obtain good style-content separation, high quality animations and smooth interpolation between sequences. Therefore hyperparameter selection for the MTDS proceeded via quantitative means (via the ELBO) and visual inspection of the qualitative criteria. The qualitative desiderata motivated split learning rates between shared and multi-task networks (cf. section 4.2), and the amount of L2 regularization. See Table 5 for the chosen values. The main learning rate η applies to the fixed parameters wrt. z (i.e. ψ 1, H), and the multi-task learning rate applies to the parameter generation parameters φ and inference parameters λ. Standard variational inference proved more reliable than amortized inference: we used a Gaussian with diagonal covariance (parameterized using softplus) for the variational posterior over each z. L2 regularization was applied to φ, ψ 1, H. Unless otherwise specified, we optimized each model using a batch size N batch = 16 for 20 000 iterations. The ELBO had often reached a plateau by this time, and training even longer ed in a worse latent representation at times (as evidenced through poor style transfer). As noted in the main text, we remove the KL penalty of eq. for the initial 2 000 iterations, and enforce a small posterior standard deviation (s λ = 10 −3) for the same duration. This is similar to finding a MAP estimate for the {z}. For the remaining iterations, the original ELBO criterion is used, and the constraint on s λ is removed. The model is implemented in PyTorch and trained on GPUs. Since we use a fairly small max. sequence length L = 64, truncated backpropagation through time was not necessary. The hyper-parameters for the benchmark models were found (Table 5) using a grid search over learning rate and regularization, as well as the optimizers {Adam, (vanilla) SGD}. We performed the search over the pooled data for all 8 styles, with a stratified sample of 12.5% held out for a validation set. Once the hyperparameters were chosen, benchmark models were also trained for 20 000 iterations, recording the validation error every 1 000 iterations on a stratified 12.5% held out sample. The model with the lowest validation error during optimization is chosen. We standardize the data so that when pooled, each dimension has zero mean and unit variance. Finally, note that as discussed in section A.3.1, the data are represented in Lagrangian form, therefore drifts in the predicted trajectory from the true one are not necessarily heavily penalized. This can be altered by changing the weights on the root velocities, but we did not do this. Inference. At test time, especially for experiment 2, we cannot expect amortized inference to perform optimally, and we consider standard inference techniques. We want to understand the nature of the posterior distributions, and so we again used the AdaIS approach of section A.1.5. In practice, each posterior was unimodal and approximately Gaussian. Furthermore, the variation in sequence space for different z in the posterior was usually fairly small, and the posterior predictive mean performed similarly to using a point estimate. Each observation from which z is inferred is of size 64 × 64 and hence the posterior is fairly concentrated. Unlike the DHO model, this is a more expensive procedure. Our k = 3 experiments took approx. 24 seconds per observation for inference. An optimization approach using standard techniques may be expected to perform similarly at a reduced computational cost. Hence unless otherwise specified, inference was done via optimization. Experiment 1 -MTL The training data for each style uses 4 subsequences chosen carefully to represent the inter-style variation. Obviously it is important that frames are consecutive rather than randomly sampled. Over the increasing size training sets, each of these subsequences is a superset of the previous one. The 6 training set sizes (2 8, 2 9, 2 10, 2 11, 2 12, 2 13 frames per style) are not exact since short subsequences are discarded (e.g. at file boundaries), and the largest set contains all the training data except the test set 3, where data are not evenly distributed over styles. The test set comprises 4 sequences from each style, each of length 64, and is the same for all experiments. A length-64 seed sequence immediately preceding each test sequence was used for inference for all models. The models are trained as described above, except for the single task (STL) models. The STL models use an identical architecture to the pooled 1-layer GRU models, except they are trained only on the data for their style. Since there is less data for these models, we train them for a Table 6: Mocap Experiment 1 (MTL): predictive MSE for length-64 predictions where training sets are a given fraction of the original dataset. maximum of 5 000 iterations. We do not train 2-layer GRUs, since the amount of data is small for most experiments. The full are given in Table 6. We use fractions of the dataset instead of absolute training set sizes to aid understanding. The performance of the MTDS appears to increase with larger k, and suggests that we need k > 3 to achieve optimal performance on unseen training data. The demonstrate substantial benefit of the MTDS over a pooled RNN model in terms of sample efficiency, but not in asymptotic performance, as might be expected. According to a paired t-test, the improvements of the k = 7 MTDS over the (1-layer, open loop) pooled GRU are significant for training set sizes 3%, 7%, 13% and 27%. 4 At a style level, the k = 7 MTDS performs at least as well as the pooled GRUs for the first four training set sizes. See Figure 10. Note that the'angry' and'childlike' styles appear to be harder than the others, most likely due to their relatively high speed. For example animations of the MTL experiments, see the linked video in section A.4.1. Table 7 provides the aggregate of experiment 2 for each of the mocap models. A visualization is given in Figure 11. The 2-layer competitors are shown here for completeness, but they achieve similar performance to the 1-layer models on aggregate. Figure 12 provides a breakdown of these on a per-style basis. Styles 5-8 appear to be easier from the point of view of the benchmarks, but the MTDS shows equal or better performance on all styles except style 5. The competitor achieve better short-term performance than the MTDS. However, note that the zero-velocity baseline performs similarly to the open-loop GRUs for the first 5 predictions. This suggests that the MTDS may be improved for these early predictions simply by interpolating from the zero-velocity baseline for small values of t. We are unable to conclude from these experiments that the benchmark models can represent the style better initially, but simply that they can smooth the transition from the seed sequence better. The classifier is learned on the original observations to distinguish between the 8 styles. We use a 512-unit GRU to encode an observation sequence (usually of 64 frames), and transform the final state via a 300-unit hidden layer MLP with sigmoid activations into multinomial emissions. The model is trained via cross-entropy with 20% of the training data held out as a validation set; training was stopped as the validation error approached 0. We perform a standardization of the gait frequency across all styles, since some styles can be identified purely by calculating the frequency. The mean frequency across all styles (1 cycle per 33 frames) is applied to all sequences via linear interpolation. In this we make use of the instantaneous phase given in the inputs. We use a k = 8 latent code for the MTDS as the model is trained on all styles. for each source style s 1, with examples j = 1,..., 4. We next seek the'archetypal' latent code z associated with each target style s 2. For each s 2, we optimize z over 20 candidate values, obtained from the posterior mean of the style s 2 in the training set. Data are generated from all {U (s1) j } and the z which provides the greatest success in style transfer is chosen. The 32 highly varied input sequences guard against overfitting -the'archetypal' codes for each style must perform well across much of the variety of the original dataset. We provide a scalar measurement of the'success' of style transfer for each pair (s 1, s 2) by using the ing'probability' that the classifier assigns the target style s 2, averaged across the four input sequences for the source s 1. The of these experiments are shown in Figure 13a for the model with multi-task bias, and Figure 13b shows the for the full MTDS. Table 3c in the main text gives the marginal of these wrt. the target style. The cells in Figure 13 give the classifier probability for the target style for each (source, target) combination, averaged over the four source inputs. Successful style transfer should in a the classifier assigning a high score in every cell of the table. For most (source, target) pairs, the full MTDS model substantially outperforms the MTBias model: it appears that MTDS can control the prediction well in the majority of cases, and the MTBias model offers reduced control in general. However, we observe for both models that it is more difficult when styles are associated with extremes of the input distribution. Specifically, both the'childlike' and'angry' styles have unusually high speed inputs, and the'old' style has unusually low speeds. Note that in order to provide style transfer, the models are mostly ignoring these correlations, even though they are very useful for prediction. Further improvements may be available, perhaps by using an adversarial loss, or applying domain knowledge to the model. This is orthogonal to our contribution, and we leave this to future work. Providing style transfer from all varieties of source style is a challenging task. For instance, some styles include sources with widely varying speeds and actions, which may be mismatched to the target style. To understand what may be more typical use of the model, we provide an easier variant of this experiment where only one example of each source style is provided, rather than four. Note nevertheless that the same z (s2) is still used across all sources s 1. The of this secondary experiment are provided in Figure 14. In this case, style transfer is successful for almost all (source, target) pairs in the case of the MTDS, except for the angry style. The MTBias model still has many notable failures. 1. In-sample predictions https://vimeo.com/362069486. The goal is to showcase the best possible performance of the models by predicting from inputs in the training set. 2. MTL examples https://vimeo.com/362122944. Examples from Experiment 1. We compare the quality of animations and fit to the ground truth for two limited training set sizes (6.7% and 13.3% of the full data). For both models, MSE to the ground truth is given, averaged over the entire predictive window (length 256). This is different to the experimental setup which uses only the first 64 frames. 3. Novel test examples https://vimeo.com/362068342. Examples from Experiment 2. We show the adaptions obtained by each model to novel sequences, in particular showcasing examples of the pooled GRU models inferring suboptimal styles wrt. MSE. Again, MSE to the ground truth is given averaged over the predictive window (length 256). 4. Style morphing https://vimeo.com/361910646. This animation demonstrates the effect of changing the latent code over time. This also demonstrates style transfer and style interpolation from experiment 3. For style morphing, we found it useful to fix the dynamical bias of the second layer (parameter b in eq. 7) wrt. z since it otherwise ed in'jumps' while interpolating between sequences. We speculate that shifting the bias induces bifurcations in the state space, whereas adapting the transition matrix allows for smooth interpolation.

Represent the TLDR for finding the corresponding scientific article.

Tailoring predictions from sequence models (such as LDSs and RNNs) via an explicit latent code.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the article for finding a single-sentence TLDR

Question: By injecting adversarial examples into training data, adversarial training is promising for improving the robustness of deep learning models. However, most existing adversarial training approaches are based on a specific type of adversarial attack. It may not provide sufficiently representative samples from the adversarial domain, leading to a weak generalization ability on adversarial examples from other attacks. Moreover, during the adversarial training, adversarial perturbations on inputs are usually crafted by fast single-step adversaries so as to scale to large datasets. This work is mainly focused on the adversarial training yet efficient FGSM adversary. In this scenario, it is difficult to train a model with great generalization due to the lack of representative adversarial samples, aka the samples are unable to accurately reflect the adversarial domain. To alleviate this problem, we propose a novel Adversarial Training with Domain Adaptation (ATDA) method. Our intuition is to regard the adversarial training on FGSM adversary as a domain adaption task with limited number of target domain samples. The main idea is to learn a representation that is semantically meaningful and domain invariant on the clean domain as well as the adversarial domain. Empirical evaluations on Fashion-MNIST, SVHN, CIFAR-10 and CIFAR-100 demonstrate that ATDA can greatly improve the generalization of adversarial training and the smoothness of the learned models, and outperforms state-of-the-art methods on standard benchmark datasets. To show the transfer ability of our method, we also extend ATDA to the adversarial training on iterative attacks such as PGD-Adversial Training (PAT) and the defense performance is improved considerably. Deep learning techniques have shown impressive performance on image classification and many other computer vision tasks. However, recent works have revealed that deep learning models are often vulnerable to adversarial examples BID14, which are maliciously designed to deceive the target model by generating carefully crafted adversarial perturbations on original clean inputs. Moreover, adversarial examples can transfer across models to mislead other models with a high probability BID9. How to effectively defense against adversarial attacks is crucial for security-critical computer vision systems, such as autonomous driving. As a promising approach, adversarial training defends from adversarial perturbations by training a target classifier with adversarial examples. Researchers have found BID7 BID11 that adversarial training could increase the robustness of neural networks. However, adversarial training often obtains adversarial examples by taking a specific attack technique (e.g., FGSM) into consideration, so the defense targeted such attack and the trained model exhibits weak generalization ability on adversarial examples from other adversaries BID7. BID22 showed that the robustness of adversarial training can be easily circumvented by the attack that combines with random perturbation from other models. Accordingly, for most existing adversarial training methods, there is a risk of overfitting to adversarial examples crafted on the original model with the specific attack. In this paper, we propose a novel adversarial training method that is able to improve the generalization of adversarial training. From the perspective of domain adaptation (DA) BID20, there is a big domain gap between the distribution of clean examples and the distribution of adversarial examples in the high-level representation space, even though adversarial perturbations are imperceptible to humans. showed that adversarial perturbations are progressively amplified along the layer hierarchy of neural networks, which maximizes the distance between the original and adversarial subspace representations. In addition, adversarial training simply injects adversarial examples from a specific attack into the training set, but there is still a large sample space for adversarial examples. Accordingly, training with the classification loss on such a training set will probably lead to overfitting on the adversarial examples from the specific attack. Even though BID24 showed that adversarial training with iterative noisy attacks has stronger robustness than the adversarial training with single-step attacks, iterative attacks have a large computational cost and there is no theoretical analysis to justify that the adversarial examples sampled in such way could be sufficiently representative for the adversarial domain. Our contributions are focused on how to improve the generalization of adversarial training on the simple yet scalable attacks, such as FGSM (Goodfellow et al.). The key idea of our approach is to formulate the learning procedure as a domain adaptation problem with limited number of target domain samples, where target domain denotes adversarial domain. Specifically, we introduce unsupervised as well as supervised domain adaptation into adversarial training to minimize the gap and increase the similarity between the distributions of clean examples and adversarial examples. In this way, the learned models generalize well on adversarial examples from different ∞ bounded attacks. We evaluate our ATDA method on standard benchmark datasets. Empirical show that despite a small decay of accuracy on clean data, ATDA significantly improves the generalization ability of adversarial training and has the transfer ability to extend to adversarial training on PGD BID11. In this section, we introduce some notations and provides a brief overview of the current advanced attack methods, as well as the defense methods based on adversarial training. Denote the clean data domain and the adversarial data domain by D and A respectively, we consider a classifier based on a neural network f (x): DISPLAYFORM0 outputs the probability distribution for an input x ∈ d, and k denotes the number of classes in the classification task. Let ϕ be the mapping at the logits layer (the last neural layer before the final softmax function), so that f (x) = sof tmax(ϕ(x)). Let be the magnitude of the perturbation. Let x adv be the adversarial image computed by perturbing the original image x. The cost function of image classification is denoted as J(x, y). We define the logits as the logits layer representation, and define the logit space as the semantic space of the logits layer representation. We divide attacks into two types: white-box attacks have the complete knowledge of the target model and can fully access the model; black-box attacks have limited knowledge of the target classifier (e.g.,its architecture) but can not access the model weights. We consider four attack methods to generate adversarial examples. For all attacks, the components of adversarial examples are clipped in.Fast Gradient Sign Method (FGSM). Goodfellow et al. introduced FGSM to generate adversarial examples by applying perturbations in the direction of the gradient. DISPLAYFORM0 As compared with other attack methods, FGSM is a simple, yet fast and efficient adversary. Accordingly, FGSM is particularly amenable to adversarial training. Projected Gradient Descent (PGD). The Projected Gradient Descent (PGD) adversary was introduced by BID11 without random start, which is a stronger iterative variant of FGSM. This method applies FGSM iteratively for k times with a budget α instead of a single step. DISPLAYFORM1 Here clip(·, a, b) function forces its input to reside in the range of [a, b]. PGD usually yields a higher success rate than FGSM does in the white-box setting but shows weaker capability in the black-box setting.. BID22 proposed R+FGSM against adversarially trained models by applying a small random perturbation of step size α before applying FGSM. DISPLAYFORM0 Momentum Iterative Method (MIM). MIM ) is a modification of the iterative FGSM and it won the first place of NIPS 2017 Adversarial Attacks Competition. Its basic idea is to utilize the gradients of the previous t steps with a decay factor µ to update the gradient at step t + 1 before applying FGSM with a budget α. DISPLAYFORM1 An intuitive technique to defend a deep model against adversarial examples is adversarial training, which injects adversarial examples into the training data during the training process. First, Goodfellow et al. proposed to increase the robustness by feeding the model with both original and adversarial examples generated by FGSM and by learning with the modified objective function. BID7 scaled the adversarial training to ImageNet BID16 and showed better by replacing half the clean example at each batch with the corresponding adversarial examples. Meanwhile, BID7 discovered the label leaking effect and suggested not to use the FGSM defined with respect to the true label y true. However, their approach has weak robustness to the RAND+FGSM adversary. BID22 proposed an ensemble adversarial training to improve robustness on black-box attacks by injecting adversarial examples transferred from a number of fixed pre-trained models into the training data. DISPLAYFORM0 For adversarial training, another approach is to train only with adversarial examples. BID13 proposed a specialization of the method (Goodfellow et al.) that learned only with the objective function of adversarial examples. BID11 demonstrated successful defenses based on adversarial training with the noisy PGD, which randomly initialize an adversarial example within the allowed norm ball before running iterative attack. However, this technique is difficult to scale to large-scale neural networks BID6 as the iterative attack increases the training time by a factor that is roughly equal to the number of iterative steps. BID24 developed a robust training method by linear programming that minimized the loss for the worst case within the perturbation ball around each clean data point. However, their approach achieved high test error on clean data and it is still challenging to scale to deep or wide neural networks. As described above, though adversarial training is promising, it is difficult to select a representative adversary to train on and most existing methods are weak in generalization for various adversaries, as the region of the adversarial examples for each clean data is large and contiguous BID21 BID19. Furthermore, generating a representative set of adversarial examples for large-scale datasets is computationally expensive. In this work, instead of focusing on a better sampling strategy to obtain representative adversarial data from the adversarial domain, we are especially concerned with the problem of how to train with clean data and adversarial examples from the efficient FGSM, so that the adversarially trained model is strong in generalization for different adversaries and has a low computational cost during the training. We propose an Adversarial Training with Domain Adaptation (ATDA) method to defense adversarial attacks and expect the learned models generalize well for various adversarial examples. Our motivation is to treat the adversarial training on FGSM as a domain adaptation task with limited number of target domain samples, where the target domain denotes adversarial domain. We combine standard adversarial training with the domain adaptor, which minimizes the domain gap between clean examples and adversarial examples. In this way, our adversarially trained model is effective on adversarial examples crafted by FGSM but also shows great generalization on other adversaries. It's known that there is a huge shift in the distributions of clean data and adversarial data in the high-level representation space. Assume that in the logit space, data from either the clean domain or the adversarial domain follow a multivariate normal distribution, i.e., DISPLAYFORM0 Our goal is to learn the logits representation that minimizes the shift by aligning the covariance matrices and the mean vectors of the clean distribution and the adversarial distribution. To implement the CORrelation ALignment (CORAL), we define a covariance distance between the clean data and the adversarial data as follows. DISPLAYFORM1 where C ϕ(D) and C ϕ(A) are the covariance matrices of the clean data and the adversarial data in the logit space respectively, and · 1 denotes the L 1 norm of a matrix. Note that L CORAL (D, A) is slightly different from the CORAL loss proposed by BID17.Similarly, we use the standard distribution distance metric, Maximum Mean Discrepancy (MMD) BID1, to minimize the distance of the mean vectors of the clean data and the adversarial data. DISPLAYFORM2 The loss function for Unsupervised Domain Adaptation (UDA) can be calculated as follows. DISPLAYFORM3 Even though the unsupervised domain adaptation achieves perfect confusion alignment, there is no guarantee that samples of the same label from clean domain and adversarial domain would map nearby in the logit space. To effectively utilize the labeled data in the adversarial domain, we introduce a supervised domain adaptation (SDA) by proposing a new loss function, denoted as margin loss, to minimize the intra-class variations and maximize the inter-class variations on samples of different domains. The SDA loss is shown in Eq. FORMULA10. DISPLAYFORM0 Here sof tplus denotes a function ln(1 + exp(·)); c ytrue ∈ R k denotes the center of y true class in the logit space; C = {c j | j = 1, 2, ..., k} is a set consisting of the logits center for each class, which will be updated as the logits changed. Similar to the center loss BID23, we update center c j for each class j: DISPLAYFORM1 where 1 condition = 1 if the condition is true, otherwise 1 condition = 0; α denotes the learning rate of the centers. During the training process, the logits center for each class can integrate the logits representation from both the clean domain and the adversarial domain. For adversarial training, iterative attacks are fairly expensive to compute and single-step attacks are fast to compute. Accordingly, we use a variant of FGSM attack BID7 ) that avoids the label leaking effect to generate a new adversarial example x adv i for each clean example x i. DISPLAYFORM0 where y target denotes the predicted class arg max{ϕ(x i)} of the model. However, in this case, the sampled adversarial examples are aggressive but not sufficiently representative due to the fact that the sampled adversarial examples always lie at the boundary of the ∞ ball of radius (see FIG1) and the adversarial examples within the boundary are ignored. For adversarial training, if we train a deep neural network only on the clean data and the adversarial data from the FGSM attack, the adversarially trained model will overfit on these two kinds of data and exhibits weak generalization ability on the adversarial examples sampled from other attacks. From a different perspective, such problem can be viewed as a domain adaptation problem with limited number of labeled target domain samples, as only some special data point can be sampled in the adversarial domain by FGSM adversary. Consequently, it is natural to combine the adversarial training with domain adaptation to improve the generalization ability on adversarial data. We generate new adversarial examples by the variant of FGSM attack shown in Eq. FORMULA1, then we use the following loss function to meet the criteria of domain adaptation while training a strong classifier. Compute the loss by Eq. and update parameters of network f by back propagation; 10: until the training converges. DISPLAYFORM1 In this section, we evaluate our ATDA method on various benchmark datasets to demonstrate the robustness and contrast its performance against other competing methods under different white-box and black-box attacks with bounded ∞ norm. Code for these experiments is available at https: //github.com/JHL-HUST/ATDA. Datasets. We consider four popular datasets, namely Fashion-MNIST BID26, SVHN BID12, CIFAR-10 and CIFAR-100 BID5 Baselines. To evaluate the generalization power on adversarial examples in both the white-box and black-box settings, we report the clean test accuracy, the defense accuracy on FGSM, PGD, R+FGSM and MIM in the non-targeted way. The common settings for these attacks are shown in Table 5 of the Appendix. We compare our ATDA method with normal training as well as several state-of-the-art adversarial training methods:• Normal Training (NT). Training with cross-entropy loss on the clean training data.• Standard Adversarial Training (SAT) (Goodfellow et al.). Training with the cross-entropy on the clean training data and the adversarial examples from the FGSM variant with perturbation to avoid label leaking.• Ensemble Adversarial Training (EAT) BID22. Training with cross-entropy on the clean training data and the adversarial examples crafted from the currently trained model and the static pre-trained models by the FGSM variant with the perturbation to avoid label leaking.• Provably Robust Training (PRT) BID24. Training with cross-entropy loss on the worst case in the ∞ ball of radius around each clean training data point. It could be seen as training with a complicated method of sampling in the ∞ ball of radius. Evaluation Setup. For each benchmark dataset, we train a normal model and various adversarial models with perturbation on a main model with ConvNet architecture, and evaluate them on various attacks bounded by. Moreover, for Ensemble Adversarial Training (EAT), we use two different models as the static pre-trained models. For black-box attacks, we test trained models on the adversarial examples transferred from a model held out during the training. All experiments are implemented on a single Titan X GPU. For all experiments, we set the hyper-parameter λ in Eq. to 1/3 and the hyper-parameter α in Eq. to 0.1. For more details about neural network architectures and training hyper-parameters, see Appendix A. We tune the networks to make sure they work, not to post concentrates on optimizing these settings. We evaluate the defense performance of our ATDA method from the perspective of classification accuracy on various datasets, and compare with the baselines. Evaluation on Fashion-MNIST. The accuracy on Fashion-MNIST are reported in TAB1. NT yields the best performance on the clean data, but generalizes poorly on adversarial examples. SAT and EAT overfit on the clean data and the adversarial data from FGSM. PRT achieves lower error against various adversaries, but higher error on the clean data. ATDA achieves stronger robustness against different ∞ bounded adversaries as compared to SAT (adversarial training on FGSM).Evaluation on SVHN. The classification accuracy on SVHN are summarized in TAB1. PRT seems to degrade the performance on the clean testing data and exhibits weak robustness on various attacks. As compared to SAT, ATDA achieves stronger generalization ability on adversarial examples from various attacks and higher accuracy on the white-box adversaries, at the same time it only loses a negligible performance on clean data. Evaluation on CIFAR-10. Compared with Fashion-MNIST and SVHN, CIFAR-10 is a more difficult dataset for classification. As PRT is challenging and expensive to scale to large neural networks due to its complexity, the of PRT are not reported. The accuracy on CIFAR-10 are summarized in TAB1. ATDA outperforms all the competing methods on most adversaries, despite a slightly lower performance on clean data. Evaluation on CIFAR-100. The CIFAR-100 dataset contains 100 image classes, with 600 images per class. Our goal here is not to achieve state-of-the-art performance on CIFAR-100, but to compare the generalization ability of different training methods on a comparatively large dataset. The on CIFAR-100 are summarized in TAB1. Compared to SAT, ATDA achieves better generalization on various adversarial examples and it does not degrade the performance on clean data. In , the accuracy provide empirical evidence that ATDA has great generalization ability on different adversaries as compared to SAT and outperforms other competing methods. To further investigate the defence performance of the proposed method, we compute two other metrics: the local loss sensitivity to perturbations and the shift of adversarial data distribution with respect to the clean data distribution. Local Loss Sensitivity. One method to quantify smoothness and generalization to perturbations for models is the local loss sensitivity BID0. It is calculated in the clean testing data as follows. The lower the value is, the smoother the loss function is. DISPLAYFORM0 The of the local loss sensitivity for the aforementioned learned models are summarized in TAB2. The suggest that adversarial training methods do increase the smoothness of the model as compared with the normal training and ATDA performs the best. Distribution Discrepancy. To quantify the dissimilarity of the distributions between the clean data and the adversarial data, we compare our learned logits embeddings with the logits embeddings of the competing methods on Fashion-MNIST. We use t-SNE BID10 for the comparison on the training data, testing data and adversarial testing data from the white-box FGSM or PGD. The comparisons are illustrated in FIG3 and we report the detailed MMD distances across domains in TAB3. Compared with NT, SAT and EAT actually increase the MMD distance across domains of the clean data and the adversarial data. In contrast, PRT and ATDA can learn domain invariance between the clean domain and the adversarial domain. Furthermore, our learned logits representation achieves the best performance on domain invariance. FIG4. For each model, we report the average accuracy rates over all white-box attacks and all black-box attacks, respectively. The illustrate that, by aligning the covariance matrix and mean vector of the clean and adversarial examples, UDA plays a key role in improving the generalization of SAT on various attacks. In general, the aware of margin loss on SDA can also improve the defense quality on standard adversarial training, but the effectiveness is not very stable over all datasets. By combining UDA and SDA together with SAT, our final algorithm ATDA can exhibits stable improvements on the standard adversarial training. In general, the performance of ATDA is slightly better than SAT+UDA. We report the average accuracy rates over all white-box attacks and all black-box attacks, respectively. ATDA can simply be extended to adversarial training on other adversaries. We now consider to extend the ATDA method to PGD-Adversarial Training (PAT) BID11: adversarial training on the noisy PGD with perturbation. By combining adversarial training on the noisy PGD with domain adaptation, we implement an extension of ATDA for PAT, called PATDA. For the noisy PGD, we set the iterated step k as 10 and the budget α as /4 according to BID11.As shown in TAB4, we evaluate the defense performance of PAT and PATDA on various datasets. On Fashion-MNIST, we observe that PATDA fails to increase robustness to most adversaries as compared to PAT. On SVHN, PAT and PATDA fail to converge properly. The are not surprising, as training with the hard and sufficient adversarial examples (from the noisy PGD) requires the neural networks with more parameters. On CIFAR-10 and CIFAR-100, PATDA achieves stronger robustness to various attacks than PAT. In general, PATDA exhibits stronger robustness to various adversaries as compared to PAT. The indicate that domain adaptation can be applied flexibly to adversarial training on other adversaries to improve the defense performance. In this study, we regard the adversarial training as a domain adaptation task with limited number of target labeled data. By combining adversarial training on FGSM adversary with unsupervised and supervised domain adaptation, the generalization ability on adversarial examples from various attacks and the smoothness on the learned models can be highly improved for robust defense. In addition, ATDA can easily be extended to adversarial training on iterative attacks (e.g., PGD) to improve the defense performance. The experimental on several benchmark datasets suggest that the proposed ATDA and its extension PATDA achieve significantly better generalization as compared with current competing adversarial training methods. This work is supported by National Natural Science Foundation. In the appendix, we show all details of the common settings, neural network architectures and training hyper-parameters for the experiments. A.1 HYPER-PARAMETERS FOR ADVERSARIES.For each dataset, the details about the hyper-parameters of various adversaries are shown in Table 5, where denotes the magnitude of adversarial perturbations. Fashion-MNIST. In the training phase, we use Adam optimizer with a learning rate of 0.001 and set the batch size to 64. For Fashion-MNIST, the neural network architectures for the main model, the static pre-trained models and the model held out during training are depicted in TAB6. For all adversarial training methods, the magnitude of perturbations is 0.1 in ∞ norm. In the training phase, we use Adam optimizer with a learning rate of 0.001 and set the batch size to 32 and use the same architectures as in Fashion-MNIST. For all adversarial training methods, the magnitude of perturbations is 0.02 in ∞ norm. FORMULA1 FC FORMULA1 FC FORMULA1 CIFAR-10. In the training phase, we use the same training settings as in SVHN. we use Adam optimizer with a learning rate of 0.001 and set the batch size to 32. In order to enhance the expressive power of deep neural networks, we use Exponential Linear Unit (ELU) BID2 as the activation function and introduce Group Normalization BID25 into the architectures. The neural network architectures for CIFAR-10 are shown in Table 7. For all adversarial training methods, the magnitude of perturbations is 4/255 in ∞ norm. CIFAR-100. We use the same training settings as in CIFAR-10. For CIFAR-100, the neural network architectures for the main model, the static pre-trained models and the model held out during training are shown in Table 8. For all adversarial training methods, the magnitude of perturbations is 4/255 in ∞ norm.

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel adversarial training with domain adaptation method that significantly improves the generalization ability on adversarial examples from different attacks.

Batch normalization reduces adversarial robustness, as well as general robustness in many cases, particularly to noise corruptions.