Question: Represent the text to find a very short summary (1-2 sentences max).

Question: In this paper, we present a new generative model for learning latent embeddings. Compared to the classical generative process, where each observed data point is generated from an individual latent variable, our approach assumes a global latent variable to generate the whole set of observed data points. We then propose a learning objective that is derived as an approximation to a lower bound to the data log likelihood, leading to our algorithm, WiSE-ALE. Compared to the standard ELBO objective, where the variational posterior for each data point is encouraged to match the prior distribution, the WiSE-ALE objective matches the averaged posterior, over all samples, with the prior, allowing the sample-wise posterior distributions to have a wider range of acceptable embedding mean and variance and leading to better reconstruction quality in the auto-encoding process. Through various examples and comparison to other state-of-the-art VAE models, we demonstrate that WiSE-ALE has excellent information embedding properties, whilst still retaining the ability to learn a smooth, compact representation. Unsupervised learning is a central task in machine learning. Its objective can be informally described as learning a representation of some observed forms of information in a way that the representation summarizes the overall statistical regularities of the data BID0. Deep generative models are a popular choice for unsupervised learning, as they marry deep learning with probabilistic models to estimate a joint probability between high dimensional input variables x and unobserved latent variables z. Early successes of deep generative models came from Restricted Boltzmann Machines BID7 and Deep Boltzmann Machines BID15, which aim to learn a compact representation of data. However, the fully stochastic nature of the network requires layer-by-layer pre-training using MCMC-based sampling algorithms, ing in heavy computation cost. BID9 consider the objective of optimizing the parameters in an auto-encoder network by deriving an analytic solution to a variational lower bound of the log likelihood of the data, leading to the Auto-Encoding Variational Bayes (AEVB) algorithm. They apply a reparameterization trick to maximally utilize deterministic mappings in the network, significantly simplifying the training procedure and reducing instability. Furthermore, a regularization term naturally occurs in their model, allowing a prior p(z) to be placed over every sample embedding q(z|x). As a , the learned representation becomes compact and smooth; see e.g. FIG0 where we learn a 2D embedding of MNIST digits using 4 different methods and visualize the aggregate posterior distribution of 64 random samples in the learnt 2D embedding space. However, because the choice of the prior is often uninformative, the smoothness constraint imposed by this regularization term can cause information loss between the input samples and the latent embeddings, as shown by the merging of individual embedding distributions in FIG0 (d) (especially in the outer areas away from zero code). Extreme effects of such behaviours can be noticed from β-VAE BID6, a derivative algorithm of AEVB which further increases the weighting on the regularizing term with the aim of learning an even smoother, disentangled representation of the data. As shown in FIG0 (e), the individual embedding distributions are almost indistinguishable, leading to an overly severe information bottleneck which can cause high rates of distortion BID16. In contrast, perfect reconstruction can be achieved using WAE , but the learnt embedding distributions appear to severely non-smooth (FIG0), indicating a small amount of noise in the latent space would cause generation process to fail. In this paper, we propose WiSE-ALE (a wide sample estimator), which imposes a prior on the bulk statistics of a mini-batch of latent embeddings. Learning under our WiSE-ALE objective does not penalize individual embeddings lying away from the zero code, so long as the aggregate distribution (the average of all individual embedding distributions) does not violate the prior significantly. Hence, our approach mitigates the distortion caused by the current form of the prior constraint in the AEVB objective. Furthermore, the objective of our WiSE-ALE algorithm is derived by applying variational inference in a simple latent variable model (Section 2) and with further approximation, we derive an analytic form of the learning objective, ing in efficient learning algorithm. In general, the latent representation learned using our algorithm enjoys the following properties: 1) smoothness, as indicated in FIG0, the probability density for each individual embedding distribution decays smoothly from the peak value; 2) compactness, as individual embeddings tend to occupy a maximal local area in the latent space with minimal gaps in between; and 3) separation, indicated by the narrow, but clear borders between neighbouring embedding distributions as opposed to the merging seen in AEVB. In summary, our contributions are:• proposing a new latent variable model that uses a single global latent variable to generate the entire dataset,• deriving a variational lower bound to the data log likelihood in our latent variable model, which allows us to impose prior constraint on the bulk statistics of a mini-batch embedding distributions,• and deriving analytic approximations to the lower bound, leading to our efficient WiSE-ALE learning algorithm. In the rest of the paper, we first review directed graphical models in Section 2. We then derive our variational lower bound and its analytic approximations in Section 3. Related work is discussed in Section 4. Experiment are analyzed in Section 5, leading to in Section 6. Here we introduce the latent variable model used in our WiSE-ALE algorithm and compare with the latent variable model used in the AEVB algorithm BID9. DISPLAYFORM0, we assume x is generated from a latent variable z ∈ R dz of a much lower dimension. Here we denote x and z as random variables, DISPLAYFORM1 as the i-th input or latent code sample (i.e. a vector), and x i and z i as the random variable for x (i) and z (i). As shown in FIG1, this generative process can be modelled by a simple directed graphical model BID8, which models the joint probability distribution DISPLAYFORM2 is the data distribution for D N and p θ (x|z) and p θ (z|x) denote the complex transformation from the latent to the input space and reverse, where the transformation mapping is parameterised by θ. The learning task is to estimate the optimal set of θ so that this latent variable model can explain the data D N well. As the inference of the latent variable z given x (i.e. p θ (z|x)) cannot be directly estimated because p(x|D N) is unknown, both AEVB (FIG1) and our WiSE-ALE FIG1 ) resort to variational method to approximate the target distribution p θ (z|x) by a proposal distribution q φ (z|x) with the modified learning objective that both θ and φ are optimised so that the model can explain the data well and q φ (z|x) approaches p θ (z|x). The primary difference between the AEVB model and our WiSE-ALE model lies in how the joint probability p θ (x, z|D N) is modelled and specifically whether we assume an individual random variable for each latent code z (i). The AEVB model assumes a pair of random variables (x i, z i) for each x (i) and estimates the joint probability as DISPLAYFORM3 The equality between Eq. 2 and Eq. 3 can only be made with the assumption that the generation process for each x i is independent (first product in Eq. 3) and each z i is also independent (second product in Eq. 3). Such interpretation of the joint probability leads to the latent variable model in FIG1 (b) and the prior constraint (often taken as N (0, I) to encourage shrinkage when no data is observed) is imposed on every z i.In contrast, our WiSE-ALE model takes a single random variable to estimate the latent distribution for the entire dataset D N. Hence, the joint probability in our model can be broken down as DISPLAYFORM4 leading to the latent variable model illustrated in FIG1. The only assumption we make in our model is assuming the generative process of different input samples given the latent distribution of the current dataset as independent, which we consider as a sensible assumption. More significantly, we do not require independence between different z i as opposed to the AEVB model, leading to a more flexible model. Furthermore, the prior constraint in our model is naturally imposed on the aggregate posterior p(z|D N) for the entire dataset, leading to more flexibility for each individual sample latent code to shape an embedding distribution to preserve a better quality of information about the corresponding input sample. Neural networks can be used to parameterize p θ (x i |z i) in the generative model and q φ (z i |x i) in the inference model from the AEVB latent variable model or p θ (x i |z) and q φ (z|x i) correspondingly from our WiSE-ALE latent variable model. Both networks can be implemented through an auto-encoder network illustrated in FIG1 (d). In this section, we first define the aggregate posterior distribution p(z|D N) which serves as a core concept in our WiSE-ALE proposal. We then derive a variational lower bound to the data log likelihood log p(D N) using p(z|D N). Further, analytic approximation to the lower bound is derived, allowing efficient optimization of the model parameters and leading to our WiSE-ALE learning algorithm. Intuition of our proposal is also discussed. Here we formally define the aggregate posterior distribution p(z|D N), i.e. the latent distribution given the entire dataset D N. Considering DISPLAYFORM0 we have the aggregate posterior distribution for the entire dataset as the average of all the individual sample posteriors. The second equality in Eq. 9 is made by approximating the integral through summation. The third equality is obtained following the conventional assumption in the VAE literature that each input sample, DISPLAYFORM1, is drawn from the dataset D N with equal probability, i.e. DISPLAYFORM2 Similarly, for the estimated aggregate posterior distribution q(z|D N), we have DISPLAYFORM3 To carry out variational inference, we minimize the KL divergence between the estimated and the true aggregate posterior distributions q φ (z|D N) and p θ (z|D N), i.e. DISPLAYFORM0 in Eq. 11 and breaking down the products and fractions inside the log, we have DISPLAYFORM1 Re-arranging the above equation, we have DISPLAYFORM2 There are two terms in the derived lower bound: 1 a reconstruction likelihood term that indicates how likely the current dataset D N are generated by the aggregate latent posterior distribution q φ (z|D N) and 2 a prior constraint that penalizes severe deviation of the aggregate latent posterior distribution q φ (z|D N) from the preferred prior p(z), acting naturally as a regularizer. By maximizing the lower bound L WiSE-ALE (φ, θ; D N) defined in Eq. 12, we are approaching to log p(D N) and, hence, obtaining a set of parameters θ and φ that find a natural balance between a good reconstruction likelihood (good explanation of the observed data) and a reasonable level of compliance to the prior assumption (achieving some preferable properties of the posterior distribution, such as smoothness and compactness). To allow fast and efficient optimization of the model parameters θ and φ, we derive analytic approximations for the two terms in our proposed lower bound (Eq. 12). To approximate 1 reconstruction likelihood term in Eq. 12, we first substitute the definition of the approximate aggregate posterior given in Eq. 10 in the expectation operation in DISPLAYFORM0 Now we can decompose the p θ (D N |z) as a product of individual sample likelihood, due to the conditional independence, i.e. DISPLAYFORM1 Substituting this into Eq. 13, we have DISPLAYFORM2 Eq. 15 can be used to evaluate the reconstruction likelihood for D N. However, learning directly with this reconstruction estimate does not lead to convergence in our experiments. We choose to simplify the reconstruction likelihood further to be able to reach convergence during learning at the cost of losing the lower bound property of the objective function L WiSE-ALE (φ, θ; D N). Firstly, we apply Jensen inequality to the term inside the expectation in Eq. 15, leading to an upper bound of the reconstruction likelihood term as DISPLAYFORM3 Now (N − 1) sample-wise likelihood distributions in the summation inside the log can be dropped with the assumption that the p θ (x (j) |z) will only be non-zero if z is sampled from the posterior distribution of the same sample x (j) at the encoder, i.e. i = j. Therefore, the approximation becomes DISPLAYFORM4 Using the approximation of the reconstruction likelihood term given by Eq. 17 rather than Eq. 15, we are able to reach convergence efficiently during learning at the cost of the estimated objective no longer remaining a lower bound to log p(D N). Details of deriving the above approximation are given in Appendix A. The 2 prior constraint term D KL q φ (z|D N) p(z) in our objective function (Eq. 12) evaluates the KL divergence between the approximate aggregate posterior distribution q φ (z|D N) and a zero-mean, unit-variance Gaussian distribution p(z). Here we assume that each sample-wise posterior distribution can be modelled by a factorial Gaussian distribution, i.e. q φ (z|x DISPLAYFORM0, where k indicates the k-th dimension of the latent variable z and µ k (x (i) ) and σ 2 k (x (i) ) are the mean and variance of the k-th dimension embedding distribution for the input x (i). Therefore, D KL q φ (z|D N) p(z) computes the KL divergence between a mixture of Gaussians (as Eq. 10) and N (0, I). There is no analytical solution for such KL divergences. Hence, we derive an analytic upper bound allowing for efficient evaluation. DISPLAYFORM1 Applying Jensen inequality, i.e. E x log f (x) ≤ log E x f (x), to the first term inside the summation in Eq. 18, we have DISPLAYFORM2 Taking advantage of the Gaussian assumption for q φ (z|x (i) ) and p(z), we can compute the expectations in Eq. 20 analytically with the quoted below and the full derivation given in Appendix B. DISPLAYFORM3 where A = 2π (σ DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 When the overall objective function L WiSE-ALE (φ, θ; D N) in Eq. 12 is maximised, this upper bound approximation will approach the true KL divergence D KL q φ (z|D N) p(z), which ensures that the prior constraint on the overall aggregate posterior distribution takes effects. Combining from Section 3.3.1 and 3.3.2, we obtain an analytic approximation L WiSE-ALE approx DISPLAYFORM0 Eq. 12, as -4 -3 -2 -1 0 1 2 3 4 5 -4 -3 -2 -1 0 1 2 3 4 5 Ours: Figure 3: Comparison between our WiSE-ALE learning scheme and the AEVB estimator. AEVB imposes the prior constraint on every sample embedding distribution, whereas our WiSE-ALE imposes the constraint to the overall aggregate embedding distribution over the entire dataset (over a mini-batch as an approximation for efficient learning).shown below: DISPLAYFORM0 where we use L φ, θ | x DISPLAYFORM1 to denote the sample-wise reconstruction likelihood (φ, θ; D N) w.r.t the model parameters φ and θ, we are able to learn a model that naturally balances between a good embedding of the observed data and some preferred properties of the latent embedding distributions, such as smoothness and compactness. DISPLAYFORM2 Comparing the objective function in our WiSE-ALE algorithm and that proposed in AEVB algorithm BID9 DISPLAYFORM0 we notice that the difference lies in the form of prior constraint and the difference is illustrated in Fig. 3. AEVB learning algorithm imposes the prior constraint on every sample embedding distribution and any deviation away from the zero code or the unit variance will incur penalty. This will cause problems, as different samples cannot be simultaneously embedded to the zero code. Furthermore, when the model becomes more certain about the embedding of a specific sample as the learning continues, it will naturally favour a posterior distribution of small variance (e.g. less than 1). In contrast, our WiSE-ALE learning objective imposes the prior constraint on the aggregate posterior distribution, i.e. the average of all the sample embeddings. Such prior constraint will allow more flexibility for each sample posterior to settle at a mean and variance value in favour for good reconstruction quality, while preventing too large mean values (acting as a regulariser) or too small variance values (ensuring smoothness of the learnt latent representation).To investigate the different behaviours of the two prior constraints more concretely, we consider only two embedding distributions q(z|x ) and q(z|x ) (red dashed lines) in a 1D latent space, as shown in FIG2. The mean values of the two embedding distributions are fixed to make the analysis simple and their variances are allowed to change. When the variances of the two embedding distributions into the latent space (the more separable q(z|x ) and q(z|x ) are in the latent space, the easier it is to distinguish x are large, such as FIG2, q(z|x ) and q(z|x ) have a large area of overlap and it is difficult to distinguish the input samples x in the latent space, indicating the embedding only introduces a small level of information loss. Overall, the prior constraint in the AEVB objective favours the embedding distributions much closer to the uninformative N (0, I) prior, leading to large area of overlap between the individual posteriors, whereas our WiSE-ALE objective allows a wide range of acceptable embedding mean and variance, which will then offer more flexibility in the learnt posteriors to maintain a good reconstruction quality. So far our derivation has been for the entire dataset D N. Given a small subset B M with M samples randomly drawn from D N, we can obtain a variational lower bound for a mini-batch as: DISPLAYFORM0 When B M is reasonably large, then DISPLAYFORM1 Given the expressions for the objective functions derived in Section 3.3, we can compute the gradient for an approximation to the lower bound of a mini-batch B M and apply stochastic gradient ascent algorithm to iteratively optimize the parameters φ and θ. We can thus apply our WiSE-ALE algorithm efficiently to a mini-batch and learn a meaningful internal representation of the entire dataset. Algorithmically, WiSE-ALE is similar to AEVB, save for an alternate objective function as per Section 3.3.3. The procedural details of the algorithm are presented in Appendix C.4 RELATED WORK BID1 proposes that a learned representation of data should exhibit some generally preferable features, such as smoothness, sparsity and simplicity. However, these attributes are not tailored to any specific downstream tasks. Bayesian decision making (see e.g. ; BID4) requires consideration of a target task and proposes that any involved latent distribution approximations should be optimised for the performance over the target task, as well as conforming to the more general properties. The AEVB algorithm BID9 learns the latent posterior distribution under a reconstruction task, while simultaneously satisfying a prior constraint, which ensures the representation is smooth and compact. However, the prior constraint of the AEVB algorithm imposes significant influence on the solution space (as discussed in Section 3.4), and leads to a sacrifice of reconstruction quality. Our WiSE-ALE algorithm, however, prioritises the reconstruction task yet still enables globally desirable properties. WiSE-ALE is not the only algorithm that considers an alternate prior form to mitigate its impact on the reconstruction quality. The Gaussian Mixture VAE BID5 uses a Gaussian mixture model to parameterise p(z), encouraging more flexible sample posteriors. The Adversarial Auto-Encoder BID13 ) matches the aggregate posterior over the latent variables with a prior distribution through adversarial training. The WAE minimises a penalised form of the Wasserstein distance between the aggregate posterior distribution and the prior, claiming a generalisation of the AAE algorithm under the theory of optimal transport . More recently, the Sinkhorn Auto-Encoder BID14 ) builds a formal analysis of auto-encoders using an optimal transport based prior and uses the Sinkhorn algorithm as an alternative to estimate the Wasserstein distance in WAE.Our work differs from these in two main aspects. Firstly, our objective function can be evaluated analytically, leading to an efficient optimization process. In many of the above work, the optimization involves adversarial training and some hyper-parameter tuning, which leading to less efficient learning or even no convergence. Secondly, our WiSE-ALE algorithm naturally finds a balance between good reconstruction quality and preferred latent representation properties, such as smoothness and compactness, as shown in FIG0. In contrast, some other work sacrifice the properties of smoothness and compactness severely for improved reconstruction quality, as shown in FIG0. Many works have indicated that those properties of the learnt latent representation are essential for tasks that require optimisation over the latent space. We evaluate our WiSE-ALE algorithm in comparison with AEVB, β-VAE and WAE on the following 3 datasets. The implementation details for all experiments are given in Appendix E.1. Sine Wave. We generated 200,000 sine waves with small random noise: x(t) = A sin(2πf t + ϕ) +, each containing 256 samples, with independently sampled frequency f ∼ Unif(0, 20Hz), phase angle ϕ ∼ Unif(0, 2π) and amplitude A ∼ Unif. 2. MNIST . 70,000 28 × 28 binary images that contain hand-written digits.3. CelebA BID12. 202,599 RGB images of aligned celebrity faces of 218 × 178 are cropped to square images of 178 × 178 and resized to 64 × 64. Throughout all experiments, our method has shown consistently superior reconstruction quality compared to AEVB, β-VAE and WAE. FIG6 offers a graphical comparison across the reconstructed samples given by different methods for the sine wave and CelebA datasets. For the sine wave dataset, our WiSE-ALE algorithms achieves almost perfect reconstruction, whereas AEVB and β-VAE often struggle with low-frequency signals and have difficulty predicting the amplitude correctly. For the CelebA dataset, our WiSE-ALE manages to predict much sharper human faces, whereas the AEVB predictions are often blurry and personal characteristics are often ignored. WAE reaches a similar level of reconstruction quality to ours in some images, but it sometimes struggles with discovering the right elevation and azimuth angles, as shown in the second to the right column in FIG6. We understand that a good latent representation should not only reconstruct well, but also preserve some preferable qualities, such as smoothness, compactness and possibly meaningful interpretation of the original data. FIG0 indicates that our WiSE-ALE automatically learns a latent representation that finds a good tradeoff between minimizing the information loss and maintaining a smooth and compact aggregate posterior distribution. Furthermore, as shown in FIG7, we compare the ELBO values given by AEVB, β-VAE and our WiSE-ALE over training for the Sine dataset. Our WiSE-ALE manages to report the highest ELBO with a significantly lower reconstruction error and a fairly good performance in the KL divergence loss. This indicates that our WiSE-ALE is able to learn an overall good quality representation that is closest to the true latent distribution which gives rise to the data observation. In this paper, we propose a new latent variable model where a global latent variable is used to generate the entire dataset. We then derive a variational lower bound to the data log likelihood, which allows us to impose a prior constraint on the bulk statistics of the aggregate posterior distribution for the entire dataset. Using an analytic approximation to this lower bound as our learning objective, we propose WiSE-ALE algorithm. We have demonstrated its ability to achieve excellent reconstruction quality, as well as forming a smooth, compact and meaningful latent representation. In the future, we would like to understand the properties of the latent embeddings learnt through our method and apply it for suitable applications. In this appendix, we omit the trainable parameters φ and θ in the expressions of distributions for simplicity. For example, q(z|x) is equivalent to q φ (z|x) and p(x|z) represents p θ (x|z). Here we demonstration that the reconstruction term E q(z|D N) log p(D N |z) in our lower bound can be computed with individual sample likelihood log p(x (i) |z) and how our reconstruction error term becomes the same as the reconstruction term in the AEVB objective. Firstly, we can substitute DISPLAYFORM0 into the reconstruction term DISPLAYFORM1 Now we can decompose the the marginal likelihood of the entire dataset as a product of individual samples, due to the conditional independence, i.e. DISPLAYFORM2 Substituting this into the reconstruction term, we have: DISPLAYFORM3 To evaluate the reconstruction term in our lower bound, we need to do the following: 1) draw a sample x (i) from the dataset D N; 2) evaluate the latent code distribution q(z|x (i) ) through the encoder function q(·|x (i) ); 3) draw samples of z according to q(z|x (i) ); 4) reconstruct input samples using the sampled latent codes z (l); 5) compute the reconstruction error w.r.t to every single input sample and sum this error. We can simplify the above evaluation. Firstly, the sampling process in Step 3 can be replaced to a sampling process at the input using the reparameterisation trick. Besides, the sum of reconstruction errors w.r.t. all the input samples can be further simplified. To do this, we need to re-arrange the above expression as DISPLAYFORM4 log a i to the terms inside the expectation. As a , we have obtain an upper bound of the reconstruction error term as DISPLAYFORM5 This upper bound can be evaluated more efficiently with the assumption that the likelihood p(x (j) |z) representing the probability of a reconstructed sample from a latent code z imitating the sample x (j) will only be non-zero if z is sampled from the embedding prediction distribution with the same sample x (j) at the encoder input. With this assumption, N − 1 posterior distributions in the inner summation will be dropped as zeros and the only non-zero term is p(x (i) |z). Therefore, the upper bound becomes DISPLAYFORM6 The constant can be omitted, because it will not affect the gradient updates of the parameters. Applying Jensen inequality, i.e. DISPLAYFORM0 to the first term of above equation, we have DISPLAYFORM1 We will look at the two summation individually. The expectation w.r.t. the aggregate posterior can be expanded as DISPLAYFORM2 We assume the posterior distribution of the latent code z given a specific input sample DISPLAYFORM3 Similarly, DISPLAYFORM4 Therefore, DISPLAYFORM5 Substituting the exponential form for Gaussian distribution, i.e. DISPLAYFORM6 to the above equation, we have DISPLAYFORM7 The exponent of the above equation can be simplified to DISPLAYFORM8 Using the following properties, i.e. DISPLAYFORM9 we can evaluate the integral needed for DISPLAYFORM10 Therefore, we have obtained the expression for the first term in our upper bound, i.e. DISPLAYFORM11 DISPLAYFORM12 To find out the expression for the second term DISPLAYFORM13, we first examine the prior distribution p(z) which is chosen to be a zero-mean unit-variance Gaussian across all latent code dimensions, i.e. DISPLAYFORM14 Therefore, DISPLAYFORM15 Substituting this expression for log p(z) into DISPLAYFORM16 ) log p(z) and examining the expectation term for now, we have DISPLAYFORM17 The first integral q(z k |x (i) ) dz k = 1. To evaluate the second integral, we substitute Equation and use the following properties, i.e. DISPLAYFORM18 As a , we have DISPLAYFORM19 Therefore, DISPLAYFORM20 Combining the first term defined in Equation and the second term defined in Equation FORMULA12, we have obtained the expression for the overall upper bound as DISPLAYFORM21 + log 2π. We carry out experiments on four datasets (Sine wave, MNIST, Teapot and CelebA) to examine different properties of the latent representation learnt from the proposed WiSE algorithm. Specifically, we compare with β-VAE on the smoothness and disentanglement of the learnt representation and compare with WAE and AEVB on the reconstruction quality. In addition, by learning a 2D embedding of the MNIST dataset, we are able to visualise the latent embedding distributions learnt from AEVB, β-VAE, WAE and our WiSE and compare the compactness and smoothness of the learnt latent space across these methods. Here we give the implementation details for each dataset. We aim to learn a latent representation in R 4 for a one second long sine wave with sampling rate of 256Hz. The network architecture for the Sine wave dataset is shown below. x is an input sample, µ and σ are the latent code mean and latent code standard deviation to define the embedding distribution q(z|x) andx is the reconstructed input sample. is an auxiliary variable drawn from unit Gaussian at the input of the encoder network so that an estimate of a sample from the embedding distribution q(z|x) can be computed. Conv We aim to learn a 2D embedding of the MNIST dataset. The network architecture is shown below. Encoder network:x ∈ R 28×28×1 → Conv Decoder network: DISPLAYFORM0 ⇒x ∈ R We use the following hyper-parameters to train the network:

Represent the short summary for finding the corresponding scientific article (long)

We propose a new latent variable model to learn latent embeddings for some high-dimensional data.

represent the tldr for finding the corresponding scientific article.

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO


Question: Represent the article for finding a single-sentence TLDR

Question: We improve the robustness of deep neural nets to adversarial attacks by using an interpolating function as the output activation. This data-dependent activation function remarkably improves both classification accuracy and stability to adversarial perturbations. Together with the total variation minimization of adversarial images and augmented training, under the strongest attack, we achieve up to 20.6%, 50.7%, and 68.7% accuracy improvement w.r.t. the fast gradient sign method, iterative fast gradient sign method, and Carlini-WagnerL2attacks, respectively. Our defense strategy is additive to many of the existing methods. We give an intuitive explanation of our defense strategy via analyzing the geometry of the feature space. For reproducibility, the code will be available on GitHub. The adversarial vulnerability BID26 of deep neural nets (DNNs) threatens their applicability in security critical tasks, e.g., autonomous cars BID0, robotics BID8, DNN-based malware detection systems BID20 BID7. Since the pioneering work by BID26, many advanced adversarial attack schemes have been devised to generate imperceptible perturbations to sufficiently fool the DNNs BID6 BID19 BID5 BID29 BID11 BID2. And not only are adversarial attacks successful in white-box attacks, i.e. when the adversary has access to the DNN parameters, but they are also successful in black-box attacks, i.e. it has no access to the parameters. Black-box attacks are successful because one can perturb an image so it misclassifies on one DNN, and the same perturbed image also has a significant chance to be misclassified by another DNN; this is known as transferability of adversarial examples BID22 ). Due to this transferability, it is very easy to attack neural nets in a blackbox fashion BID14 BID4. In fact, there exist universal perturbations that can imperceptibly perturb any image and cause misclassification for any given network . There is much recent research on designing advanced adversarial attacks and defending against adversarial perturbation. In this work, we propose to defend against adversarial attacks by changing the DNNs' output activation function to a manifold-interpolating function, in order to seamlessly utilize the training data's information when performing inference. Together with the total variation minimization (TVM) and augmented training, we show state-of-the-art defense on the CIFAR-10 benchmark. Moreover, we show that adversarial images generated from attacking the DNNs with an interpolating function are more transferable to other DNNs, than those ing from attacking standard DNNs. Defensive distillation was recently proposed to increase the stability of DNNs which dramatically reduces the success rate of adversarial attacks BID21, and a related approach BID27 ) cleverly modifies the training data to increase robustness against black-box attacks, and adversarial attacks in general. To counter the adversarial perturbations, BID9 proposed to use image transformation, e.g., bit-depth reduction, JPEG compression, TVM, and image quilting. A similar idea of denoising the input was later explored by BID17, where they divide the input into patches, denoise each patch, and then reconstruct the image. These input transformations are intended to be non-differentiable, thus making adversarial attacks more difficult, especially for gradient-based attacks. BID25 noticed that small adversarial perturbations shift the distribution of adversarial images far from the distribution of clean images. Therefore they proposed to purify the adversarial images by PixelDefend. Adversarial training is another family of defense methods to improve the stability of DNNs BID6 BID15 BID18. And GANs are also employed for adversarial defense BID24. In BID1, the authors proposed a straight-through estimation of the gradient to attack the defense methods that is based on the obfuscated gradient. Meanwhile, many advanced attack methods have been proposed to attack the DNNs BID29 BID11.Instead of using softmax functions as the DNNs' output activation, BID28 utilized a class of non-parametric interpolating functions. This is a combination of both deep and manifold learning which causes the DNNs to sufficiently utilize the geometric information of the training data. The authors show a significant amount of generalization accuracy improvement, and the are more stable when one only has a limited amount of training data. In this section, we summarize the architecture, training, and testing procedures of the DNNs with the data-dependent activation BID28 ). An overview of training and testing of the standard DNNs with softmax output activation is shown in FIG0 and (b), respectively. In the kth iteration of training, given a mini-batch of training data X, Y, the procedure is:Forward propagation: Transform X into features by a DNN block (ensemble of convolutional layers, nonlinearities and others), and then pass this output through the softmax activation to obtain the predictionsỸ: DISPLAYFORM0 Then the loss is computed (e.g., cross entropy) between Y andỸ: L = Loss(Y,Ỹ).Backpropagation: Update weights (Θ k−1, W k−1) by gradient descent (learning rate γ): DISPLAYFORM1 Once the model is optimized, the predicted labels for testing data X are: BID28 proposed to replace the data-agnostic softmax activation by a data-dependent interpolating function, defined in the next section. Under review as a conference paper at ICLR 2019 DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 te m } be a subset of X which are labeled with label function g(x). We want to interpolate a function u that is defined on the entire manifold and can be used to label the entire dataset X. The harmonic extension is a natural and elegant approach to find such an interpolating function, which is defined by minimizing the Dirichlet energy functional: DISPLAYFORM5 with the boundary condition: DISPLAYFORM6 where w(x, y) is a weight function, typically chosen to be Gaussian: DISPLAYFORM7 ) with σ being a scaling parameter. The Euler-Lagrange equation for Eq. FORMULA5 is: DISPLAYFORM8 By solving the linear system (Eq. FORMULA8), we obtain labels u(x) for unlabeled data x ∈ X/X te. This interpolation becomes invalid when the labeled data is tiny, i.e., |X te | |X/X te |. To resolve this issue, the weights of the labeled data is increased in the Euler-Lagrange equation, which gives: DISPLAYFORM9 The solution u(x) to Eq. is named weighted nonlocal Laplacian (WNLL), denoted as DISPLAYFORM10 is the one-hot labels for the example x. In both training and testing of the WNLL-activated DNNs, we need to reserve a small portion of data/label pairs, denoted as (X te, Y te), to interpolate the label for new data Y. We name the reserved data (X te, Y te) as the template. Directly replacing softmax by WNLL has difficulties in back propagation, namely the true gradient ∂L ∂Θ is difficult to compute since WNLL defines a very complex implicit function. Instead, to train WNLL-activated DNNs, a proxy via an auxiliary neural net (FIG0) is employed. On top of the original DNNs, we add a buffer block (a fully connected layer followed by a ReLU), and followed by two parallel branches, WNLL and the linear (fully connected) layers. The auxiliary DNNs can be trained by alternating between training DNNs with linear and WNLL activations, respectively. The training loss of the WNLL activation function is backpropped via a straight-through estimation approach BID1 BID3. At test time, we remove the linear classifier from the neural nets and use the DNN and buffer blocks together with WNLL to predict new data (FIG0 ; here for simplicity, we merge the buffer block to the DNN block. For a given set of testing data X, and the labeled template {(X te, Y te)}, the predicted labels for X is given bỹ DISPLAYFORM0 We consider three benchmark attack methods in this work, namely, the fast gradient sign method (FGSM) BID6, iterative FGSM (IFGSM) BID13, and CarliniWagner's L 2 (CW-L2) BID5 attacks. We denote the classifier defined by the DNNs with softmax activation asỹ = f (θ, x) for a given instance (x, y). FGSM finds the adversarial image x by maximizing the loss L(x, y), subject to the l ∞ perturbation ||x −x|| ∞ ≤ with as the attack strength. Under the first order approximation i.e., DISPLAYFORM0, the optimal perturbation is given by DISPLAYFORM1 IFGSM iterates FGSM to generate enhanced adversarial images, i.e., DISPLAYFORM2 where m = 1, · · ·, M, x = x and x = x (M), with M be the number of iterations. The CW-L2 attack is proposed to circumvent defensive distillation. For a given image-label pair (x, y), and ∀t = y, CW-L2 searches the adversarial image that will be classified to class t by solving the optimization problem: min DISPLAYFORM3 where δ is the adversarial perturbation (for simplicity, we ignore the dependence of θ in f).The equality constraint in Eq. FORMULA15 is hard to satisfy, so instead Carlini et al. consider the surrogate DISPLAYFORM4 where Z(x) is the logit vector for an input x, i.e., output of the neural net before the softmax layer. Z(x) i is the logit value corresponding to class i. It is easy to see that f (x + δ) = t is equivalent to g(x + δ) ≤ 0. Therefore, the problem in Eq. can be reformulated as DISPLAYFORM5 where c ≥ 0 is the Lagrangian multiplier. By letting δ = This unconstrained optimization problem can be solved efficiently by the Adam optimizer BID12. All three of the attacks clip the values of the adversarial image x to between 0 and 1. In this work, we focus on untargeted attacks and defend against them. For a given small batch of testing images (X, Y) and template (X te, Y te), we denote the DNNs modified with WNLL as output activation asỸ = WNLL(Z({X, X te}), Y te ), where Z({X, X te}) is the composition of the DNN and buffer blocks as shown in FIG0. By ignoring dependence of the loss function on the parameters, the loss function for DNNs with WNLL activation can be written asL(X, Y, X te, Y te). The above attacks for DNNs with WNLL activation on the batch of images, X, are formulated below. DISPLAYFORM0 • IFGSM DISPLAYFORM1 where m = 1, 2, · · ·, N; X = X and X = X (M).• CW-L2 DISPLAYFORM2 where i is the logit values of the input images X.Based on our numerical experiments, the batch size of X has minimal influence on the adversarial attack and defense. In all of our experiments we choose the batch size of X to be 500. Similar to BID28, we choose the size of the template to be 500.We apply the above attack methods to ResNet-56 with either softmax or WNLL as the output activation function. For IFGSM, we run 10 iterations of Eqs. FORMULA14 and FORMULA5 to attack DNNs with two different output activations, respectively. For CW-L2 attacks (Eqs.) in both scenarios, we set the parameters c = 10 and κ = 0. FIG2 depicts three randomly selected images (horse, automobile, airplane) from the CIFAR-10 dataset, their adversarial versions by different attack methods on ResNet-56 with two kinds of activation functions, and the TV minimized images. All attacks successfully fool the classifiers to classify any of them correctly. FIG2 shows that FGSM and IFGSM with perturbation = 0.02 changes the contrast of the images, while it is still easy for humans to correctly classify them. The adversarial images of the CW-L2 attacks are imperceptible, however they are extremely strong in fooling DNNs. FIG2 shows the images of (a) with a stronger attack, = 0.08. With a larger, the adversarial images become more noisy. The TV minimized images of FIG2 (a) and (b) are shown in FIG2 and FORMULA3, respectively. The TVM removes a significant amount of detailed information from the original and adversarial images, meanwhile it also makes it harder for humans to classify both the TV-minimized version of the original and adversarial images. Visually, it is hard to discern the adversarial images ing from attacking the DNNs with two types of output layers. We consider the geometry of features of the original and adversarial images. We randomly select 1000 training and 100 testing images from the airplane and automobile classes, respectively. We consider two visualization strategies for ResNet-56 with softmax activation: extract the original 64D features output from the layer before the softmax, and apply the principle component analysis (PCA) to reduce them to 2D. However, the principle components (PCs) do not encode the entire geometric information of the features. Alternatively, we add a 2 by 2 fully connected (FC) layer before the softmax, then utilize the 2D features output from this newly added layer. We verify that the newly added layer does not change the performance of ResNet-56 as shown in FIG3, and that the training and testing performance remains essentially the same for these two cases. Figure 4 (a) and (b) show the 2D features generated by ResNet-56 with additional FC layer for the original and adversarial testing images, respectively, where we generate the adversarial images by using FGSM (= 0.02). Before adversarial perturbation FIG4 ), there is a straight line that can easily separate the two classes. The small perturbation causes the features to overlap and there is no linear classifier that can easily separate these two classes FIG4 ). The first two PCs of the 64D features of the clean and adversarial images are shown in FIG4 and FORMULA3, respectively. Again, the PCs are well separated for clean images, while adversarial perturbation causes overlap and concentration. The bottom charts of FIG4 depict the first two PCs of the 64D features output from the layer before the WNLL. The distributions of the unperturbed training and testing data are the same, as illustrated in panels (e) and (f). The new features are better separated which indicates that DNNs with WNLL are more robust to small random perturbation. Panels (g) and (h) plot the features of the adversarial and TV minimized adversarial images in the test set. The adversarial attacks move the automobiles' features to the airplanes' region and TVM helps to eliminate the outliers. Based on our computation, most of the adversarial images of the airplane classes can be correctly classified with the interpolating function. The training data guides the interpolating function to classify adversarial images correctly. The fact that the adversarial perturbations change the features' distribution was also noticed in BID25. DISPLAYFORM0 To defend against adversarials, we combine the ideas of data-dependent activation, input transformation, and training data augmentation. We train ResNet-56, respectively, on the original training data, the TV minimized training data, and a combination of the previous two. On top of the datadependent activation output and augmented training, we further apply the TVM BID23 used by BID9 to transform the adversarial images to boost defensive performance. The basic idea is to reconstruct the simplest image z from the sub-sampled image, X x, with X the mask filled by a Bernoulli binary random variable, by solving the following TVM problem where λ T V > 0 is the regularization constant. DISPLAYFORM0 7 NUMERICAL To verify the efficacy of attack methods for DNNs with WNLL output activation, we consider the transferability of adversarial images. We train ResNet-56 on the aforementioned three types of training data with either softmax or WNLL activation. After the DNNs are trained, we attack them by FGSM, IFGSM, and CW-L2 with different. Finally, we classify the adversarial images by using ResNet-56 with the opponent activation. We list the mutual classification accuracy on adversarial images in Table. 1. The adversarial images ing from attacking DNNs with two types of activation functions are both transferable, as the mutual classification accuracy is significantly lower than testing on the clean images. Overall we see a remarkably higher accuracy when applying ResNet-56 with WNLL activation to classify the adversarial images ing from attacking ResNet-56 with softmax activation. For instance, for DNNs that are trained on the original images and attacked by FGSM, DNNs with the WNLL classifier have at least 5.4% higher accuracy (56.3% v.s. 61.7% ( = 0.08)). The accuracy improvement is more significant in many other scenarios.7.2 ADVERSARIAL DEFENSE FIG5 plots the of adversarial defense by combining the WNLL activation, TVM, and training data augmentation. Panels (a), (b) and (c) show the testing accuracy of ResNet-56 with and without defense on CIFAR-10 data for FGSM, IFGSM, and CW-L2, respectively. It can be observed that with increasing attack strength,, the testing accuracy decreases rapidly. FGSM is a relatively weak attack method, as the accuracy remains above 53.5% (= 0.1) even with the strongest attack. Meanwhile, the defense maintains accuracy above 71.8% (= 0.02). FIG5 (b) and (c) show that both IFGSM and CW-L2 can fool ResNet-56 near completely even with small. The defense maintains the accuracy above 68.0%, 57.2%, respectively, under the CW-L2 and IFGSM attacks. Compared to state-of-the-art defensive methods on CIFAR-10, PixelDefend, our method is much simpler and faster. Without adversarial training, we have shown our defense is more stable to IFGSM, and more stable to all three attacks under the strongest attack than PixelDefend BID25. Moreover, our defense strategy is additive to adversarial training and many other defenses including PixelDefend. To analyze the defensive contribution from each component of the defensive strategy, we separate the three parts and list the testing accuracy in Table. 2. Simple TVM cannot defend FGSM attacks except when the DNNs are trained on the augmented data, as shown in the first and fourth horizontal blocks of the table. WNLL activation improves the testing accuracy of adversarial attacks significantly and persistently. Augmented training can improve the stability consistently as well. In this paper, by analyzing the influence of adversarial perturbations on the geometric structure of the DNNs' features, we propose to defend against adversarial attack by applying a data-dependent activation function, total variation minimization on the adversarial images, and training data augmentation. Results on ResNet-56 with CIFAR-10 benchmark reveal that the defense improves robustness to adversarial perturbation significantly. Total variation minimization simplifies the adversarial images, which is very useful in removing adversarial perturbation. Another interesting direction to explore is to apply other denoising methods to remove adversarial perturbation.

We proposal strategies for adversarial defense based on data dependent activation function, total variation minimization, and training data augmentation

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: This work presents a scalable solution to continuous visual speech recognition. To achieve this, we constructed the largest existing visual speech recognition dataset, consisting of pairs of text and video clips of faces speaking (3,886 hours of video). In tandem, we designed and trained an integrated lipreading system, consisting of a video processing pipeline that maps raw video to stable videos of lips and sequences of phonemes, a scalable deep neural network that maps the lip videos to sequences of phoneme distributions, and a production-level speech decoder that outputs sequences of words. The proposed system achieves a word error rate (WER) of 40.9% as measured on a held-out set. In comparison, professional lipreaders achieve either 86.4% or 92.9% WER on the same dataset when having access to additional types of contextual information. Our approach significantly improves on previous lipreading approaches, including variants of LipNet and of Watch, Attend, and Spell (WAS), which are only capable of 89.8% and 76.8% WER respectively. Deep learning techniques have allowed for significant advances in lipreading over the last few years BID6 BID72 BID30 BID80. However, these approaches have often been limited to narrow vocabularies, and relatively small datasets BID6 BID72 BID80. Often the approaches focus on single-word classification BID26 BID11 BID76 BID67 BID44 BID68 BID45 BID51 BID52 BID46 BID29 BID4 BID69 BID75 and do not attack the continuous recognition setting. In this paper, we contribute a novel method for large-vocabulary continuous visual speech recognition. We report substantial reductions in word error rate (WER) over the state-of-the-art approaches even with a larger vocabulary. Assisting people with speech impairments is a key motivating factor behind this work. Visual speech recognition could positively impact the lives of hundreds of thousands of patients with speech impairments worldwide. For example, in the U.S. alone 103,925 tracheostomies were performed in 2014 , a procedure that can in a difficulty to speak (disphonia) or an inability to produce voiced sound (aphonia). While this paper focuses on a scalable solution to lipreading using a vast diverse dataset, we also expand on this important medical application in Appendix A. The discussion there has been provided by medical experts and is aimed at medical practitioners. We propose a novel lipreading system, illustrated in Figure 1, which transforms raw video into a word sequence. The first component of this system is a data processing pipeline used to create the Large-Scale Visual Speech Recognition (LSVSR) dataset used in this work, distilled from YouTube videos and consisting of phoneme sequences paired with video clips of faces speaking (3,886 hours of video). The creation of the dataset alone required a non-trivial combination of computer vision and machine learning techniques. At a high-level this process takes as input raw video and annotated audio segments, filters and preprocesses them, and produces a collection of aligned phoneme and lip frame sequences. Compared to previous work on visual speech recognition, our pipeline uses landmark smoothing, a blurriness filter, an improved speaking classifier network and outputs phonemes. The details of this process are described in Section 3. Figure 1: The full visual speech recognition system introduced by this work consists of a data processing pipeline that generates lip and phoneme clips from YouTube videos (see Section 3), and a scalable deep neural network for phoneme recognition combined with a production-grade word-level decoding module used for inference (see Section 4).Next, this work introduces a new neural network architecture for lipreading, which we call Vision to Phoneme (V2P), trained to produce a sequence of phoneme distributions given a sequence of video frames. In light of the large scale of our dataset, the network design has been highly tuned to maximize predictive performance subject to the strong computational and memory limits of modern GPUs in a distributed setting. In this setting we found that techniques such as group normalization BID79 to be key to the reported . Furthermore, our approach is the first to combine a deep learning-based visual speech recognition model with production-grade word-level decoding techniques. By decoupling phoneme prediction and word decoding as is often done in speech recognition, we are able to arbitrarily extend the vocabulary without retraining the neural network. Details of our model and this decoding process are given in Section 4. By design, the trained model only performs well under optimal lighting conditions, within a certain distance from a subject, and at high quality. It does not perform well in other contexts. Finally, this entire lipreading system in an unprecedented WER of 40.9% as measured on a held-out set from our dataset. In comparison, professional lipreaders achieve either 86.4% or 92.9% WER on the same dataset, depending on the amount of context given. Similarly, previous state-of-the-art approaches such as variants of and of Watch, Attend, and Spell (WAS) demonstrated WERs of only 89.8% and 76.8% respectively. While there is a large body of literature on automated lipreading, much of the early work focused on single-word classification and relied on substantial prior knowledge BID10 BID38 BID53 BID35 BID48 BID82 BID23 BID49 ). For example, BID20 predicted continuous sequences of tri-visemes using a traditional HMM model with visual features extracted from a codebook of clustered mouth region images. The predicted visemes were used to distinguish sentences from a set of 150 possible sentences. Furthermore, BID55 predict words and sequences digits using HMMs, introduce multi-stream HMMs, and improve the performance by using visual features in addition to the lip contours. Later, BID10 used coupled HMMs to jointly model audio and visual streams to predict sequences of digits. BID43 used HMMs for sentence-level speech recognition in noisy environments of the IBM ViaVoice dataset by fusing handcrafted visual and audio features. More recent attempts using traditional speech, vision and machine learning pipelines include the works of BID19 BID47; and BID7. For further details, we refer the reader to the survey material of BID57 and BID83.However, as noted by BID83 and BID6, until recently generalization across speakers and extraction of motion features have been considered open problems. Advances in deep learning have made it possible to overcome these limitations, but most works still focus on single-word classification, either by learning visual-only representations BID26 BID11 BID76 BID67 BID75, multimodal audio-visual representations BID44 BID68 BID45 BID51 BID52, or combining deep networks with traditional speech techniques (e.g. HMMs and GMM-HMMs) BID46 BID29 BID4 BID69.LipNet BID6 was the first end-to-end model to tackle sentence-level lipreading by predicting character sequences. The model combined spatiotemporal convolutions with gated recurrent units (GRUs) and was trained using the CTC loss function. LipNet was evaluated on the GRID corpus BID15, a limited grammar and vocabulary dataset consisting of 28 hours of 5-word sentences, where it achieved 4.8% and 11.4% WER in overlapping and unseen speaker evaluations respectively. By comparison, the performance of competent human lipreaders on GRID was 47.7%. LipNet is the closest model to our neural network. Several similar architectures were subsequently introduced in the works of BID72 who study audio-visual feature fusion, BID30 who work on a small subset of 18 phonemes and 11 words to predict digit sequences, and BID80 who presented a model cascading CTC with attention. were the first to use sequence-to-sequence models with attention to tackle audiovisual speech recognition with a real-world dataset. The model "Watch, Listen, Attend and Spell" (WLAS), consists of a visual (WAS) and an audio (LAS) module. To evaluate WLAS, the authors created LRS, the largest dataset at that point with approximately 246 hours of clips from BBC news broadcasts, and introduced an efficient video processing pipeline. The authors reported 50.2% WER, with the performance of professional lipreaders being 87.6% WER. extended the work to multi-view sentence-level lipreading, achieving 62.8% WER for profile views and 56.4% WER for frontal views. Both and pre-learn features with the audio-video synchronization classifier of BID12, and fix these features in order to compensate for the large memory requirements of their attention networks. Contemporaneously with our work, BID2 presented LRS3-TED, a dataset generated from English language talks available online. Using pre-learned features BID0 presented a seq2seq and a CTC architecture based on character-level self-attention transformer models. On LRS3-TED, these models achieved a WER of 57.9% and 61.8% respectively. Other related advances include works using vision for silent speech reconstruction BID32 BID16 BID3 BID18 and for separating an audio signal to individual speech sources BID17 BID1.In contrast to the approach of BID6, our model (V2P) uses a network to predict a sequence of phoneme distributions which are then fed into a decoder to produce a sequence of words. This flexible design enables us to easily accommodate very large vocabularies, and in fact we can extend the size of the vocabulary without having to retrain the deep network. Unlike previous work, V2P is memory and computationally efficient without requiring pre-trained features.Finally, the data processing pipeline used in this work in a significantly larger and more diverse training dataset than in all previous efforts. While the first large-vocabulary lipreading dataset was IBM ViaVoice BID43, more recently the far larger LRS and MV-LRS datasets were generated from BBC news broadcasts, and the LRS3-TED dataset was generated from conference talks. MV-LRS and LRS3-TED are the only publicly available large-vocabulary datasets, although both are limited to academic usage. In comparison, our dataset (LSVSR) is an order of magnitude greater than any previous dataset with 3,886 hours of audio-video-text pairs. In addition, the content is much more varied (i.e. not news-specific), ing in a 2.2× larger vocabulary of 127,055 words. FIG1 shows a comparison of sentence-level (word sequence) visual speech recognition datasets. In this section we discuss the data processing pipeline, again illustrated in Figure 1, used to create the LSVSR dataset. Our pipeline makes heavy use of large-scale parallel processing and is implemented as a number of independent modules and filters on top of FlumeJava BID9. In particular, our dataset is extracted from public YouTube videos. This is a common strategy for building datasets in ASR and speech enhancement BID33 BID31 BID65 BID17.In our case, we build on the work of BID33 to extract audio clips paired with transcripts, yielding 140,000 hours of audio segments. After post-processing we obtain a dataset consisting of paired video and phoneme sequences, where video sequences are represented as identically-sized frames (here, 128×128) stacked in the time-dimension. Although our pipeline is used to process clips pre-selected from YouTube BID33, only about 2% of clips satisfy our filtering criteria. Finally, by eliminating the components marked by dashes in Figure 1, i.e. those components whose primary use are in producing paired training data, this same pipeline can be used in combination with a trained model to predict word sequences from raw videos. In what follows we describe the individual components that make up this pipeline. Length filter, language filter. The duration of each segment extracted from YouTube is limited to between 1 and 12 seconds, and the transcripts are filtered through a language classifier BID61 to remove non-English utterances. For evaluation, we further remove the utterances containing fewer than 6 words. Finally, the aligned phoneme sequences are obtained via a standard forced alignment approach using a lexicon with multiple pronunciations BID33. The phonetic alphabet is a reduced version of X-SAMPA with 40 phonemes plus silence. Raw videos, shot boundary detection, face detection. Constant spatial padding in each video segment is eliminated before a standard, thresholding color histogram classifier BID36 identifies and removes segments containing shot boundaries. FaceNet BID62 is used to detect and track faces in every remaining segment. Clip quality filter. Speech segments are joined with the set of tracked faces identified in the previous step and filtered based on the quality of the video, removing blurry clips and clips including faces with an eye-to-eye width of less than 80 pixels. Frame rates lower than 23fps are also eliminated BID60 BID71. We allow a range of input frame rates-varying frame rates has an effect similar to different speaking paces-however, frame rates above 30fps are downsampled. Face landmark smoothing. The segments are processed by a face landmark tracker and the ing landmark positions are smoothed using a temporal Gaussian kernel. Intuitively, this simplifies learning filters for the 3D convolution layers by reducing spatiotemporal noise. Empirically, our preliminary studies showed smoothing was crucial for achieving optimal performance. Next, following previous literature, we keep segments where the face yaw and pitch remain within ±30°. Models trained outside this range perform worse.View canonicalization. We obtain canonical faces using a reference canonical face model and by applying an affine transformation on the landmarks. Then, we use a thumbnail extractor which is configured to crop the area around the lips of the canonical face. Speaking filter. Using the extracted and smoothed landmarks, minor lip movements and nonspeaking faces are discarded using a threshold filter. This process involves computing the mouth openness in all frames, normalizing by the size of the face bounding box, and then thresholding on the standard deviation of the normalized openness. This classifier has very low computational cost, but high recall, e.g. voice-overs are not handled. It's contribution was very important to process approximately 16 years of audio-video-text pairs within reasonable time. Speaking classifier. As a final step, we build V2P-Sync, a neural network architecture to verify the audio and video channel alignment inspired by the work of BID12 and BID74. However, V2P-Sync takes advantage of face landmark smoothing, 3D convolutions, and high resolution inputs. V2P-Sync uses longer time segments as inputs and spatiotemporal convolutions as compared to the spatial-only convolutions of Chung & Zisserman, and view canonicalization and higher resolution inputs (128 × 128 vs 100 × 60) as compared to Torfi et al.. These characteristics facilitate the extraction of temporal features which is key to our task. V2P-Sync, takes as input a pair of a log mel-spectrogram and 9 grayscale video frames and produces an embedding for each using two separate neural network architectures. If the Euclidean distance of the audio and video embeddings is less than a given threshold the pair is classified as synchronized. The architecture is trained using a contrastive loss similar to Chung & Zisserman. Since there is no labeled data for training, the initial unfiltered pairs are used as positive samples with negative samples generated by randomly shifting the video of an unfiltered pair. After convergence the dataset is filtered using the trained model, which is then fine-tuned on the ing subset of the initial dataset. The final model is used to filter the dataset a second time, achieving an accuracy of 81.2%. This accuracy is improved as our audio-video pairs are processed by sliding V2P-Sync on 100 equally spaced segments and their scores are averaged. For further architectural details, we refer the reader to Appendix F. This work introduces the V2P model, which consists first of a 3d convolutional module for extracting spatiotemporal features from a given video clip. These features are then aggregated over time with a temporal module which outputs a sequence of phoneme distributions. Given input video clips and target phoneme sequences the model is trained using the CTC loss function. Finally, at test-time, a decoder based on finite state transducers (FSTs) is used to produce a word sequence given a sequence of phoneme distributions. For further details we refer the reader to Appendix G.Neural network architecture. Although the use of optical-flow filters as inputs is commonplace in lipreading BID37 BID22 BID81 BID70 BID77 BID63, in this work we designed a vision module based on VGG BID64 to explicitly address motion feature extraction. We adapted VGG to make it volumetric, which proved crucial in our preliminary empirical evaluation and has been established in previous literature BID6. The intuition behind this is the importance of spatiotemporal relationships in human visual speech recognition, e.g. measuring how lip shape changes over time. Furthermore, the receptive field of the vision module is 11 video frames, roughly 0.36-0.44 seconds, or around twice the typical duration of a phoneme. One of the main challenges in training a large vision module is finding an effective balance between performance and the imposed constraints of GPU memory. Our vision module consists of 5 convolutional layers with filters. By profiling a number of alternative architectures, we found that high memory usage typically came from the first two convolutional layers. To reduce the memory footprint we limit the number of convolutional filters in these layers, and since the frame is centered around the lips, we omit spatial padding. Since phoneme sequences can be quite long, but with relatively low frame rate (approximately 25-30 fps), we maintain padding in the temporal dimension and always convolve with unit stride in order to avoid limiting the number of output tokens. Despite tuning the model to reduce the number of activations, we are still only able to fit 2 batch elements on a GPU. Hence, we distribute training across 64 workers in order to achieve a batch size of 128. Due to communication costs, batch normalization is expensive if one wants to aggregate the statistics across all workers, and using only two examples per batch in noisy normalization statistics. Thus, instead of batch normalization, we use group normalization BID79, which divides the channels into groups and computes the statistics within these groups. This provides more stable learning regardless of batch size. The outputs of the convolutional stack are then fed into a temporal module which performs longerscale aggregation of the extracted features over time. In constructing this component we evaluated a number of recurrent neural network and dilated convolutional architectures, the latter of which are evaluated later as baselines. The best architecture presented performs temporal aggregation using a stack of 3 bidirectional LSTMs BID27 ) with a hidden state of 768, interleaved with group normalization. The output of these LSTM layers is then fed through a final MLP layer to produce a sequence of exactly T conditionally independent phoneme distributions p(u t |x). This entire model is then trained using the CTC loss we describe next. This model architecture is similar to that of the closest related work, LipNet BID6, but differs in a number of crucial ways. In comparison to our work, LipNet used GRU units and dropout, both of which we found to perform poorly in preliminary experiments. Our model is also much bigger: LipNet consists of only 3 convolutional layers of filters and 3 GRU layers with hidden state of size 256. Although the small size of LipNet means that it does not require any distributed computation to reach effective batch sizes, we will see that this drop in size coincides with a similar drop in performance. Finally, while both models use a CTC loss for training, the architecture used in V2P is trained to predict phonemes rather than characters; as we argue shortly this provides V2P with a much simpler mechanism for representing word uncertainty. CTC is a loss function for the parameterization of distributions over sequences of label tokens, without requiring alignments of the input sequence to the label tokens BID21. To see how CTC works, let V denote the set of singletimestep label tokens. To align a label sequence with size-T sequences given by the temporal module, CTC allows the model to output blank symbols and repeat consecutive symbols. Let the function B: (V ∪ {}) * → V * be defined such that, given a string potentially containing blank tokens, it deletes adjacent duplicate characters and removes any blanks. The probability of observing label sequence y can then be obtained by marginalizing over all possible alignments of this label, DISPLAYFORM0, where x is input video. For example, if T = 5 the probability of sequence'bee' is given by p(be e) + p(be e) + · · · + p(bbe e) + p(be ee). Note that there must be a blank between the'e' characters to avoid collapsing the sequence to'be'.Since CTC prevents us from using autoregressive connections to handle inter-timestep dependencies of the label sequence, the marginal distributions produced at each timestep of the temporal module are conditionally independent, as pointed out above. Therefore, to restore temporal dependency of the labels at test-time, CTC models are typically decoded with a beam search procedure that combines the probabilities with that of a language model. Rationale for phonemes and CTC. In speech recognition, whether on audio or visual signals, there are two main sources of uncertainty: uncertainty in the sounds that are in the input, and uncertainty in the words that correspond to these sounds. This suggests modelling DISPLAYFORM1 where the approximation is by the assumption that a given word sequence often has a single or dominant pronunciation. While previous work uses CTC to model characters given audio or visual input directly BID6 BID5, we argue this is problematic as the conditional independence of CTC timesteps means that the temporal module must assign a high probability to a single sequence in order to not produce spurious modes in the CTC distribution. To explain why modeling characters with CTC is problematic, consider two character sequences "fare" and "fair" that are homophones, i.e. they have the same pronunciation (i.e. /f :/). The difficulty we will describe is independent of the model used, so we will consider a simple unconditional model where each character c is assigned probability given by the parameters π c t = P (u t = c) and the probability of a sequence is given by its product, e.g. p(fare) = π. The maximum likelihood estimate, arg max π p(fare)p(fair), however, assigns equal 1/4 probability to each of "fare", "fair", "faie", "farr", as shown in FIG2, ing in two undesirable words. Ultimately this difficulty arises due to the independence assumption of CTC and the many-to-many mapping of characters to words 1. This same difficulty arises if we replace the parameters above with the outputs of a network mapping from videos to tokens. An additional source of uncertainty in visual speech recognition is introduced by the fact that the information required to disambiguate different phonemes is not visible (e.g. the position of the tongue). The ing visually similar phonemes are called visemes. This can be seen in Appendix C where the ratio of insertions and substitutions when computing the edit distance of a visual model is substantially higher than the ratio of an audio model trained on the same dataset. Furthermore, recent literature suggests that phoneme-to-viseme mappings may differ per speaker BID8, making it difficult to incorporate this knowledge in the design. Thus, using phonemes, which have a one-to-many mapping to words, allows the temporal model to only model visual uncertainty, and the word uncertainty can instead be handled by the decoder described below. Alternatively to using phonemes with CTC, some previous work solves this problem using RNN transducers or sequence-to-sequence with attention, which jointly model all sources of uncertainty. However, showed in the context of acoustic speech recognition that these models were unable to significantly outperform a baseline CTC model (albeit using context-dependent phonemes and further sequence-discriminative training) when combined with a decoding pipeline similar to ours. Hence, for reasons of performance and easier model training, especially important with our large model, we choose to output phonemes rather than words or characters directly. Additionally, and crucial for many applications, CTC also provides extra flexibility over alternatives. The fact that the lexicon (phoneme to word mapping) and language model are separate and part of the decoder, affords one the ability to trivially change the vocabulary and language model (LM) arbitrarily. This allows for visual speech recognition in narrower domains or updating the vocabulary and LM with new words without requiring retraining of the phoneme recognition model. This is nontrivial in other models, where the language model is part of the RNN.Decoding. As described earlier, our model produces a sequence of phoneme distributions; given these distributions we use an industry-standard decoding method using finite state transducers (FSTs) to arrive at word sequences. Such techniques are extensively used in speech recognition (e.g. BID40 BID39 ; we refer the reader to the thorough presentation of BID41 . In our work we make use of a combination of three individual (weighted) FSTs, or WFSTs. The first CTC postprocessing FST removes duplicate symbols and CTC blanks. Next, a lexicon FST maps input phonemes to output words. Third, an n-gram language model with backoff can be represented as a WFST from words to words. In our case, we use a 5-gram model with Katz backoff with about 50 million n-grams and a vocabulary size of about one million. The composition of these three FSTs another WFST transducing from phoneme sequences to (reweighted) word sequences. Finally, a search procedure is employed to find likely word sequences from phoneme distributions. We examine the performance of V2P trained on LSVSR with hyperparameters tuned on a validation set. We evaluate it on a held-out test set roughly 37 minutes long, containing approximately 63,000 video frames and 7100 words. We also describe and compare against a number of alternate methods from previous work. In particular, we show that our system gives significant performance improvements over professional lipreaders as well previous state-of-the-art methods for visual speech recognition. Except for V2P-NoLM, all models used the same 5-gram word-level language model during decoding. To construct the validation and test sets we removed blurry videos by thresholding the variance of the Laplacian of each frame BID50; we kept them in the training set as a form of data augmentation. Professional lipreaders. We consulted a professional lipreading company to measure the difficulty of LSVSR and hence the impact that such a model could have. Since the inherent ambiguity in lipreading necessitates relying on context, we conducted experiments both with and without context. In both cases we generate modified clips from our test set, but cropping the whole head in the video, as opposed to just the mouth region used by our model. The lipreaders could view the video up to 10 times, at half or normal speed each time. To measure without-context performance, we selected clips with transcripts that had at least 6 words. To measure how much context helps performance, we selected clips with at least 12 words, and presented to the lipreader the first 6 words, the title, and the category of the video, then asked them to transcribe the rest of the clip. The lipreaders transcribed a subset of our test set containing 153 and 274 videos with and without context, respectively. Audio-Ph. For an approximate bound on performance, we train an audio speech recognition model on the audio of the utterances. The architecture is based on Deep Speech 2 BID5, but trained to predict phonemes rather than characters. Baseline-LipNet-Ch. Using our training setup, we replicate the character-level CTC architecture of LipNet BID6. As with the phoneme models, we use an FST decoding pipeline and the same language model, but instead of a phoneme-based lexicon we use a character-level one as described in BID40.Baseline-LipNet-Ph. We also train LipNet to predict phonemes, still with CTC and using the same FST-based decoding pipeline and language model. Baseline-LipNet-Large-Ph. Recall from the earlier discussion that LipNet uses dropout, whereas V2P makes heavy use of group normalization, crucial for our small batches per worker. For a fair size-wise comparison, we introduce a replica of V2P, that uses GRUs, dropout, and no normalization. Baseline-Seq2seq-Ch. Using our training setup, we compared to a variant of the previous stateof-the-art sequence-to-sequence architecture of WAS that predicts character sequences. Although their implementation was followed as closely as possible, training end-toend quickly exceeded the memory limitations of modern GPUs. To work around these problems, the authors kept the convolutional weights fixed using a pretrained network from audio-visual synchronization classification BID12 ), which we were unable to use as their network inputs were processed differently. Instead, we replace the 2D convolutional network with the improved lightweight 3D visual processing network of V2P. From our empirical evaluation, including preliminary experiments not reported here and as shown by earlier work BID6, we believe that the 3D spatiotemporal aggregation of features benefits performance. After standard beam search decoding, we use the same 5-gram word LM as used for the CTC models to perform reranking. V2P-FullyConv. Identical to V2P, except the LSTMs in the temporal aggregation module are replaced with 6 dilated temporal convolution layers with a kernel size of 3 and dilation rates of, yielding a fully convolutional model with 12 layers. V2P-NoLM. Identical to V2P, except during decoding, where the LM is replaced with a dictionary consisting of 100k words. The words are then weighted by their smoothed frequency in the training data, essentially a uni-gram language model. TAB1 shows the phoneme error rate, character error rate, and word error rate for all of the models, and the number of parameters for each. The error rates are computed as the sum of the edit distances of the predicted and ground-truth sequence pairs divided by total ground-truth length. We also compute and display the standard error associated with each rate, estimated by bootstrap sampling. These show that the variant of LipNet tested in this work is approximately able to perform on-par with professional lipreaders with WER of 86.4% and 89.8% respectively, even when the given professional is given additional context. Similarly, we see that the WAS variant provides a substantial reduction to this error, ing in a WER of 76.8%. However, the full V2P method presented in this Professional w/o context − − − 92.9 ± 0.9 Professional w/ context − − − 86.4 ± 1.4 Audio-Ph 58M 12.5 ± 0.5 11.5 ± 0.6 18.3 ± 0.9Baseline-LipNet-Ch 7M − 64.6 ± 0.5 93.0 ± 0.6 Baseline-LipNet-Ph 7M 65.8 ± 0.4 72.8 ± 0.5 89.8 ± 0.5 Baseline-Seq2seq-Ch 15M − 49.9 ± 0.6 76.8 ± 0.8 Baseline-LipNet-Large-Ph 40M 53.0 ± 0.5 54.0 ± 0.8 72.7 ± 1.0 V2P-FullyConv 29M 41.3 ± 0.6 36.7 ± 0.9 51.6 ± 1.2 V2P-NoLM 49M 33.6 ± 0.6 34.6 ± 0.8 53.6 ± 1.0 V2P 49M 33.6 ± 0.6 28.3 ± 0.9 40.9 ± 1.2 work is able to further halve the WER, obtaining a value of 40.9% at testing time. Interestingly, we see that although the bi-directional LSTM provides the best performance, using a fully-convolutional network still in performance that is significantly better than all previous methods. Finally, although we see that the full V2P model performs best, removing the language model only in a drop of approximately 13 WER to 53.6%.By predicting phonemes directly, we also side-step the need to design phoneme-to-viseme mappings BID8. The inherent uncertainty is instead modelled directly in the predictive distribution. For instance, using edit distance alignments of the predictions to the ground-truths, we can determine which phonemes were most frequently erroneously included or missed, as shown in FIG4. Here we normalize the rates of deletions vs insertions, however empirically we saw that deletions were much more common than inclusions. Among these errors the most common include phonemes that are often occluded by the teeth (/d/, /n/, and /t/) as well as the most common English vowel /@/. Finally, by differentiating the likelihood of the phoneme sequence with respect to the inputs using guided backpropagation , we compute the saliency maps shown in the top row of FIG6 as a white overlay. The entropy at each timestep of the phoneme predictive distribution is shown as well. A full confusion matrix, absolute deletion/insertion/substitution counts, and additional saliency maps are shown in Appendices B, C and E. 47.0 ± 1.6 55.1 ± 0.9 DISPLAYFORM0 To demonstrate the generalization power of our V2P approach, we also compare it to the of the TM-seq2seq model of BID0 on LRS3-TED BID2. Unlike LSVSR, the LRS3-TED dataset includes faces at angles between ±90°instead of ±30°, and clips may be shorter than one second. Despite the fact that we do not train or fine-tune V2P on LRS3-TED, our approach still outperforms the state-of-the-art model trained on that dataset in terms of test set accuracy. In particular, we conducted two experiments. First, we evaluated performance on a subset of the LRS3-TED test set filtered according to the same protocol used to construct LSVSR, by removing instances with larger face angles and shorter clips (Filtered Test). Second, we tested on the full unfiltered test set (Full Test). In both cases, V2P outperforms TM-seq2seq, achieving WERs of 47.0 ± 1.6 and 55.1 ± 0.9 respectively. This shows that our approach is able to generalize well, achieving state-of-the-art performance on datasets, with different conditions, on which it was not trained. We presented a novel, large-scale visual speech recognition system. Our system consists of a data processing pipeline used to construct a vast dataset-an order of magnitude greater than all previous approaches both in terms of vocabulary and the sheer number of example sequences. We described a scalable model for producing phoneme and word sequences from processed video clips that is capable of nearly halving the error rate of the previous state-of-the-art methods on this dataset, and achieving a new state-of-the-art in a dataset presented contemporaneously with this work. The combination of methods in this work represents a significant improvement in lipreading performance, a technology which can enhance automatic speech recognition systems, and which has enormous potential to improve the lives of speech impaired patients worldwide. A MEDICAL APPLICATIONS As a consequence of injury or disease and its associated treatment, millions of people worldwide have communication problems preventing them from generating sound. As hearing aids and cochlear transplants have transformed the lives of people with hearing loss, there is potential for lip reading technology to provide alternative communication strategies for people who have lost their voice. Aphonia is the inability to produce voiced sound. It may from injury, paralysis, removal or other disorders of the larynx. Common examples of primary aphonia include bilateral recurrent laryngeal nerve damage as a of thyroidectomy (removal of the thyroid gland and any tumour) for thyroid cancer, laryngectomy (surgical removal of the voice box) for laryngeal cancers, or tracheostomy (the creation of an alternate airway in the neck bypassing the voicebox). Dysphonia is difficulty in speaking due to a physical disorder of the mouth, tongue, throat, or vocal cords. Unlike aphonia, patients retain some ability to speak. For example, in Spasmodic dysphonia, a disorder in which the laryngeal muscles go into periods of spasm, patients experience breaks or interruptions in the voice, often every few sentences, which can make a person difficult to understand. We see this work having potential medical applications for patients with aphonia or dysphonia in at least two distinct settings. Firstly, an acute care setting (i.e. a hospital with an emergency room and an intensive care unit), patients frequently undergo elective (planned) or emergency (unplanned) procedures (e.g. Tracheostomy) which may in aphonia or dysphonia. In the U.S. 103,925 tracheostomies were performed in 2014, ing in an average hospital stay of 29 days . Similarly, in England and Wales 15,000 tracheostomies are performed each year.Where these procedures are unplanned, there is often no time or opportunity to psychologically prepare the patient for their loss of voice, or to teach the patient alternative communication strategies. Some conditions that necessitate tracheotomy, such as high spinal cord injuries, also affect limb function, further hampering alternative communication methods such as writing. Even where procedures are planned, such as for head and neck cancers, despite preparation of the patient through consultation with a speech and language therapist, many patients find their loss of voice highly frustrating especially in the immediate post-operative period. Secondly, where surgery has left these patients cancer-free, they may live for many years, even decades without the ability to speak effectively, in these patients we can envisage that they may use this technology in the community, after discharge from hospital. While some patients may either have tracheotomy reversed, or adapt to speaking via a voice prosthesis, electro-larynx or esophageal speech, many patients do not achieve functional spoken communication. Even in those who achieve good face-to-face spoken communication, few laryngectomy patients can communicate effectively on the telephone, and face the frequent frustration of being hung-up on by call centres and others who do not know them. Acute care applications. It is widely acknowledged that patients with communication disabilities, including speech impairment or aphonia can pose significant challenges in the clinical environment, especially in acute care settings, leading to potentially poorer quality of care BID42. While some patients will be aware prior to surgery that they may wake up unable to speak, for many patients in the acute setting (e.g. Cervical Spinal Cord Injury, sudden airway obstruction) who wake up following an unplanned tracheotomy, their sudden inability to communicate can be phenomenally distressing. Community applications. Patients who are discharged from hospital without the ability to speak, or with poor speech quality, face a multitude of challenges in day-to-day life which limits their independence, social functioning and ability to seek employment. We hypothesize that the application of technology capable of lip-reading individuals with the ability to move their facial muscles, but without the ability to speak audibly could significantly improve quality of life for these patients. Where the application of this technology improves the person's ability to communicate over the telephone, it would enhance not only their social interactions, but also their ability to work effectively in jobs that require speaking over the phone. Finally, in patients who are neither able to speak, nor to move their arms, this technology could represent a step-change in terms of the speed at which they can communicate, as compared to eye-tracking or facial muscle based approaches in use today. To compute the confusion matrix and the insertion/deletion chart shown in the main text in FIG4, we first compute the edit distance dynamic programming matrix between each predicted sequence of phonemes and the corresponding ground-truth. Then, a backtrace through this matrix gives an alignment of the two sequences, consisting of edit operations paired with positions in the prediction/ground-truth sequences. Counting the correct phonemes and the substitutions yields the confusion matrix Figure 7. The reader can note that the diagonal is strongly dominant. A few groups are commonly confused as expected due to their visual similarity, such as {/d/, /n/, /t/}, and to a lesser extent {/b/, /p/}.Counting insertions/deletions yields FIG4 in the main text, showing which phonemes are most commonly omitted (deleted), or less frequently, erroneously inserted. Using the same method for computing the edit distance as described in Appendix B, TAB3 shows the absolute numbers of insertions/deletions/substitutions for the Audio-Ph and the V2P model. As it can be seen, the percentage of insertions and substitutions is substantially higher when using the visual channel (V2P) compared to the audio channel (Audio-Ph). Different phonemes can be visually identical and the information to disambiguate them is missing from the visual channel, this may to a higher insertions and substitutions rate. TAB4 are optimized using a batch size of 128, batch normalization, and Adam BID28 ) with a learning rate of 10 −4 and default hyperparameters: first and second momentum coefficients 0.9 and 0.999 respectively, and = 10 −8 for numerical stability. Layer Filter size Stride Output channels Input conv1 3 × 3 × 3 1 × 2 × 2 16 9 × 128 × 128 × 1 pool1 1 × 2 × 2 1 × 2 × 2 7 × 63 × 63 × 16 conv2 3 × 3 × 3 1 × 1 × 1 32 7 × 31 × 31 × 16 pool2 1 × 2 × 2 1 × 2 × 2 5 × 29 × 29 × 32 conv3 3 × 3 × 3 1 × 1 × 1 64 5 × 14 × 14 × 32 pool3 1 × 2 × 2 1 × 2 × 2 3 × 12 × 12 × 64 conv4 3 × 3 × 3 1 × 1 × 1 128 3 × 6 × 6 × 64 pool4 1 × 2 × 2 1 × 2 × 2 1 × 4 × 4 × 128 fc5 1 × 1 × 1 256 512 fc6 1 × 1 × 1 64 256 and default hyperparameters: first and second momentum coefficients 0.9 and 0.999 respectively, and = 10 −8 for numerical stability. Furthermore, to accelerate learning, a curriculum schedule limits the video duration, starting from 2 seconds and gradually increasing to a maximum length of 12 seconds over 200,000 training steps. Finally, image transformations are also applied to augment the image frames to help improve invariance to filming conditions. This is accomplished by first randomly mirroring the videos horizontally, followed by random changes to brightness, contrast, saturation, and hue.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work presents a scalable solution to continuous visual speech recognition.

Represent the TLDR for finding the corresponding scientific article

The Wave-U-Net architecture, recently introduced by Stoller et al for music source separation, is highly effective for speech enhancement, beating the state of the art.