Question: Represent the scientific article to find a short TLDR

Question: Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret. We propose a variational message-passing algorithm for variational inference in such models. We make three contributions. First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (VAE). Second, we establish conditions under which such inference networks enable fast amortized inference similar to VAE. Finally, we derive a variational message passing algorithm to perform efficient natural-gradient inference while retaining the efficiency of the amortized inference. By simultaneously enabling structured, amortized, and natural-gradient inference for deep structured models, our method simplifies and generalizes existing methods. To analyze real-world data, machine learning relies on models that can extract useful patterns. Deep Neural Networks (DNNs) are a popular choice for this purpose because they can learn flexible representations. Another popular choice are probabilistic graphical models (PGMs) which can find interpretable structures in the data. Recent work on combining these two types of models hopes to exploit their complimentary strengths and provide powerful models that are also easy to interpret BID10 BID14 BID0 BID3.To apply such hybrid models to real-world problems, we need efficient algorithms that can extract useful structure from the data. However, the two fields of deep learning and PGMs traditionally use different types of algorithms. For deep learning, stochastic-gradient methods are the most popular choice, e.g., those based on back-propagation. These algorithms are not only widely applicable, but can also employ amortized inference to enable fast inference at test time BID17 BID12. On the other hand, most popular algorithms for PGMs exploit the model's graphical conjugacy structure to gain computational efficiency, e.g., variational message passing (VMP) BID18, expectation propagation BID16, Kalman filtering BID4 BID5, and more recently natural-gradient variational inference BID9 and stochastic variational inference BID8. In short, the two fields of deep learning and probabilistic modelling employ fundamentally different inferential strategies and a natural question is, whether we can design algorithms that combine their respective strengths. There have been several attempts to design such methods in the recent years, e.g., BID14; BID3; BID0; BID10; BID2. Our work in this paper is inspired by the previous work of BID10 that aims to combine message-passing, natural-gradient, and amortized inference. Our proposed method in this paper simplifies and generalizes the method of BID10.To do so, we propose Structured Inference Networks (SIN) that incorporate the PGM structure in the standard inference networks used in variational auto-encoders (VAE) BID12 BID17. We derive conditions under which such inference networks can enable fast amortized inference similar to VAE. By using a recent VMP method of BID11, we The generative models are just like the decoder in VAE but they employ a structured prior, e.g., Fig. (a) has a mixture-model prior while Fig. (b) has a dynamical system prior. SINs, just like the encoder in VAE, mimic the structure of the generative model by using parameters φ. One main difference is that in SIN the arrows between y n and x n are reversed compared to the model, while rest of the arrows have the same direction.derive a variational message-passing algorithm whose messages automatically reduce to stochasticgradients for the deep components of the model, while perform natural-gradient updates for the PGM part. Overall, our algorithm enables Structured, Amortized, and Natural-gradient (SAN) updates and therefore we call our algorithm the SAN algorithm. We show that our algorithm give comparable performance to the method of BID10 while simplifying and generalizing it. The code to reproduce our is available at https://github.com/emtiyaz/vmp-for-svae/. We consider the modelling of data vectors y n by using local latent vectors x n. Following previous works BID10 BID0 BID14, we model the output y n given x n using a neural network with parameters θ NN, and capture the correlations among data vectors y:= {y 1, y 2, . . ., y N} using a probabilistic graphical model (PGM) over the latent vectors x:= {x 1, x 2, . . ., x N}. Specifically, we use the following joint distribution: DISPLAYFORM0 where θ NN and θ PGM are parameters of a DNN and PGM respectively, and θ:= {θ NN, θ PGM}.This combination of probabilistic graphical model and neural network is referred to as structured variational auto-encoder (SVAE) by BID10. SVAE employs a structured prior p(x|θ PGM) to extract useful structure from the data. SVAE therefore differs from VAE BID12 where the prior distribution over x is simply a multivariate Gaussian distribution p(x) = N (x|0, I) with no special structure. To illustrate this difference, we now give an example. Example (Mixture-Model Prior): Suppose we wish to group the outputs y n into K distinct clusters. For such a task, the standard Gaussian prior used in VAE is not a useful prior. We could instead use a mixture-model prior over x n, as suggested by BID10, DISPLAYFORM1 where z n ∈ {1, 2, . . ., K} is the mixture indicator for the n'th data example, and π k are mixing proportions that sum to 1 over k. Each mixture component can further be modelled, e.g., by using a Gaussian distribution p(x n |z n = k):= N (x n |µ k, Σ k) giving us the Gaussian Mixture Model (GMM) prior with PGM hyperparameters DISPLAYFORM2. The graphical model of an SVAE with such priors is shown in FIG0. This type of structured-prior is useful for discovering clusters in the data, making them easier to interpret than VAE.Our main goal in this paper is to approximate the posterior distribution p(x, θ|y). Specifically, similar to VAE, we would like to approximate the posterior of x by using an inference network. In VAE, this is done by using a function parameterized by DNN, as shown below: DISPLAYFORM3 where the left hand side is the posterior distribution of x, and the first equality is obtained by using the distribution of the decoder in the Bayes' rule. The right hand side is the distribution of the encoder where q is typically an exponential-family distribution whose natural-parameters are modelled by using a DNN f φ with parameters φ. The same function f φ (·) is used for all n which reduces the number of variational parameters and enables sharing of statistical strengths across n. This leads to both faster training and faster testing BID17.Unfortunately, for SVAE, such inference networks may give inaccurate predictions since they ignore the structure of the PGM prior p(x|θ PGM). For example, suppose y n is a time-series and we model x n using a dynamical system as depicted in FIG0. In this case, the inference network of FORMULA3 is not an accurate approximation since it ignores the time-series structure in x. This might in inaccurate predictions of distant future observations, e.g., prediction for an observation y 10 given the past data {y 1, y 2, y 3} would be inaccurate because the inference network has no path connecting x 10 to x 1, x 2, or x 3. In general, whenever the prior structure is important in obtaining accurate predictions, we might want to incorporate it in the inference network. A solution to this problem is to use an inference network with the same structure as the model but to replace all its edges by neural networks BID14 BID3 ). This solution is reasonable when the PGM itself is complex, but might be too aggressive when the PGM is a simple model, e.g., when the prior in FIG0 is a linear dynamical system. Using DNNs in such cases would dramatically increase the number of parameters which will lead to a possible deterioration in both speed and performance. BID10 propose a method to incorporate the structure of the PGM part in the inference network. For SVAE with conditionally-conjugate PGM priors, they aim to obtain a mean-field variational inference by optimizing the following standard variational lower bound 1: DISPLAYFORM4 where q(x|λ x) is a minimal exponential-family distribution with natural parameters λ x. To incorporate an inference network, they need to restrict the parameter of q(x|λ x) similar to the VAE encoder shown in, i.e., λ x must be defined using a DNN with parameter φ. For this purpose, they use a two-stage iterative procedure. In the first stage, they obtain λ * x by optimizing a surrogate lower bound where the decoder in is replaced by the VAE encoder of (highlighted in blue), DISPLAYFORM5 The optimal λ * x is a function of θ and φ and they denote it by λ * x (θ, φ). In the second stage, they substitute λ * x into and take a gradient step to optimize L(λ * x (θ, φ), θ) with respect to θ and φ. This is iterated until convergence. The first stage ensures that q(x|λ x) is defined in terms of φ similar to VAE, while the second stage improves the lower bound while maintaining this restriction. The advantage of this formulation is that when the factors q(x n |f φ (y n)) are chosen to be conjugate to p(x|θ PGM), the first stage can be performed efficiently using VMP. However, the overall method might be difficult to implement and tune. This is because the procedure is equivalent to an implicitly-constrained optimization 2 that optimizes with the constraint λ * x (θ, φ) = arg max λxL (λ x, θ, φ). Such constrained problems are typically more difficult to solve than their unconstrained counterparts, especially when the constraints are nonconvex BID6. Theoretically, the convergence of such methods is difficult to guarantee when the constraints are violated. In practice, this makes the implementation difficult because in every iteration the VMP updates need to run long enough to reach close to a local optimum of the surrogate lower bound. Another disadvantage of the method of BID10 is that its efficiency could be ensured only under restrictive assumptions on the PGM prior. For example, the method does not work for PGMs that contain non-conjugate factors because in that case VMP cannot be used to optimize the surrogate lower bound. In addition, the method is not directly applicable when λ x is constrained and when p(x|θ PGM) has additional latent variables (e.g., indicator variables z n in the mixture-model example). In summary, the method of BID10 might be difficult to implement and tune, and also difficult to generalize to cases when PGM is complex. In this paper, we propose an algorithm to simplify and generalize the algorithm of BID10. We propose structured inference networks (SIN) that incorporate the structure of the PGM part in the VAE inference network. Even when the graphical model contains a non-conjugate factor, SIN can preserve some structure of the model. We derive conditions under which SIN can enable efficient amortized inference by using stochastic gradients. We discuss many examples to illustrate the design of SIN for many types of PGM structures. Finally, we derive a VMP algorithm to perform natural-gradient variational inference on the PGM part while retaining the efficiency of the amortized inference on the DNN part. We start with the design of inference networks that incorporate the PGM structure into the inference network of VAE. We propose the following structured inference network (SIN) which consists of two types of factors, DISPLAYFORM0 The DNN factor here is similar to while the PGM factor is an exponential-family distribution which has a similar graph structure as the PGM prior p(x|θ PGM). The role of the DNN term is to enable flexibility while the role of the PGM term is to incorporate the model's PGM structure into the inference network. Both factors have their own parameters. φ NN is the parameter of DNN and φ PGM is the natural parameter of the PGM factor. The parameter set is denoted by φ:= {φ NN, φ PGM}.How should we choose the two factors? As we will show soon that, for fast amortized inference, these factors need to satisfy the following two conditions. The first condition is that the normalizing constant 3 log Z(φ) is easy to evaluate and differentiate. The second condition is that we can draw samples from SIN, i.e., x * (φ) ∼ q(x|y, φ) where we have denoted the sample by x * (φ) to show its dependence on φ. An additional desirable, although not necessary, feature is to be able to compute the gradient of x * (φ) by using the reparameterization trick. Now, we will show that given these two conditions we can easily perform amortized inference. We show that when the above two conditions are met, a stochastic gradient of the lower bound can be computed in a similar way as in VAE. For now, we assume that θ is a deterministic variable (we will relax this in the next section). The variational lower bound in this case can be written as follows: DISPLAYFORM1 The first term above is identical to the lower bound of the standard VAE, while the rest of the terms are different (shown in blue). The second term differs due to the PGM prior in the generative model. In VAE, p(x|θ PGM) is a standard normal, but here it is a structured PGM prior. The last two terms arise due to the PGM term in SIN. If we can compute the gradients of the last three terms and generate samples x * (φ) from SIN, we can perform amortized inference similar to VAE. Fortunately, the second and third terms are usually easy for PGMs, therefore we only require the gradient of Z(φ) to be easy to compute. This confirms the two conditions required for a fast amortized inference. The ing expressions for the stochastic gradients are shown below where we highlight in blue the additional gradient computations required on top of a VAE implementation (we also drop the explicit dependence of x * (φ) over φ for notational simplicity). DISPLAYFORM2 DISPLAYFORM3 The gradients of Z(φ) and x * (φ) might be cheap or costly depending on the type of PGM. For example, for LDS, these require a full inference through the model which costs O(N) computation and is infeasible for large N. However, for GMM, each x n can be independently sampled and therefore computations are independent of N. In general, if the latent variables in PGM are highly correlated (e.g., Gaussian process prior), then Bayesian inference is not computationally efficient and gradients are difficult to compute. In this paper, we do not consider such difficult cases and assume that Z(φ) and x * (φ) can be evaluated and differentiated cheaply. We now give many examples of SIN that meet the two conditions required for a fast amortized inference. When p(x|θ PGM) is a conjugate exponential-family distribution, choosing the two factors is a very easy task. In this case, we can let q(x|φ PGM) = p(x|φ PGM), i.e., the second factor is the same distribution as the PGM prior but with a different set of parameters φ PGM. To illustrate this, we give an example below when the PGM prior is a linear dynamical system. Example (SIN for Linear Dynamical System (LDS)): When y n is a time series, we can model the latent x n using an LDS defined as p(x|θ):= N (x 0 |µ 0, Σ 0) N n=1 N (x n |Ax n−1, Q), where A is the transition matrix, Q is the process-noise covariance, and µ 0 and Σ 0 are the mean and covariance of the initial distribution. Therefore, θ PGM:= {A, Q, µ 0, Σ 0}. In our inference network, we choose q(x|φ PGM) = p(x|φ PGM) as show below, where φ PGM:= {Ā,Q,μ 0,Σ 0} and, since our PGM is a Gaussian, we choose the DNN factor to be a Gaussian as well: DISPLAYFORM4 where m n:= m φNN (y n) and V n:= V φNN (y n) are mean and covariance parameterized by a DNN with parameter φ NN. The generative model and SIN are shown in FIG0, respectively. The above SIN is a conjugate model where the marginal likelihood and distributions can be computed in O(N) using the forward-backward algorithm, a.k.a. Kalman smoother BID1. We can also compute the gradient of Z(φ) as shown in BID13.When the PGM prior has additional latent variables, e.g., the GMM prior has cluster indicators z n, we might want to incorporate their structure in SIN. This is illustrate in the example below. Example (SIN for GMM prior): The prior shown in has an additional set of latent variables z n. To mimic this structure in SIN, we choose the PGM factor as shown below with parameters DISPLAYFORM5, while keeping the DNN part to be a Gaussian distribution similar to the LDS case: DISPLAYFORM6 The model and SIN are shown in FIG0 and 1b, respectively. Fortunately, due to conjugacy of the Gaussian and multinomial distributions, we can marginalize x n to get a closed-form expression for log Z(φ):= n log k N (m n |μ k, V n +Σ k)π k. We can sample from SIN by first sampling from the marginal q(z n = k|y, φ) ∝ N m n |μ k, V n +Σ k π k. Given z n, we can sample x n from the following conditional: DISPLAYFORM7 In all of the above examples, we are able to satisfy the two conditions even when we use the same structure as the model. However, this may not always be possible for all conditionally-conjugate exponential family distributions. However, we can still obtain samples from a tractable structured mean-field approximation using VMP. We illustrate this for the switching state-space model in Appendix A. In such cases, a drawback of our method is that we need to run VMP long enough to get a sample, very similar to the method of BID10. However, our gradients are simpler to compute than theirs. Their method requires gradients of λ * (θ, φ) which depends both on θ and φ (see Proposition 4.2 in BID10). In our case, we require gradient of Z(φ) which is independent of θ and therefore is simpler to implement. An advantage of our method over the method of BID10 is that our method can handle non-conjugate factors in the generative model. When the PGM prior contains some non-conjugate factors, we might replace them by their closest conjugate approximations while making sure that the inference network captures the useful structure present in the posterior distribution. We illustrate this on a Student's t mixture model. To handle outliers in the data, we might want to use the Student's t-mixture component in the mixture model shown in, i.e., we set p(x n |z n = k) = T (x n |µ k, Σ k, γ k) with mean µ k, scale matrix Σ k and degree of freedom γ k. The Student's t-distribution is not conjugate to the multinomial distribution, therefore, if we use it as the PGM factor in SIN, we will not be able to satisfy both conditions easily. Even though our model contains a t-distribution components, we can still use the SIN shown in that uses a GMM factor. We can therefore simplify inference by choosing an inference network which has a simpler form than the original model. In theory, one can do this even when all factors are non-conjugate, however, the approximation error might be quite large in some cases for this approximation to be useful. In our experiments, we tried this for non-linear dynamical system and found that capturing non-linearity was essential for dynamical systems that are extremely non-linear. Previously, we assumed θ PGM to be deterministic. In this section, we relax this condition and assume θ PGM to follow an exponential-family prior p(θ PGM |η PGM) with natural parameter η PGM. We derive a VMP algorithm to perform natural-gradient variational inference for θ PGM. Our algorithm works even when the PGM part contains non-conjugate factors, and it does not affect the efficiency of the amortized inference on the DNN part. We assume the following mean-field approximation: q(x, θ|y):= q(x|y, φ)q(θ PGM |λ PGM) where the first term is equal to SIN introduced in the previous section, and the second term is an exponential-family distribution with natural parameter λ PGM. For θ NN and φ, we will compute point estimates. We build upon the method of BID11 which is a generalization of VMP and stochastic variational inference (SVI). This method enables natural-gradient updates even when PGM contains non-conjugate factors. This method performs natural-gradient variational inference by using a mirror-descent update with the Kullback-Leibler (KL) divergence. To obtain natural-gradients with respect to the natural parameters of q, the mirror-descent needs to be performed in the mean parameter space. We will now derive a VMP algorithm using this method. We start by deriving the variational lower bound. The variational lower bound corresponding to the mean-field approximation can be expressed in terms of L SIN derived in the previous section. Compute q(x|y, φ) for SIN shown in either by using an exact expression or using VMP. DISPLAYFORM0 Sample x * ∼ q(x|y, φ), and compute ∇ φ Z and ∇ φ x *. Update λ PGM using the natural-gradient step given in. Update θ NN and φ using the gradients given in FORMULA8 - FORMULA0 with θ PGM ∼ q(θ PGM |λ PGM). 7: until ConvergenceWe will use a mirror-descent update with the KL divergence for q(θ PGM |λ PGM) because we want natural-gradient updates for it. For the rest of the parameters, we will use the usual Euclidean distance. We denote the mean parameter corresponding to λ PGM by µ PGM. Since q is a minimal exponential family, there is a one-to-one map between the mean and natural parameters, therefore we can reparameterize q such that q(θ PGM |λ PGM) = q(θ PGM |µ PGM). Denoting the values at iteration t with a superscript t and using Eq. 19 in BID11 with these divergences, we get: DISPLAYFORM0 DISPLAYFORM1 where β 1 to β 3 are scalars,, is an inner product, and ∇L t is the gradient at the value in iteration t. As shown by BID11, the maximization in FORMULA0 can be obtained in closed-form: DISPLAYFORM2 When the prior p(θ PGM |η PGM) is conjugate to p(x|θ PGM), the above step is equal to the SVI update of the global variables. The gradient itself is equal to the message received by θ PGM in a VMP algorithm, which is also the natural gradient with respect to λ PGM. When the prior is not conjugate, the gradient can be approximated either by using stochastic gradients or by using the reparameterization trick BID11. Therefore, this update enables natural-gradient update for PGMs that may contain both conjugate and non-conjugate factors. The update of the rest of the parameters can be done by using a stochastic-gradient method. This is because the solution of the update FORMULA0 is equal to a stochastic-gradient descent update (one can verify this by simplify taking the gradient and setting it to zero). We can compute the stochasticgradients by using a Monte Carlo estimate with a sample θ * DISPLAYFORM3 where θ *:= {θ * PGM, θ NN}. As discussed in the previous section, these gradients can be computed similar to VAE-like by using the gradients given in-. Therefore, for the DNN part we can perform amortized inference, and use a natural-gradient update for the PGM part using VMP.The final algorithm is outlined in Algorithm 1. Since our algorithm enables Structured, Amortized, and Natural-gradient (SAN) updates, we call it the SAN algorithm. Our updates conveniently separate the PGM and DNN computations. Step 3-6 operate on the PGM part, for which we can use existing implementation for the PGM.Step 7 operates on the DNN part, for which we can reuse VAE implementation. Our algorithm not only generalizes previous works, but also simplifies the implementation by enabling the reuse of the existing software. The main goal of our experiments is to show that our SAN algorithm gives similar to the method of BID10. For this reason, we apply our algorithm to the two examples considered in BID10, namely the latent GMM and latent LDS (see FIG0). In this section we discuss for latent GMM. An additional for LDS is included in Appendix C. Our show that, similar to the method of BID10 our algorithm can learn complex Even with 70% outliers, SAN-TMM performs better than SAN-GMM with 10% outliers. DISPLAYFORM0 Figure 3: Top row is for the Pinwheel dataset, while the bottom row is for the Auto dataset. Point clouds in the of each plot show the samples generated from the learned generative model, where each mixture component is shown with a different color and the color intensities are proportional to the probability of the mixture component. The points in the foreground show data samples which are colored according to the true labels. We use K = 10 mixture components to train all models. For the Auto dataset, we show only the first two principle components.representations with interpretable structures. The advantage of our method is that it is simpler and more general than the method of BID10.We compare to three baseline methods. The first method is the variational expectation-maximization (EM) algorithm applied to the standard Gaussian mixture model. We refer to this method as'GMM'. This method is a clustering method but does not use a DNN to do so. The second method is the VAE approach of BID12, which we refer to as'VAE'. This method uses a DNN but does not cluster the outputs or latent variables. The third method is the SVAE approach of BID10 applied to latent GMM shown in FIG0. This method uses both a DNN and a mixture model to cluster the latent variables. We refer to this as'SVAE'. We compare these methods to our SAN algorithm applied to latent GMM model. We refer to our method as'SAN'. All methods employ a Normal-Wishart prior over the GMM hyperparameters (see BID1 for details).We use two datasets. The first dataset is the synthetic two-dimensional Pinwheel dataset (N = 5000 and D = 2) used in BID10. The second dataset is the Auto dataset (N = 392 and D = 6, available in the UCI repository) which contains information about cars. The dataset also contains a five-class label which indicates the number of cylinders in a car. We use these labels to validate our . For both datasets we use 70% data for training and the rest for testing. For all methods, we tune the step-sizes, the number of mixture components, and the latent dimensionality on a validation set. We train the GMM baseline using a batch method, and, for VAE and SVAE, we use minibatches of size 64. DNNs in all models consist of two layers with 50 hidden units and an output layer of dimensionality 6 and 2 for the Auto and Pinwheel datasets, respectively. FIG1 and 2b compare the performances during training. In FIG1, we compare to SVAE and GMM, where we see that SAN converges faster than SVAE. As expected, both SVAE and SAN achieve similar performance upon convergence and perform better than GMM. In FIG1, we compare to VAE and GMM, and observe similar trends. The performance of GMM is represented as a constant because it converges after a few iterations already. We found that the implementation provided by BID10 does not perform well on the Auto dataset which is why we have not included it in the comparison. We also compared the test log-likelihoods and imputation error which show very similar trends. We omit these due to space constraints. In the of each plot in Figure 3, we show samples generated from the generative model. In the foreground, we show the data with the true labels. These labels were not used during training. The plots (a)- FORMULA29 show for the Pinwheel dataset, while plots FORMULA29 - FORMULA29 shows for the Auto dataset. For the Auto dataset, each label corresponds to the number of cylinders present in a car. We observe that SAN can learn meaningful clusters of the outputs. On the other hand, VAE does not have any mechanisms to cluster and, even though the generated samples match the data distribution, the are difficult to interpret. Finally, as expected, both SAN and VAE learn flexible patterns while GMM fails to do so. Therefore, SAN enables flexible models that are also easy to interpret. An advantage of our method over the method of BID10 is that our method applies even when PGM contains non-conjugate factors. Now, we discuss a for such a case. We consider the SIN for latent Student's t-mixture model (TMM) discussed in Section 3. The generative model contains the student's t-distribution as a non-conjugate factor, but our SIN replaces it with a Gaussian factor. When the data contains outliers, we expect the SIN for latent TMM to perform better than the SIN for latent GMM. To show this, we add artificial outliers to the Pinwheel dataset using a Gaussian distribution with a large variance. We fix the degree of freedom for the Student's t-distribution to 5. We test on four different levels of noise and report the test MSE averaged over three runs for each level. FIG1 shows a comparison of GMM, SAN on latent GMM, and SAN on latent TMM where we see that, as the noise level is increased, latent TMM's performance degrades slower than the other methods (note that the y-axis is in log-scale). Even with 70% of outliers, the latent TMM still performs better than the latent GMM with only 10% of outliers. This experiment illustrates that a conjugate SIN can be used for inference on a model with a non-conjugate factor. We propose an algorithm to simplify and generalize the algorithm of BID10 for models that contain both deep networks and graphical models. Our proposed VMP algorithm enables structured, amortized, and natural-gradient updates given that the structured inference networks satisfy two conditions. The two conditions derived in this paper generally hold for PGMs that do not force dense correlations in the latent variables x. However, it is not clear how to extend our method to models where this is the case, e.g., Gaussian process models. It is possible to use ideas from sparse Gaussian process models and we will investigate this in the future. An additional issue is that our are limited to small scale data. We found that it is non-trivial to implement a message-passing framework that goes well with the deep learning framework. We are going to pursue this direction in the future and investigate good platforms to integrate the capabilities of these two different flavors of algorithms. In SLDS, we introduce discrete variable z n ∈ {1, 2, . . ., K} that are sampled using a Markov chain: p(z n = i|z n−1 = j) = π ij such that π ij sum to 1 over all i given j. The transition for LDS is defined conditioned on z n: p(x n |x n−1, z n = i, θ PGM):= N (x n |A i x n−1, Q i) where A i and Q i are parameters for the i'th indicator. These two dynamics put together define the SLDS prior p(x, z|θ PGM). We can use the following SIN which uses the SLDS prior as the PGM factor but with parameters φ PGM instead of θ PGM. The expression for q(x, z|y, φ) is shown below: DISPLAYFORM0 Even though the above model is a conditionally-conjugate model, the partition function is not tractable and sampling is also not possible. However, we can use a structured mean-field approximation. First, we can combine the DNN factor with the Gaussian observation of SLDS factor and then use a mean-field approximation q(x, z|y, φ) ≈ q(x|λ x)q(z|λ z), e.g., using the method of BID5. This will give us a structured approximation where the edges between y n and x n and z n and z n−1 are maintained but x n and z n independent of each other. In this section we give detailed derivations for the SIN shown in. We derive the normalizing constant Z(φ) and show how to generate samples from SIN.We start by a simple rearrangement of SIN defined in: DISPLAYFORM0 DISPLAYFORM1 where the first step follows from the definition, the second step follows by taking the sum over k outside, and the third step is obtained by defining each component as a joint distribution over x n and the indicator variable z n.We will express this joint distribution as a multiplication of the marginal of z n and conditional of x n given z n. We will see that this will give us the expression for the normalizing constant, as well as a way to sample from SIN.We can simplify the joint distribution further as shown below. The first step follows from the definition. The second step is obtained by swapping m n and x n in the first term. The third step is obtained by completing the squares and expressing the first term as a distribution over x n (the second and third terms are independent of x n).q(x n, z n = k|y n, φ) ∝ N (x n |m n, V n)N (x n |μ k,Σ k)π k DISPLAYFORM2 where Σ Using the above we get the marginal of z n and conditional of x n given z n: DISPLAYFORM3 q(x n |z n = k, y n, φ):= N (x n | µ n, Σ n)The normalizing constant of the marginal of z n is obtained by simply summing over all k: DISPLAYFORM4 and since q(x n |z n = k, y n, φ) is already a normalized distribution, we can write the final expression for the SIN as follows: DISPLAYFORM5 q(x n |z n = k, y n, φ)q(z n = k|y n, φ)where components are defined in FORMULA1, FORMULA1, and. The normalizing constant is available in closed-form and we can sample z n first and then generate x n. This completes the derivation. In this experiment, we apply our SAN algorithm to the latent LDS discussed in Section 3. For comparison, we compare our method, Structured Variational Auto-Encoder (SVAE) BID10, and LDS on the Dot dataset used in BID10. Our show that our method achieves comparable performance to SVAE. For LDS, we perform batch learning for all model parameters using the EM algorithm. For SVAE and SAN, we perform mini-batch updates for all model parameters. We use the same neutral network architecture as in BID10, which contains two hidden layers with tanh activation function. We repeat our experiments 10 times and measure model performance in terms of the following mean absolute error for τ -steps ahead prediction. The error measures the absolute difference between the ground truth and the generative outputs by averaging across generated . 27) where N is the number of testing time series with T time steps, d is the dimensionality of observation y, and observation y * t+τ,n denotes the ground-truth at time step t + τ. From FIG4, we can observe that our method performs as good as SVAE and outperforms LDS. Our method is slightly robust than SVAE. In FIG5, there are generated images obtained from all methods. From FIG5, we also see that our method performs as good as SAVE and is able to recover the ground-truth observation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a variational message-passing algorithm for models that contain both the deep model and probabilistic graphical model.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the scientific article to find a short TLDR

Question: Modern deep neural networks have a large amount of weights, which make them difficult to deploy on computation constrained devices such as mobile phones. One common approach to reduce the model size and computational cost is to use low-rank factorization to approximate a weight matrix. However, performing standard low-rank factorization with a small rank can hurt the model expressiveness and significantly decrease the performance. In this work, we propose to use a mixture of multiple low-rank factorizations to model a large weight matrix, and the mixture coefficients are computed dynamically depending on its input. We demonstrate the effectiveness of the proposed approach on both language modeling and image classification tasks. Experiments show that our method not only improves the computation efficiency but also maintains (sometimes outperforms) its accuracy compared with the full-rank counterparts. Modern neural networks usually contain millions of parameters BID4 BID8, and they are difficult to be deployed on mobile devices with limited computation resources. To solve this problem, model compression techniques are proposed in recent years. Low-rank factorization is a popular way of reducing the matrix size. It has been extensively explored in the literature BID5 BID6 BID3 BID10. Mathematically, a large weight matrix W ∈ R m×n is factorized to two small rank-d matrices U ∈ R m×d, V ∈ R n×d with W = U V T. Since both U and V are dense, no sparsity support is required from specialized hardware. It naturally fits the general-purpose, off-the-shelf CPUs and GPUs. To significantly reduce the model size and computation, the rank d in the low-rank factorization needs to be small. However, a small rank can limit the expressiveness of the model BID9 and lead to worse performance. To understand the limitations, given a n-dim feature vector h, we observe that DISPLAYFORM0, is a linear projection from a high-dimensional space (n dims) to a low-dimensional space (d dims). This can lead to a significant loss of information. The conflict between the rank d and the model expressiveness prevents us from obtaining a both compact and accurate model. To address the dilemma, we propose to increase the expressiveness by learning an adaptive, inputdependent factorization, rather than performing a fixed factorization of a weight matrix. To do so, we use a mixture of multiple low-rank factorizations. The mixing weights are computed based on the input. This creates an adaptive linear projection from a high-dimensional space to a low-dimensional space. Compared to the conventional low-rank factorization, the proposed approach can significantly improve its performance while only introducing a small additional cost. DISPLAYFORM1 where z can be treated as the middle layer. Techniques like pooling can be applied to compute π to make it efficient. We propose to use an unnormalized learned mixture of low-rank factorizations whose mixing weights are computed adaptively based on the input. More specifically, denoting the input by h and the number of mixture components by K, we decompose a large weight matrix by DISPLAYFORM0 where π(·): R n → R K is the function which maps each input to its mixture coefficients. For example, π can be a small neural network. This introduces a small amount of extra parameters and computation. We will later discuss the details of efficient ways to implement the mixture function π. If π k, k = 1,..., K, is chosen to be constant (input independent), it can be absorbed into either DISPLAYFORM1. Thus, the proposed method reduces to the low-rank factorization. This is evidenced by rewriting DISPLAYFORM2. In other words, the conventional low-rank factorization can be considered as a special case of our method. FIG0 depicts the proposed framework. Adaptive mixing weights π(h). The mixing weights can encode important information that we can use to increase the expressiveness of the projected low-dimensional space. Under our framework, the generation of the mixing weights π(h) is flexible. A straight-forward approach is to use a non-linear transformation of the input to the weight matrix. For example, π(h) = σ(P h), where σ is a non-linear transformation, such as sigmoid or hyperbolic tangent function, and P ∈ R K×n is an extra weight matrix. This adds some extra parameters and computation to the model since the linear projection that we construct is R n → R K. To further reduce the parameter and computation in the mixing weights π, we propose the following strategies. Pooling before projection. We do not require the whole input to compute the mixture function π. Instead, we can apply pooling to the input h before projection. For example, a global average pooling can be applied if the input is a 3D tensor (for images); for a 1D vector, we can segment the vector and average each segmentations. By applying pooling, we can both save the computation and better capture the global information. Random projection. To reduce the number of parameters in the linear projection of h, we can use a random matrix P random in place of a fully adjustable P, i.e. π(h) = σ(P random h). Note that we can simply save a seed to recover the random matrix, but it still requires the same amount of memory and computation as the fully adjustable linear projection of h. Increased expressiveness. The adaptive mixing weights introduce a non-linear transformation into the high-to-low-dimensional projection that can be more expressive. Since each W (h) is a data-dependent low-rank matrix, there is no constant linear weight independent to the input (even a full-rank matrix) that can mimic the transformation W (h). Generating the whole weight matrices can be very expensive. Our method can be seen as a swift approach to generate the weights by adaptively adjusting mixing weights for the linear bottleneck. It assigns weights into groups and dynamically controls them at the group level. Recurrent neural networks for language modeling. Recurrent neural networks (RNNs) are widely used in language modeling, machine translation and sequence modeling in general. We adopt the same Long Short Term Memory (LSTM) models and follow the settings from a previous state-ofthe-art model BID11 for language modeling, and use Penn Tree Bank (PTB) as well as Text8 datasets. More specifically, we use the medium sized model introduced in BID11.We test three variants of the proposed model against regular low-rank factorization, each with different ways of computing mixing weights, namely MIX-ALL-PJ: direct linear projection of the input vector h, MIX-POOL-PJ: linear projection after segment-based mean pooling of the input vector h, and MIX-RND-PJ: use a random projection for the input vector h. Among these adaptive projection methods, MIX-ALL-PJ has a large amount of extra parameters, MIX-POOL-PJ has a small amount of extra parameters, and MIX-RND-PJ has no extra parameters. We compute the FLOPs of a single time-step of applying LSTM, and the perplexity associated to different settings. The are shown in FIG1. Firstly, with adaptive mixtures, the low-rank factorization model achieved 40% reduction in FLOPs, and even surpassed the performance of the full matrix baseline by decreasing the perplexity by 1.7 points. Secondly, the use of adaptive mixtures can significantly improve the performance compared with regular, non-adaptive low-rank factorization. Thirdly, using pooling before projection can be a good choice for computing the mixing weights π. It not only reduces the computation and parameter size, but can better capture the global information and achieve better accuracy. CNN for image recognition. We further demonstrate the effectiveness of the proposed approach on compressing CNN models on ImageNet BID1. We chose to use modern compact CNN models as the baseline (which are harder to compress), rather than using the bulky CNN models (which is easier to compress). Specifically, we choose to compress the point-wise convolution in depth-wise separable convolutions BID0, MobileNet BID2 BID7 in particular. TAB0 shows the comparison of different state-of-art compact convolutional models. We observed that compared to the regular low-rank factorization of MobileNet model (a.k.a. MobileNet V2), the proposed method achieves significantly better (2.5% and 2% for two different Low-rank MobileNet settings, respectively), while only adding negligible extra FLOPs (less than 1%).

A simple modification to low-rank factorization that improves performances (in both image and language tasks) while still being compact.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.


Question: Represent the article for finding a single-sentence TLDR

Question: Deep learning training accesses vast amounts of data at high velocity, posing challenges for datasets retrieved over commodity networks and storage devices. We introduce a way to dynamically reduce the overhead of fetching and transporting training data with a method we term Progressive Compressed Records (PCRs). PCRs deviate from previous formats by leveraging progressive compression to split each training example into multiple examples of increasingly higher fidelity, without adding to the total data size. Training examples of similar fidelity are grouped together, which reduces both the system overhead and data bandwidth needed to train a model. We show that models can be trained on aggressively compressed representations of the training data and still retain high accuracy, and that PCRs can enable a 2x speedup on average over baseline formats using JPEG compression. Our hold across deep learning architectures for a wide range of datasets: ImageNet, HAM10000, Stanford Cars, and CelebA-HQ. Distributed deep learning exploits parallelism to reduce training time, and consists of three key components: the data pipeline (storage), the forward/backward computation (compute), and the variable synchronization (network). A plethora of work has investigated scaling deep learning from a compute-or network-bound perspective (e.g., ; ; ; ; ; ; ; ; ; . However, little attention has been paid toward scaling the storage layer, where training starts and training data is sourced. Unfortunately, hardware trends point to an increasing divide between compute and networking or storage bandwidth (; ;). For example, the transportation of data for machine learning is a key factor in the design of modern data centers , which are expected to be serviced by slow, yet high capacity, storage media for the foreseeable future (; ;). This, combined with the memory wall-a lack of bandwidth between compute and memory-suggests that, while computation may be sufficient moving forward, the mechanisms for moving data to the compute may not (; ; ;). The storage pipeline is therefore a natural area to seek improvements in overall training times, which manifest from the storage medium, through the network, and into the compute nodes. In this work, we propose a novel on-disk format called Progressive Compressed Records (PCRs) as a way to reduce the bandwidth cost associated with training over massive datasets. Our approach leverages a compression technique that decomposes each data item into deltas, each of which increases data fidelity. PCRs utilize deltas to dynamically compress entire datasets at a fidelity suitable for each application's needs, avoiding duplicating the dataset (potentially many times) at various fidelity levels. Applications control the trade-off between dataset size (and, thus, bandwidth) and fidelity, and a careful layout of deltas ensures that data access is efficient at a storage medium level. As a , we find that for a variety of popular deep learning models and datasets, bandwidth (and therefore training time) can be easily reduced by 2× on average relative to JPEG compression without affecting model accuracy. Overall, we make the following contributions: 1. In experiments with multiple architectures and several large-scale image datasets, we show that neural network training is robust to data compression in terms of test accuracy and training loss; however, the amount of compression that can be tolerated varies across learning tasks. 2. We introduce Progressive Compressed Records (PCRs), a novel on-disk format for training data. PCRs combine progressive compression and careful data placement to enable applications to dynamically choose the fidelity of the dataset they consume, reducing data bandwidth. 3. We demonstrate that by using PCRs, training speed can be improved by 2× on average over standard formats using JPEG compression. This is achieved by selecting a lower data fidelity, which, in turn, reduces the amount of data read without significantly impairing model performance. Two complementary concepts make up the process of storing training data: the layout of the data on the storage medium and the representation of the data. Data layout is important because it can help fully utilize the bandwidth potential of the underlying storage system. Data representation is important because it can reduce the amount of data transferred per data unit (i.e., a bandwidth requirement reduction). An example of data representation within the scope of this work is compression, which increases the computation per bit-a key property to consider as computation increases faster than bandwidth to storage. Compression may lower image quality by introducing artifacts or blur. Figure 1: A hard disk with multiple data formats. Other storage media have the same space, bandwidth, and locality considerations. File-per-image formats have highly random behavior. Record formats encode many records at various data qualities to save bandwidth and have sequential behavior for a given fidelity. PCRs maintain the sequential behavior of record formats at multiple fidelities without space overheads. Record Layouts. Learning from data requires sampling points from a training set, which can cause small, random accesses that are detrimental to the performance of the storage device. Record layouts, such as TensorFlow's TFRecords (TFRecords) or MXNet's ImageRecord (ImageRecord), attempt to alleviate this problem by batching data points together to increase access locality. Batches of training data (i.e., dataset subsets) are then accessed together, amortizing delays in access time across multiple data points. These batches of data are called records. The key to any record layout is the serialization, which is the conversion of data structures into byte streams. Record designs have different performance properties (e.g., space or access time) when written to disk, as shown in Figure 1. Image Compression. Compressed forms are commonly used to represent training data. JPEG is one of the most popular formats for image compression and is used ubiquitously in machine learning (e.g., ; ; ;). Most compression formats (including JPEG) only allow for the compression level, i.e., the trade-off between data size and fidelity, to be set at encoding time, which often in choosing this level independent of the application. This can in over-compression, which may negatively impact application convergence quality, or under-compression, which in excess data size, and thus, slower storage system performance. Worse, deep learning pipelines often involve an application-defined post-processing step (e.g., data augmentation), which may further distort an image and obscure the relationship between image fidelity and model accuracy (; ; ;). While setting encoding-time parameters is unavoidable, the ability to decompress data as it becomes available (i.e., dynamic compression) provides a means to avoid some of the bandwidth expenses of under-compression by simply terminating decompression once sufficient fidelity is reached. In Figure 2, we provide a high-level illustration of the JPEG algorithm, which can be customized to support dynamic compression. First, an image is split into blocks of size 8 × 8. Each block is converted into the frequency domain, such that frequency 0 is the average color of the block, and higher frequencies encode rapid changes in the block. The low frequencies, such as the average value of the block, store the bulk of the perceptually-relevant content in the image (e.g., knowing the block is mostly blue is more important than knowing a white wave is rippling through it). Quantization, which discards information from the block and in compression, thus prioritizes discarding higher frequencies. The ing quantized table is then serialized into a flat form. Since data is rendered on a screen from left to right, top to bottom, it makes sense to encode the data in this manner, which in a sequential format 1. Decoding the ing data is simply a matter of inverting (albeit losslessly) the process that we just described. Figure 2: Top: JPEG carves an image into blocks, which are then converted into frequencies, quantized, and serialized. Progressive compression writes out a subset of important coefficients from each block before re-visiting the block. Bottom: Higher scans have diminishing fidelity returns. Progressive Image Compression. Progressive formats allow data to be read at varying degrees of compression without duplication. With the sequential case, data is ordered by blocks, and thus, partially reading the data in "holes" in the image for unread blocks . Dynamic compression ensures that all blocks get some information (deltas) before revising them (with more deltas). As progressive formats are simply a different traversal of the quantization matrix, with all else being equal, they contain the same information as sequential JPEG (JPEGTran LibJPEG). Progressive JPEG, combined with an additional rearrangement of data, forms the basis of the idea behind PCRs. In Figure 2, non-progressive formats serialize the image matrix in one pass, while progressive formats serialize the matrix in disjoint groups of deltas which are called scans. Scans are ordered by importance (e.g., the first few scans improve fidelity more than subsequent scans). Thus, any references to images generated from scan n will implicitly assume that the image decoder had access to all prior scans (i.e., {scan 1, scan 2, . . ., scan (n − 1)}). The bottom of Figure 2 shows how image fidelity improves from a single scan to utilizing all scans. In this section, we introduce a novel record format for machine learning training called Progressive Compressed Records (PCRs). PCRs are a combination of both layout and data representation. Efficient layouts guarantees that hardware is fully utilized (in terms of bandwidth), while efficient data representations can reduce the total amount of work that is required of the system. To this end, we introduce the concept of scan groups in Section 3.1, which leverage both layout and progressive compression to obtain dynamic compression, allowing both high performance reads while reducing the amount of data read. Using progressive compression, scan groups break images into deltas, which are then rearranged in order to facilitate reduced, yet sequential, data access. In Section 3.2, we discuss how PCRs are implemented, covering both creating PCRs (encoding) and reading them (decoding). The benefits of the PCR implementation boiling down to a bit shuffle are that: 1) PCRs are easy to implement, 2) they are fundamentally lossless, and 3) processing them is fast. As we demonstrate in Section 4, while PCRs can be implemented easily, they manifest in large speedups for a variety of scenarios. Further, PCRs can be generalized beyond images and JPEG. Scan groups are a collection of scans (deltas) of the same fidelity. Scan groups combine layout with progressive compression to allow reading subsets of the compressed data with high hardware efficiency. PCRs make the assumption that the entire training data will be read at the same fidelity. Using this assumption, scan groups rearrange the data such that all deltas of the same fidelity are grouped together. This, in turn, enables groups of deltas to be read together sequentially, which creates dynamicity in the decoding process. Since scans are sorted by importance, and scan groups are a set of scans, the scan groups are also sorted by importance. To paint a clear representation of how scan groups work, we point the reader to Figure 3. PCRs begin with some metadata which is assumed to be needed by all machine learning tasks, such as labels or bounding boxes. In practice, metadata is small in size, and, thus, the space overheads are negligible. The metadata is followed by scan groups, which consist of scans. The scan 1 representation of the shark in Figure 2 will be available in its record once data is read up to offset 1. Likewise, the scan 3 representation will be available once the record is read up to offset 3, and the representation will be more crisp as 3 scans were used per image, rather than 1. Reading up to the end of the record yields the most complete representation of the image. As scan groups consist of groups of the same fidelity, every image contained in a record is available at the same fidelity at the same group offset. Users of PCRs can read data at a certain scan fidelity by simply reading the on-disk byte stream from the start of the PCR (i.e., offset 0) to the byte offset corresponding to the corresponding scan group. Partially reading the records in bandwidth savings without re-encoding the data. There are two fundamental PCR implementation details: the encoding process and the decoding process. The encoding process transforms a set of JPEG files into a directory, which contains 1) a database for PCR metadata and 2) at least one.pcr file. The decoding process, which takes the directory as input and yields a set of JPEG images, efficiently inverts a subset of the encoding. The dataset is split into many PCRs, and, thus, the training process is reading tens to hundreds of.pcr files per epoch. The data loader is where the PCR decoding library interfaces with the inputs provided to deep learning libraries (e.g., TensorFlow , MXNet, PyTorch ). Below, we describe how each of these steps is done. Figure 3: PCRs encode label metadata followed by all scan groups. Accessing the dataset at a lower fidelity requires reading up to a certain address offset. Encoding. Given a set of images, the PCR encoder must break the images into scans, group the scans into scan groups, and sort the scan groups by fidelity. Once the groups are sorted, the PCR encoder can serialize the groups while taking note of their offsets (so that subsets may later be decoded). The metadata (e.g., labels) is prepended to the serialized representation, and the serialized representation is written to disk. We focus on grouping JPEG due to its generality, but PCRs can use any dataset-level progressive format. Images can be decomposed in both space and fidelity; other data modalities (e.g., video) may also have time. Our implementation uses JPEGTRAN (JPEGTran Man Page) to losslessly transform the set of JPEG images into a set of progressive JPEG images. With the default settings, each JPEG is broken up into 10 scans. The encoder scans the binary representation of the progressive JPEG files, searching for the markers that designate the end of a scan group. The encoder thus has access to all 10 offsets within the JPEG files that can be used to determine the boundaries between scan regions. Forming scan groups requires grouping the scan regions with the same fidelity together, which can be done in one pass over the set of images corresponding to that PCR. This grouping must be reversible, as the decoding process will un-group the scans to reconstruct the original images. This grouping can be done with existing serialization libraries. We use Protobuf (Protobuf) to serialize the groups as well as the labels. However, it is key that every group (and the metadata) be serialized as a separate message, as Protobuf can rearrange the contents within a message, and thus can rearrange the ordering of the groups themselves. We finally concatenate the contents of the messages and write them out as one file. As shown in Appendix A.5, any record format conversion can be expensive; PCRs benefit from requiring only a single conversion for multiple tasks. Decoding. To decode a PCR file, one has to first lookup the file's scan group offsets in the database. The offsets provide sufficient information to do a partial read of the file (e.g., instead of reading the entire file, we read only enough bytes to read up to the desired scan group). Decoding the JPEGs requires inverting the PCR scan-group grouping process for the available scan-groups prior to JPEG decode. Since we are missing scan-groups, we terminate the byte stream with an End-of-Image (EOI) JPEG token-this technique allows most JPEG decoders to render the byte stream with only the available subset of scans. The bulk of the inverse conversion is done in 150 lines of C++ code. Loader. We implemented PCR loaders using PyTorch's dataloader as well as DALI 's ExternalSource operator to return batches of images at a configurable fidelity (with the corresponding labels). We find that a pipeline abstraction simplifies loader design, since recordbased datasets can be easily iterated sequentially. In contrast, the PyTorch Dataloader abstrac-tion, which assumes that we can index randomly into an in-memory data structure (e.g., i = RandInt(0, n); (x, y) = data[i];), is harder to use for constantly fetching record formats off disk. Our implementation, while being only several hundred lines of code, obtains image rates that are competitive (e.g., faster/slower depending on number of scans) with the included DALI TFRecord loader, showing that PCRs can be implemented efficiently (i.e., fast enough to rarely bottleneck data loading) with a low amount of engineering effort. This section presents our evaluation of PCRs using a suite of large-scale image datasets. As large images are more taxing to a system's network and storage, our evaluation focuses on datasets with high-resolution images. We describe our experimental setup in Section 4.1. We present our evaluation in Section 4.2, showing that halving data bandwidth per image in comparable accuracy but with half the training time. In Section 4.3, we analyze the intuitive relationship between objective measures of image fidelity and time-to-accuracy. Finally, in Section 4.4, we present that trace the training time speedups to the data loading times themselves. Our evaluation uses the ImageNet ILSVRC , HAM10000 , Stanford Cars , and CelebA-HQ datasets, which are described below. See Appendix A.4 for additional details. • ImageNet-100 ImageNet provides a wide diversity of classes, of which we focus on the first 100 to make training times more tractable. Since classes are roughly ordered by ImageNet categories, this in a fine-grained, i.e., hard to classify, multiclass task. We convert the dataset into PCRs in batches of 1024, which in 126 PCRs. We use the full ImageNet dataset in Appendix A.7. • HAM10000 We split the HAM10000 dataset randomly 80%/20% between train and test. We convert the dataset into PCRs in batches of 64, which in 125 PCRs of similar size as the ones used for ImageNet-100. • Stanford Cars The Stanford Cars dataset is another fine-grained classification dataset, since all images are cars, and there are 196 classes spread over 16k images. We believe this dataset highlights some of the worst-case training scenarios, as it is considerably easier to predict highly compressed variants of unrelated images by exploiting low frequency image statistics (e.g., planes vs. frogs). We explore a coarse-grained version of Cars in Appendix A.6. Cars has 63 PCRs. • CelebAHQ-Smile CelebA-HQ is a high-resolution derivative of the CelebA dataset , which consists of 30k celebrity faces at 1024 2. We use the annotations provided by CelebA to construct a smiling or not smiling dataset. We split the 30k dataset into 80%/20% train/test, and we convert the training set into 93 PCRs. All datasets utilize resizing, crop, and horizontal-flip augmentations, as is standard for ImageNet training. We provide examples of scan groups for these datasets in Appendix A.8. Training Regime. We use pretrained ImageNet weights for HAM10000 and Cars due to the limited amount of training data. We use standard ImageNet training, starting the learning rate at 0.1 (with gradual warmup ) and dropping it on epoch 30 and 60 by 10×. After augmentations, all inputs are of size 224 × 224. The pretrained experiments (HAM10000 and Cars) start at a learning rate of 0.01 to avoid changing the initialization too aggressively. We use fp16 training as it in an additional 10% images per second (see Appendix A.3). We use a ResNet18 and ShuffleNetv2 architecture for our experiments with a batch size of 128 per each worker. We run each experiment at least 3 times to obtain confidence intervals given different random seeds and sources of non-determinism such as multi-threading and I/O., and one node is used as a Ceph metadata server (MDS). The remaining 10 nodes are used as machine learning workers for the training process. This means there is a 2:1 ratio between compute and storage nodes. We use PyTorch (v1.12) with NVIDIA Apex (Apex) (v0.1) and NVIDIA DALI (v0.14.0). We use at least four worker threads to prefetch data in the loader. While we focus on this particular distributed setting, we observe similar time-to-accuracy gains on a single machine with eight GPUs sharing the same disk, and we believe the will generalize to different setups. The time-to-accuracy for ResNet18 training are presented in Figure 4, while those of ShuffleNetv2 are presented in Figure 6. See Appendix A.2 for a tabular view and Appendix A.1 for the corresponding training loss . All scan groups within a dataset were run for the same amount of epochs, so lower scan groups finish earlier. 90 epochs are shown for ImageNet, 150 epochs are shown for HAM10000, 250 epochs are shown for Stanford Cars, and 90 epochs are shown for CelebAHQ-Smile. We sample the test accuracy every 15 epochs for non-ImageNet datasets to reduce interference with training measurements. To avoid measuring implementation differences with other loaders, our evaluation focuses on the differences obtained by reading various amounts of scan groups. Reading all the data (up to scan group 10) is the baseline. First, we note that for all datasets, except for Cars, PCRs provide a 2× boost to time-to-accuracy compared to the baseline. The reason for this speedup is that lower scan groups are smaller. As shown in Figure 5, scan group 5 is roughly half the size of the baseline, and scan group 1 is a fifth of scan group 5 (i.e., a potential 10× bandwidth savings). This trend holds across datasets (see Appendix A.1). As we will discuss in Section 4.4, the space savings manifest in reduced dataloader latencies. Second, we note that there is an inherent trade-off between convergence quality and the speedup attained by using less storage resources. In general, although lower fidelity scan groups allow the system to operate more efficiently, they do so at the expense of model convergence. Scan group 1, the lowest fidelity scan, performs poorly, especially on Cars, where fine-grained details are important. Scan groups limit the maximum achievable accuracy on a task; if learning plateaus prematurely, applications should raise the scan group in a manner similar to dropping the learning rate. Third, the relative rankings of scan groups are relatively stable across models and datasets, which reduces tuning efforts in choosing the appropriate scan group. We further relate these rankings to the fidelity of the scan groups in Section 4.3. Our is that, for most datasets, scan group 5 costs half as much in terms of bandwidth, but reaches the same level of test accuracy as the baseline-thus, it is a good default. This is most apparent for ImageNet and HAM10000, which are challenging enough for small variations in image fidelity to make a commensurate difference in test accuracy. In contrast, Cars is too fine-grained to allow images to be degraded, and CelebAHQ-Smile is too coarse-grained for image degradation to matter. We use MSSIM , a standard measure of image similarity, to compare how various scans approximate the reference image, and we show the in Figure 7. We find that there is a strong connection between MSSIM and the ing final test accuracy, especially when comparing scan groups within a task. Our preliminary tests demonstrate that scan groups that have very similar MSSIM perform very similarly, which is why only groups 1, 2, 5, and the baseline are shown. Due to the way progressive JPEG is coded by default, groups tend to cluster (e.g., 2, 3, and 4 are usually similar, while 5 introduces a difference). We note that MSSIM being poor (e.g., scan group 1 for cars) or MSSIM being close to baseline (e.g., scan group 5 for HAM10000) are good predictors of relative test accuracy within tasks. MSSIM can therefore be used as a diagnostic for choosing scans. The datasets we evaluated show that data loading can slow down the training process. To highlight these slowdowns, and the improvements PCRs achieve by not using all scan groups, we present the loading time of data for the ResNet18 ImageNet-100 run in Figure 8. We obtain similar for the other datasets. The baseline of using all scan group in high periodic loading stalls, where the prefetching queue was drained. Upon blocking, training cannot proceed until the worker threads obtain a full batch of data. Periods of (mostly) no stalls are caused by both threads pre-fetching the data and single records servicing multiple minibatches. Using fewer scan groups reduces the amount of data read, which in lower magnitude stalls. We observe these stalls with both DALI and PyTorch loaders. Training Over Large Datasets. Training with massive amounts of parallelism (thus stressing system bandwidth) while achieving near-linear speedup has been the focus of previous work, and it highlights a practical need for efficient data pipelines at scale. A common objective is training mod-els over ImageNet in a record amount of time (; ; ; ;). This line of work, while demonstrating immense bandwidth needs, typically keeps data in memory, avoiding storage problems altogether. Recently, the high performance computing community has become interested in training models at massive scale (27k GPUs) . Since each GPU matches a disk in bandwidth, the dataset was partitioned among the local memory/storage of the nodes, avoiding the distributed filesystem. Our work attempts to reduce the storage bottleneck altogether, such that anything from a couple disks to a distributed file system could service many GPUs. A separate line of work shows that I/O is a significant bottleneck for certain tasks and proposes optimizing I/O via a set of deep-learning specific optimization to LMDB . In contrast, our focus is more on data representation, which is independent of the internals of the storage system. Production systems such as TFX have used custom Protobuf parsers to get 2-5× speedups for simple (e.g., linear) models; these techniques are complementary to ours and reduce loader computational overheads. Dataset Reduction Techniques. The availability of larger datasets has spawned interest in learning algorithms that guaranteed both "good" model accuracy and lower computational complexity. Data reduction techniques, such as sketching, coresets, clustering, and sampling, have been used to reduce the size of a training set (; ; ; ; ; ;). A different approach is to use the unaltered training set, but reduce the size of the active training set to reduce bandwidth requirements . In contrast, we modify the data representation and layout to be more efficient across a wide variety of models. Compression. Finally, the reduction of data size via compression methods is ubiquitous across computer systems. To avoid costly model transmission/storage, prior work compressed neural network models (b; a; ; ; ;). Similarly, dataset distillation compresses a model's parameters into a few training examples. Our work attempts to compress data for training, and not the network itself. Prior work has looked into optimizing training systems by compressing neural network training network traffic (; ; ; ; . This trend is not specific to machine learning; prior work in databases, computer memories, and the web used compression to reduce system bandwidth requirements (; ;). Our work focuses on bandwidth for ML data pipelines by utilizing the compression robustness found in most models. Other work modifies models to be able to directly train on compressed representations for the purpose of avoiding decoding or reducing model complexity (; ; ;). Our work differs in motivation, as we do not focus on model computation or make modifications to the models. Previous work has investigated how image degradation (e.g., JPEG artifacts) affect inference (; ; ;); in contrast, our work focuses on the effects of compression on training. To continue making advances in machine learning, researchers will need access to larger and larger datasets, which will eventually spill into (potentially distributed) storage systems. Storage and networking bandwidth, which are precious resources, can be better utilized with efficient compression formats. We introduce a novel record format, Progressive Compressed Records (PCRs), that trades off data fidelity with storage and network demands, allowing the same model to be trained with 2× less storage bandwidth while retaining model accuracy. PCRs use progressive compression to split training examples into multiple examples of increasingly higher fidelity without the overheads of naive approaches. PCRs avoid duplicating space, are easy to implement, and can be applied to a broad range of tasks dynamically. While we apply our format in this work specifically to images with JPEG compression, PCRs are general enough to handle various data modalities or additional compression techniques; future work will include exploring these directions in fields outside of visual classification, such as audio generation or video segmentation. Below, we provide additional experiment plots that were omitted in the main text. Figure 9 and Figure 10 contain the loss over time for the ResNet-18 and ShuffleNetv2 experiments shown in Section 4. Figure 11 extends Figure 5 to show the scan sizes for all datasets. It is worth noting that Top-5 accuracies mirror the Top-1 accuracies trends for ImageNet and Cars. To measure the effect of compression without accounting for time, we show accuracy vs. epoch plots in Figure 12 and Figure 13. While compression can itself be viewed as a data augmentation (e.g., removing high frequency features that can possibly cause overfitting), we notice that it does not usually improve accuracy. Rather, most of the gains in time-to-accuracy are from faster image rates. The size in bytes of various levels of scans read. Scan group 0 is shown, which contains only labels and is typically ∼100 bytes. Each scan adds roughly a constant amount of data (i.e., linear scaling), although certain scans add considerably more than others (i.e., sizes sometimes cluster) due to techniques like chroma subsampling. Using all 10 scans can require over an order of magnitude more bandwidth than 1-2 scans. A.2 TIME TO CONVERGENCE TABLE We provide a table of time-to-accuracy in Table 1 This speed, combined with additional optimizations such as multi-core parallelism (e.g., we would expect 4× these rates with 4 cores), suggests that while decoding can be an issue, the penalty from using progressive images can be managed more easily than a storage bottleneck (i.e., compute can usually be traded for storage bandwidth). Further, some of the decoding can actually be moved to an accelerator, like the GPU used for training, something which is already available via nvJPEG 2. Reducing this computational expense by optimizing the implementation or reducing the amount of scans (since our experiments only use 4 distinct scans) is left as future work. Image Loading Rates. We provide image loading rates observed during training in Table 2. Using more scans slows down training significantly, as can be seen in the image rates. It is worth noting that these rates vary considerably during runtime (due to stalls), and ShuffleNetv2 is capable of a higher maximum training rate than ResNet-18. Further, as the number of scans is reduced, image rates approach the maximum achievable by the cluster for each model. Below we describe the characteristics of the used datasets. ImageNet-100 Creation. The ImageNet-100 dataset was constructed by subsampling 100 classes out of the 1000 classes found in the ImageNet ILSVRC dataset . These classes were chosen arbitrarily to limit computation time-they are the first 100 classes of ImageNet in sorted directory listing form i.e., n01440764-n01855672. CelebAHQ-Smile Creation. The CelebAHQ dataset was created as a high quality version of the CelebA dataset . CelebA contains attributes for each face, such as whether the face is smiling or not. CelebAHQ-Smile utilizes these attributes to construct a dataset of 30k faces, where each face is assigned a binary variable for smiling or not. While the CelebA dataset was subsampled to construct CelebAHQ, we do not subsample CelebAHQ further (i.e., we use all 30k images it contains). Record and Image Quality Details. We provide the dataset size details for the encoded datasets in Table 3. As the original (e.g., lossless) images are hard to find, we estimate the JPEQ qual-ity setting of the training set with ImageMagick using identify -format'%Q'. The JPEG quality setting determines the level of frequency quantization outlined in Figure 2. Intuitively, one would expect that higher quality JPEG images could allow more aggressive PCR compression rates for a fixed resolution, since each image has more redundant information on average. ImageNet and HAM10000 both have high quality images. CelebAHQ has lower quality images, but they are downscaled to 256 × 256 for training purposes, which increases the information density in the image (e.g., blurry images can be made to appear less blurry by downsampling), a fact exploited in prior work. Cars is neither high JPEG quality or large resolution. Under-compressing images (perhaps at high resolution) during the initial JPEG compression may allow for a larger range of viable scan groups. We provide bandwidth-optimized record baselines in Figure 14, where we re-encode the images using a statically-chosen level of compression. These baselines re-encode the images with 50% quality and 90% JPEG quality, respectively, to reduce dataset size at a fixed level of fidelity. It is worth noting that re-encoding images compounds with the original JPEG compression, so the re-encoded image quality may be lower than 50% or 90% quality compared to the images in their original lossless form. This is in contrast to PCRs, which losslessly convert the images into a progressive format, which allows dynamic access to the level of fidelity. We observe that both the baseline method of dataset bandwidth reduction and the PCR method can take considerable encoding time, since the encoding time scales proportionally to the dataset size. We also observe that the PCR method is competitive (1.15× to 2.98×) to that of the baseline in terms of encoding time. PCRs avoid having to re-encode a dataset at multiple fidelity levels, and, therefore, they can save both storage space and encoding time. Converting the full ImageNet into record format takes roughly 16× longer than the 6 minutes needed for the 10× smaller subsampled dataset-the PCR conversion is 96 minutes (53 minutes are spent in JPEG conversion). One reason for this additional slowdown is that any system caches (e.g., in the distributed filesystem or the file cache on the converter node) are less likely to see a cache hit due to the working set size being larger. Although the exact conversion times are dependent on implementation, hardware, and the dataset, conversions times can be in the range of one hour of compute time per 100 GB. We provide experiments validating that compression needs vary within the same dataset for different tasks in Figure 15 and Figure 16, which show accuracy and loss, respectively. This experiment simply coarsens the granularity of the classification task, and demonstrates that lower scan groups can be used for tasks which are easier. The full range of classes is used for Baseline (i.e, car make, model, and year create a unique class), only car make is used for Make-Only, and a binary classification task of Corvette detection is used for Is-Corvette. We can see that compared to the original task, the coarse tasks reduce the gap between scan groups, and the binary task closes the gap even more. This suggests that as the tasks get easier, the tolerance for lower scan groups grows. Simply re-assigning the class labels to a coarser class reduces the complexity of the task and closes the accuracy gap across scan groups. A fixed PCR record encoding (i.e., without re-encoding) can support multiple tasks at the optimal quality, whereas static approaches may need one encoding per task. Some training methodologies, such as Progressive GAN training , utilize

Represent the TLDR for finding the corresponding scientific article

We propose a simple, general, and space-efficient data format to accelerate deep learning training by allowing sample fidelity to be dynamically selected at training time

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.