Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Ability to quantify and predict progression of a disease is fundamental for selecting an appropriate treatment. Many clinical metrics cannot be acquired frequently either because of their cost (e.g. MRI, gait analysis) or because they are inconvenient or harmful to a patient (e.g. biopsy, x-ray). In such scenarios, in order to estimate individual trajectories of disease progression, it is advantageous to leverage similarities between patients, i.e. the covariance of trajectories, and find a latent representation of progression. Most of existing methods for estimating trajectories do not account for events in-between observations, what dramatically decreases their adequacy for clinical practice. In this study, we develop a machine learning framework named Coordinatewise-Soft-Impute (CSI) for analyzing disease progression from sparse observations in the presence of confounding events. CSI is guaranteed to converge to the global minimum of the corresponding optimization problem. Experimental also demonstrates the effectiveness of CSI using both simulated and real dataset. The course of disease progression in individual patients is one of the biggest uncertainties in medical practice. In an ideal world, accurate, continuous assessment of a patient's condition helps with prevention and treatment. However, many medical tests are either harmful or inconvenient to perform frequently, and practitioners have to infer the development of disease from sparse, noisy observations. In its simplest form, the problem of modeling disease progressions is to fit the curve of y(t), t ∈ [t min, t max] for each patient, given sparse observations y:= (ỹ(t 1),...,ỹ(t n)). Due to the highdimensional nature of longitudinal data, existing usually restrict solutions to subspace of functions and utilize similarities between patients via enforcing low-rank structures. One popular approach is the mixed effect models, including Gaussian process approaches and functional principal components . While generative models are commonly used and have nice theoretical properties, their could be sensitive to the underlying distributional assumptions of observed data and hard to adapt to different applications. Another line of research is to pose the problem of disease progression estimation as an optimization problem. Kidzinski and Hastie. Kidziński & proposed a framework which formulates the problem as a matrix completion problem and solve it using matrix factorization techniques. This method is distribution-free and flexible to possible extensions. Meanwhile, both types of solutions model the natural progression of disease using observations of the targeted variables only. They fail to incorporate the existence and effect of human interference: medications, therapies, surgeries, etc. Two patients with similar symptoms initially may have different futures if they choose different treatments. Without that information, predictions can be way-off. To the best of our knowledge, existing literature talks little about modeling treatment effect on disease progression. In Kidziński & , authors use concurrent observations of auxillary variables (e.g. oxygen consumption to motor functions) to help estimate the target one, under the assumption that both variables reflect the intrinsic latent feature of the disease and are thus correlated. Treatments of various types, however, rely on human decisions and to some extent, an exogenous variable to the development of disease. Thus they need to modeled differently. In this work, we propose a model for tracking disease progression that includes the effects of treatments. We introduce the Coordinatewise-Soft-Impute (CSI) algorithm for fitting the model and investigate its theoretical and practical properties. The contribution of our work is threefold: First, we propose a model and an algorithm CSI, to estimate the progression of disease which incorporates the effect of treatment events. The framework is flexible, distribution-free, simple to implement and generalizable. Second, we prove that CSI converges to the global solution regardless of the initialization. Third, we compare the performance of CSI with various other existing methods on both simulated data and a dataset of Gillette Children's Hospital with patients diagnosed with Cerebral Palsy, and demonstrate the superior performances of CSI. The rest of the paper is organized as follows. In Section 2 we state the problem and review existing methods. Next, in Section 3 we describe the model and the algorithm. Theoretic properties of the algorithm are derived in Section 4. Finally, in Section 5 and 6 we provides empirical of CSI on the simulated and the real datesets respectively. We discuss some future directions in Section 7. Let y(t) be the trajectory of our objective variable, such as the size of tumor, over fixed time range t ∈ [t min, t max], and N be the number of patients. For each patient 1 ≤ i ≤ N, we measure its trajectory y i (t) at n i irregularly time points t i = [t i,1, t i,2, ..., t i,ni] and denote the as. We are primarily interested in estimating the disease progression trajectories To fit a continuous curve based on discrete observations, we restrict our estimations to a finitedimensional space of functions. Let {b i, i ∈ N} be a fixed basis of L 2 ([t min, t max]) (e.g. splines, Fourier basis) and b = {b i : 1 ≤ i ≤ K} be first K dimensions of it. The problem of estimating y i (t) can then be reduced to the problem of estimating the coefficients Though intuitive, the above method has two main drawbacks. First, when the number of observations per patient is less than or equal to the number of basis functions K, we can perfectly fit any curve without error, leading to overfitting. Moreover, this direct approach ignores the similarities between curves. Different patients may share similar trend of the trajectories which could potentially imporve the prediction. Below we describe two main lines of research improving on this, the mixed-effect model and the matrix completion model. In mixed-effect models, every trajectory y i (t) is assumed to be composed of two parts: the fixed effect µ(t) = m b(t) for some m ∈ R K that remains the same among all patients and a random effect w i ∈ R K that differs for each i ∈ {1, . . ., N}. In its simplest form, we assume where Σ is the K × K covariance matrix, σ is the standard deviation and are functions µ(t) and b(t) evaluated at the times t i, respectively. Estimations of model parameters µ, Σ can be made via expectation maximization (EM) algorithm . Individual coefficients w i can be estimated using the best unbiased linear predictor (BLUP) . In linear mixed-effect model, each trajectory is estimated with |w i | = K degrees of freedom, which can still be too complex when observations are sparse. One typical solution is to assume a low-rank structure of the covariance matrix Σ by introducing a contraction mapping A from the functional basis to a low-dimensional latent space. More specifically, one may rewrite the LMM model as where A is a K × q matrix with q < K andw i ∈ R q is the new, shorter random effect to be estimated. Methods based on low-rank approximations are widely adopted and applied in practice and different algorithms on fitting the model have been proposed (; ;). In the later sections, we will compare our algorithm with one specific implementation named functional-Principle-Component-Analysis (fPCA) , which uses EM algorithm for estimating model parameters and latent variables w i. While the probabilistic approach of mixed-effect models offers many theoretical advantages including convergence rates and inference testing, it is often sensitive to the assumptions on distributions, some of which are hard to verify in practice. To avoid the potential bias of distributional assumptions in mixed-effect models, Kidzinski and Hastie (Kidziński &) formulate the problem as a sparse matrix completion problem. We will review this approach in the current section. To reduce the continuous-time trajectories into matrices, we discretize the time range T ×K be the projection of the K-truncated basis b onto grid G. by rounding the time t i,j of every observation y i (t i,j) to the nearest time grid and regarding all other entries as missing values. Due to sparsity, we assume that no two observation y i (t i,j)'s are mapped to the same entry of Y. Let Ω denote the set of all observed entries of Y. For any matrix A, let P Ω (A) be the projection of A onto Ω, i.e. P Ω (A) = M where M i,j = A i,j for (i, j) ∈ Ω and M i,j = 0 otherwise. Similarly, we define P ⊥ Ω (A) = A − P Ω (A) to be the projection on the complement of Ω. Under this setting, the trajectory prediction problem is reduced to the problem of fitting a N × K matrix W such that W B ≈ Y on observed indices Ω. The direct way of estimating W is to solve the optimization problem where · F is the Fröbenius norm. Again, if K is larger than the number of observations for some subject we will overfit. To avoid this problem we need some additional constraints on W. A typical approach in the matrix completion community is to introduce a nuclear norm penalty-a relaxed version of the rank penalty while preserving convexity (; Candès &). The optimization problem with the nuclear norm penalty takes form where λ > 0 is the regularization parameter, · F is the Fröbenius norm, and · * is the nuclear norm, i.e. the sum of singular values. In Kidziński & , a Soft-Longitudinal-Impute (SLI) algorithm is proposed to solve (2.2) efficiently. We refer the readers to Kidziński & for detailed description of SLI while noting that it is also a special case of our algorithm 1 defined in the next section with µ fixed to be 0. In this section, we introduce our model on effect of treatments in disease progression. A wide variety of treatments with different effects and durations exist in medical practice and it is impossible to build a single model to encompass them all. In this study we take the simplified approach and regard treatment, with the example of one-time surgery in mind, as a non-recurring event with an additive effect on the targeted variable afterward. Due to the flexibility of formulation of optimization problem (2.1), we build our model based on matrix completion framework of Section 2.2. More specifically, let s(i) ∈ G be the time of treatment of the i'th patient, rounded to the closest τ k ∈ G (s(i) = ∞ if no treatment is performed). We encode the treatment information as a N × T zero-one matrix I S, where (I S) i,j = 1 if and only τ j ≥ s(i), i.e. patient i has already taken the treatment by time τ j. Each row of I S takes the form of (0, · · ·, 0, 1, · · ·, 1). Let µ denote the average additive effect of treatment among all patients. In practice, we have access to the sparse observation matrix Y and surgery matrix I S and aim to estimate the treatment effect µ and individual coefficient matrix W based on Y, I S and the fixed basis matrix B such that W B + µI S ≈ Y. Again, to avoid overfitting and exploit the similarities between individuals, we add a penalty term on the nuclear norm of W. The optimization problem is thus expressed as: for some λ > 0. Though the optimization problem (3.1) above does not admit an explicit analytical solution, it is not hard to solve for one of µ or W given the other one. For fixed µ, the problem reduces to the optimization problem (2.2) withỸ = Y − µI S and can be solved iteratively by the SLI algorithm Kidziński & , which we will also specify later in Algorithm 1. For fixed W, we have arg min where Ω S is the set of non-zero indices of I S. Optimization problem (3.2) can be solved by taking derivative with respect to µ directly, which yieldŝ The clean formulation of (3.3) motivates us to the following Coordinatewise-Soft-Impute (CSI) algorithm (Algorithm In the definition, we define operator S λ as for any matrix X,). Note that if we set µ ≡ 0 throughout the updates, then we get back to our base model SLI without treatment effect. In this section we study the convergence properties of Algorithm 1. Fix the regularization parameter λ ) be the value of (µ, W) in the k'th iteration of the algorithm, the exact definition of which is provided below in (4.4). We prove that Algorithm 1 reduces the loss function at each iteration and eventually converges to the global minimizer. λ ) converges to a limit point (μ λ,Ŵ λ) which solves the optimization problem: Moreover, (μ λ,Ŵ λ) satisfies that The proof of Theorem 1 relies on five technique Lemmas stated below. The detailed proofs of the lemmas and the proof to Theorem 1 are provided in Appendix A. The first two lemmas are on properties of the nuclear norm shrinkage operator S λ defined in Section 3.1. Lemma 1. Let W be an N × K matrix and B is an orthogonal T × K matrix of rank K. The solution to the optimization problem min W is defined in Section 3.1. Lemma 2. Operator S λ (·) satisfies the following inequality for any two matrices W 1, W 2 with matching dimensions: Lemma 1 shows that in the k-th step of Algorithm 1, is the minimizer for function The next lemma proves the sequence of loss functions is monotonically decreasing at each iteration. Lemma 3. For every fixed λ ≥ 0, the k'th step of the algorithm (µ Then with any starting point (µ The next lemma proves that differences F both converge to 0. Lemma 4. For any positive integer k, we have W Finally we show that if the sequence {(µ λ)} k, it has to converge to a solution of (4.1). In this section we illustrate properties of our Coordinatewise-Soft-Impute (CSI) algorithm via simulation study. The simulated data are generated from a mixed-effect model with low-rank covariance structure on W: for which the specific construction is deferred to Appendix B. Below we discuss the evaluation methods as well as the from simulation study. We compare the Coordinatewise-Soft-Impute (CSI) algorithm specified in Algorithm 1 with the vanilla algorithm SLI (corresponding toμ = 0 in our notation) defined in Kidziński & and the fPCA algorithm defined in based on mixed-effect model. We train all three algorithms on the same set of basis functions and choose the tuning parameters λ (for CSI and SLI) and R (for fPCA) using a 5-fold cross-validation. Each model is then re-trained using the whole training set and tested on a held-out test set Ω test consisting 10% of all data. The performance is evaluated in two aspects. First, for different combinations of the treatment effect µ and observation density ρ, we train each of the three algorithms on the simulated data set, and compute the relative squared error between the ground truth µ and estimationμ., i.e., RSE(μ) = (μ − µ) 2 /µ 2. Meanwhile, for different algorithms applied to the same data set, we compare the mean square error between observation Y and estimationŶ over test set Ω test, namely, We train our algorithms with all combinations of treatment effect µ ∈ {0, 0.2, 0.4, · · ·, 5}, observation rate ρ ∈ {0.1, 0.3, 0.5}, and thresholding parameter λ ∈ {0, 1, · · ·, 4} (for CSI or SLI) or rank R ∈ {2, 3, · · ·, 6} (for fPCA). For each fixed combination of parameters, we implemented each algorithm 10 times and average the test error. The are presented in Table 1 and Figure 1. From Table 1 and the left plot of Figure 1, we have the following findings: 1. CSI achieves better performance than SLI and fPCA, regardless of the treatment effect µ and observation rate ρ. Meanwhile SLI performs better than fPCA. 2. All three methods give comparable errors for smaller values of µ. In particular, our introduction of treatment effect µ does not over-fit the model in the case of µ = 0. 3. As the treatment effect µ increases, the performance of CSI remains the same whereas the performances of SLI and fPCA deteriorate rapidly. As a , CSI outperforms SLI and fPCA by a significant margin for large values of µ. For example, when ρ = 0.1, the MSE(Ŷ) of CSI decreases from 72.3% of SLI and 59.6% of fPCA at µ = 1 to 12.4% of SLI and 5.8% of fPCA at µ = 5. 4. All three algorithms suffer a higher MSE(Ŷ) with smaller observation rate ρ. The biggest decay comes from SLI with an average 118% increase in test error from ρ = 0.5 to ρ = 0.1. The performances of fPCA and CSI remains comparatively stable among different observation rate with a 6% and 12% increase respectively. This implies that our algorithm is tolerant to low observation rate. To further investigate CSI's ability to estimate µ, we plot the relative squared error ofμ using CSI with different observation rate in the right plot of Figure 1. As shown in Figure 1, regardless of the choice of observation rate ρ and treatment effect µ, RSE(μ) is always smaller than 1% and most of the estimations achieves error less than 0.1%. Therefore we could conclude that, even for sparse matrix Y, the CSI algorithm could still give very accurate estimate of the treatment effect µ. In this section, we apply our methods to real dataset on the progression of motor impairment and gait pathology among children with Cerebral Palsy (CP) and evaluate the effect of orthopaedic surgeries. Cerebral palsy is a group of permanent movement disorders that appear in early childhood. Orthopaedic surgery plays a major role in minimizing gait impairments related to CP . However, it could be hard to correctly evaluate the outcome of a surgery. For example, the seemingly positive outcome of a surgery may actually due to the natural improvement during puberty. Our objective is to single out the effect of surgeries from the natural progression of disease and use that extra piece of information for better predictions. We analyze a data set of Gillette Children's Hospital patients, visiting the clinic between 1994 and 2014, age ranging between 4 and 19 years, mostly diagnosed with Cerebral Palsy. The data set contains 84 visits of 36 patients without gait disorders and 6066 visits of 2898 patients with gait pathologies. Gait Deviation Index (GDI), one of the most commonly adopted metrics for gait functionalities , was measured and recorded at each clinic visit along with other data such as birthday, subtype of CP, date and type of previous surgery and other medical . Our main objective is to model individual disease progression quantified as GDI values. Due to insufficiency of data, we model surgeries of different types and multiple surgeries as a single additive effect on GDI measurements following the methodology from Section 3. We test the same three methods CSI, SLI and fPCA as in Section 5, and compare them to two benchmarks-the population mean of all patients (pMean) and the average GDI from previous visits of the same patient (rMean). All three algorithms was trained on the spline basis of K = 9 dimensions evaluated at a grid of T = 51 points, with regularization parameters λ ∈ {20, 25, ..., 40} for CSI and SLI and rank constraints r ∈ {2, . . ., 6} for fPCA. To ensure sufficient observations for training, we cross validate and test our models on patients with at least 4 visits and use the rest of the data as a common training set. The effective size of 2-fold validation sets and test set are 5% each. We compare the of each method/combination of parameters using the mean square error of GDI estimations on held-out entries as defined in (5.1). We run all five methods on the same training/validation/test set for 40 times and compare the mean and sd of test-errors. The are presented in Table 2 and Figure 2. Compared with the null model pMean (Column 2 of Table 2), fPCA gives roughly the same order of error; CSI, SLI and rowMean provide better predictions, achieving 62%, 66% and 73% of the test errors respectively. In particular, our algorithm CSI improves the of vanilla model SLI by 7%, it also provide a stable estimation with the smallest sd across multiple selections of test sets. We take a closer look at the low-rank decomposition of disease progression curves provided by algorithms. Fix one run of algorithm CSI with λ = 30, there are 6 non-zero singular value vectors, which we will refer as principal components. We illustrate the top 3 PCs scaled with corresponding singular values in Figure 3a. An example of predicted curve from patient ID 5416 is illustrated in Figure 3b, where the blue curve represents the prediction without estimated treatment effectμ = 4.33, green curve the final prediction and red dots actual observations. It can be seen that the additive treatment effect helps to model the sharp difference between the exam before exam (first observation) and later exams. In this paper, we propose a new framework in modeling the effect of treatment events in disease progression and prove a corresponding algorithm CSI. To the best of our knowledge, it's the first comprehensive model that explicitly incorporates the effect of treatment events. We would also like to mention that, although we focus on the case of disease progression in this paper, our framework is quite general and can be used to analyze data in any disciplines with sparse observations as well as external effects. There are several potential extensions to our current framework. Firstly, our framework could be extended to more complicated settings. In our model, treatments have been characterized as the binary matrix I S with a single parameter µ. In practice, each individual may take different types of surgeries for one or multiple times. Secondly, the treatment effect may be correlated with the latent variables of disease type, and can be estimated together with the random effect w i. Finally, our framework could be used to evaluate the true effect of a surgery. A natural question is: does surgery really help? CSI provides estimate of the surgery effect µ, it would be interesting to design certain statistical hypothesis testing/casual inference procedure to answer the proposed question. Though we are convinced that our work will not be the last word in estimating the disease progression, we hope our idea is useful for further research and we hope the readers could help to take it further. Proof of Lemma 1. Note that the solution of the optimization problem is given byÂ = S λ (Z) (see for a proof). Therefore it suffices to show the minimizer of the optimization problem (A.1) is the same as the minimizer of the following problem: Using the fact that A 2 F = Tr(AA) and B B = I K, we have On the other hand as desired. Proof of Lemma 2. We refer the readers to the proof in Mazumder et al. (2010, Section 4, Lemma 3). Proof of Lemma 3. First we argue that µ λ, µ) and the first inequality immediately follows. We have Taking derivative with respect to µ directly gives µ λ, µ), as desired. For the rest two inequalities, notice that Here the (A.2) holds because we have (A.3) follows from the fact that W Proof of Lemma 4. First we analyze the behavior of {µ Meanwhile, the sequence is decreasing and lower bounded by 0 and therefore converge to a non-negative number, yielding the differences as desired. The sequence {W (k) λ } is slightly more complicated, direct calculation gives 2 F, (A.6) where (A.5) follows from Lemma 2, (A.6) can be derived pairing the 4 terms according to P Ω and P ⊥ Ω. By definition of µ where (A.7) follows from the Cauchy-Schwartz inequality. Combining (A.6) with (A.7), we get Now we are left to prove that the difference sequence {W} converges to zero. Combining (A.4) and (A.7) it suffices to prove that P and the left hand side converges to 0 because which completes the proof. Taking limits on both sides gives us the desire . Proof of Theorem 1. Let (μ λ,Ŵ λ) be one limit point then we have: here (A.8) uses Lemma 5 and (A.9) uses Lemma 2. guarantees 0 ∈ ∂ W f λ (Ŵ λ,μ). By taking derivative directly we have 0 = ∂ µ f λ (Ŵ λ,μ). Therefore (Ŵ λ,μ) is a stationary point for f λ (W, µ). Notice that the loss function f λ (W, µ) is a convex function with respect to (W, µ). Thus we have proved that the limit point (Ŵ λ,μ) minimizes the function f λ (W, µ). Let G be the grid of T equidistributed points and let B be the basis of K spline functions evaluated on grid G. We will simulate the N × K observation matrix Y with three parts Y = W B + µI S + E, where W follows a mixture-Gaussian distribution with low rank structure, I S is the treatment matrix with uniformly distributed starting time and E represents the i.i.d. measurement error. The specific procedures is described below. 1. Generating W given parameters κ ∈, r 1, r 2 ∈ R, s 1, s 2 ∈ R K ≥0: (a) Sample two K × K orthogonal matrices V 1, V 2 via singular-value-decomposing two random matrix. where diag[s] is the diagonal matrix with diagonal elements s, "·" represents coordinatewise multiplication, and we are recycling t, 1 − t and r i γ i to match the dimension. 2. Generating I S given parameter p tr ∈. (a) For each k = 1,..., N, sample T k uniformly at random from {1, . . ., T /p tr}. (b) Set I S ← (1{j ≥ T i}) 1≤i≤N,1≤j≤T. 3. Given parameter ∈ R ≥0, E is drawn from from i.i.d. Normal samples. 4. Given parameter µ ∈ R, let Y 0 ← W B + µI S + E. 5. Given parameter ρ ∈, drawn 0-1 matrix I Ω from i.i.d. Bernoulli(ρ) samples. Let Ω denote the set of non-zero entries of I Ω, namely, the set of observed data. Set, where Y ij = (Y 0) ij if (I Ω) ij = 1 NA otherwise. In actual simulation, we fix the auxiliary parameters as follows, The remaining parameters are treatment effect µ and observation rate ρ, which we allow to vary across different trials.

Represent the TLDR for finding the corresponding scientific article

A novel matrix completion based algorithm to model disease progression with events

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question:
represent the scientific article to find a short tldr.
------
For instance you may be given 'Recovering sparse conditional independence graphs from data is a fundamental problem in machine learning with wide applications. A popular formulation of the problem is an $\ell_1$ regularized maximum likelihood estimation. Many convex optimization algorithms have been designed to solve this formulation to recover the graph structure. Recently, there is a surge of interest to learn algorithms directly based on data, and in this case, learn to map empirical covariance to the sparse precision matrix. However, it is a challenging task in this case, since the symmetric positive definiteness (SPD) and sparsity of the matrix are not easy to enforce in learned algorithms, and a direct mapping from data to precision matrix may contain many parameters. We propose a deep learning architecture, GLAD, which uses an Alternating Minimization (AM) algorithm as our model inductive bias, and learns the model parameters via supervised learning. We show that GLAD learns a very compact and effective model for recovering sparse graphs from data. Recovering sparse conditional independence graphs from data is a fundamental problem in high dimensional statistics and time series analysis, and it has found applications in diverse areas. In computational biology, a sparse graph structure between gene expression data may be used to understand gene regulatory networks; in finance, a sparse graph structure between financial timeseries may be used to understand the relationship between different financial assets. A popular formulation of the problem is an 1 regularization log-determinant estimation of the precision matrix. Based on this convex formulation, many algorithms have been designed to solve this problem efficiently, and one can formally prove that under a list of conditions, the solution of the optimization problem is guaranteed to recover the graph structure with high probability. However, convex optimization based approaches have their own limitations. The hyperparameters, such as the regularization parameters and learning rate, may depend on unknown constants, and need to be tuned carefully to achieve the recovery . Furthermore, the formulation uses a single regularization parameter for all entries in the precision matrix, which may not be optimal. It is intuitive that one may obtain better recovery by allowing the regularization parameters to vary across the entries in the precision matrix. However, such flexibility will lead to a quadratic increase in the number of hyperparameters, but it is hard for traditional approaches to search over a large number of hyperparameters. Thus, a new paradigm may be needed for designing more effective sparse recovery algorithms. Recently, there has been a surge of interest in a new paradigm of algorithm design, where algorithms are augmented with learning modules trained directly with data, rather than prescribing every step of the algorithms. This is meaningful because very often a family of optimization problems needs to be solved again and again, similar in structures but different in data. A data-driven algorithm may be able to leverage this distribution of problem instances, and learn an algorithm which performs better than traditional convex formulation. In our case, the sparse graph recovery problem may also need to be solved again and again, where the underlying graphs are different but have similar degree distribution, the magnitude of the precision matrix entries, etc. For instance, gene regulatory networks may be rewiring depending on the time and conditions, and we want to estimate them from gene In our experiments, we show that the AM architecture provides very good inductive bias, allowing the model to learn very effective sparse graph recovery algorithm with a small amount of training data. In all cases, the learned algorithm can recover sparse graph structures with much fewer data points from a new problem, and it also works well in recovering gene regulatory networks based on realistic gene expression data generators. Related works. considers CNN based architecture that directly maps empirical covariance matrices to estimated graph structures. Previous works have parameterized optimization algorithms as recurrent neural networks or policies in reinforcement learning. For instance, considered directly parameterizing optimization algorithm as an RNN based framework for learning to learn. approach the problem of automating algorithm design from reinforcement learning perspective and represent any particular optimization algorithm as a policy. learn combinatorial optimzation over graph via deep Q-learning. These works did not consider the structures of our sparse graph recovery problem. Another interesting line of approach is to develop deep neural networks based on unfolding an iterative algorithm;;. developed ALISTA which is based on unrolling the Iterative Shrinkage Thresholding Algorithm (ISTA). developed'ADMM-Net', which is also developed for compressive sensing of MRI data. Though these seminal works were primarily developed for compressive sensing applications, they alluded to the general theme of using unrolled algorithms as inductive biases. We thus identify a suitable unrolled algorithm and leverage its inductive bias to solve the sparse graph recovery problem. Given m observations of a d-dimensional multivariate Gaussian random variable X = [X 1, . . ., X d], the sparse graph recovery problem aims to estimate its covariance matrix Σ * and precision matrix Θ * = (Σ *) −1. The ij-th component of Θ * is zero if and only if X i and X j are conditionally independent given the other variables {X k} k =i,j. Therefore, it is popular to impose an 1 regularization for the estimation of Θ * to increase its sparsity and lead to easily interpretable models. , the problem is formulated as the 1 -regularized maximum likelihood estimation Θ = arg min Θ∈S d ++ − log(det Θ) + tr(ΣΘ) + ρ Θ 1,off, where Σ is the empirical covariance matrix based on m samples, S d ++ is the space of d × d symmetric positive definite matrices (SPD), and Θ 1,off = i =j |Θ ij | is the off-diagonal 1 regularizer with regularization parameter ρ. This estimator is sensible even for non-Gaussian X, since it is minimizing an 1 -penalized log-determinant Bregman divergence. The sparse precision matrix estimation problem in Eq. is a convex optimization problem which can be solved by many algorithms. We give a few canonical and advanced examples which are compared in our experiments: G-ISTA. G-ISTA is a proximal gradient method, and it updates the precision matrix iteratively The step sizes ξ k is determined by line search such that Θ k+1 is SPD matrix. ADMM. Alternating direction methods of multiplier transform the problem into an equivalent constrained form, decouple the log-determinant term and the 1 regularization term, and in the following augmented Lagrangian form with a penalty parameter λ: Taking U:= β/λ as the scaled dual variable, the update rules for the ADMM algorithm are BCD. Block-coordinate decent methods updates each column (and the corresponding row) of the precision matrix iteratively by solving a sequence of lasso problems. The algorithm is very efficient for large scale problems involving thousands of variables. Apart from various algorithms, rigorous statistical analysis has also been provided for the optimal solution of the convex formulation in Eq.. established consistency of the estimator Θ in Eq. in terms of both Frobenius and spectral norms, at rate scaling roughly as with high probability, where s is the number of nonzero entries in Θ *. This statistical analysis also reveal certain limitations of the convex formulation: The established consistency is based on a set of carefully chosen conditions, including the lower bound of sample size, the sparsity level of Θ *, the degree of the graph, the magnitude of the entries in the covariance matrix, and the strength of interaction between edge and non-edge in the precision matrix (or mutual incoherence on the Hessian Γ * := Σ * ⊗ Σ *). In practice, it may be hard to a problem to satisfy these recovery conditions. Therefore, it seems that there is still room for improving the above convex optimization algorithms for recovering the true graph structure. Prior to the data-driven paradigm for sparse recovery, since the target parameter Θ * is unknown, the best precision matrix recovery method is to resort to a surrogate objective function (for instance, equation 1). Optimally tuning the unknown parameter ρ is a very challenging problem in practice. Instead, we can leverage the large amount of simulation or real data and design a learning algorithm that directly optimizes the loss in equation 9. Furthermore, since the log-determinant estimator in Eq. is NOT directly optimizing the recovery objective Θ − Θ * 2 F, there is also a mismatch in the optimization objective and the final evaluation objective (refer to the first experiment in section 5.1). This increase the hope one may improve the by directly optimizing the recovery objective with the algorithms learned from data. In the remainder of the paper, we will present a data-driven method to learn an algorithm for precision matrix estimation, and we call the ing algorithm GLAD (stands for Graph recovery Learning Algorithm using Data-driven training). We ask the question of Given a family of precision matrices, is it possible to improve recovery for sparse graphs by learning a data-driven algorithm? More formally, suppose we are given n precision matrices {Θ * (i) } n i=1 from a family G of graphs and m samples {x (i,j) } m j=1 associated with each Θ * (i). These samples can be used to form n sample. We are interested in learning an algorithm for precision matrix estimation by solving a supervised learning problem,, where f is a set of parameters in GLAD(·) and the output of GLAD f (Σ (i) ) is expected to be a good estimation of Θ * (i) in terms of an interested evaluation metric L. The benefit is that it can directly optimize the final evaluation metric which is related to the desired structure or graph properties of a family of problems. However, it is a challenging task to design a good parameterization of GLAD f for this graph recovery problem. We will explain the challenges below and then present our solution. In the literature on learning data-driven algorithms, most models are designed using traditional deep learning architectures, such as fully connected DNN or recurrent neural networks. But, for graph recovery problems, directly using these architectures does not work well due to the following reasons. First, using a fully connected neural network is not practical. Since both the input and the output of graph recovery problems are matrices, the number of parameters scales at least quadratically in d. Such a large number of parameters will need many input-output training pairs to provide a decent estimation. Thus some structures need to be imposed in the network to reduce the size of parameters and sample complexity. Second, structured models such as convolution neural networks (CNNs) have been applied to learn a mapping from Σ to Θ * . Due to the structure of CNNs, the number of parameters can be much smaller than fully connected networks. However, a recovered graph should be permutation invariant with respect to the matrix rows/columns, and this constraint is very hard to be learned by CNNs, unless there are lots of samples. Also, the structure of CNN is a bias imposed on the model, and there is no guarantee why this structure may work. Third, the intermediate produced by both fully connected networks and CNNs are not interpretable, making it hard to diagnose the learned procedures and progressively output increasingly improved precision matrix estimators. Fourth, the SPD constraint is hard to impose in traditional deep learning architectures. Although, the above limitations do suggest a list of desiderata when designing learning models: Small model size; Minimalist learning; Interpretable architecture; Progressive improvement; and SPD output. These desiderata will motivate the design of our deep architecture using unrolled algorithms. To take into account the above desiderata, we will use an unrolled algorithm as the template for the architecture design of GLAD. The unrolled algorithm already incorporates some problem structures, such as permutation invariance and interpretable intermediate ; but this unrolled algorithm does not traditionally have a learning component, and is typically not directly suitable for gradient-based approaches. We will leverage this inductive bias in our architecture design and augment the unrolled algorithm with suitable and flexible learning components, and then train these embedded models with stochastic gradient descent. GLAD model is based on a reformulation of the original optimization problem in Eq. with a squared penalty term, and an alternating minimization (AM) algorithm for it. More specifically, we consider a modified optimization with a quadratic penalty parameter λ: and the alternating minimization (AM) method for solving it: where η ρ/λ (θ):= sign(θ) max(|θ| − ρ/λ, 0). The derivation of these steps are given in Appendix A. We replace the penalty constants (ρ, λ) by problem dependent neural networks, ρ nn and Λ nn. These neural networks are minimalist in terms of the number of parameters as the input dimensions are mere {3, 2} for {ρ nn, Λ nn} and outputs a single value. Algorithm 1 summarizes the update equations for our unrolled AM based model, GLAD. Except for the parameters in ρ nn and Λ nn, the constant t for initialization is also a learnable scalar parameter. This unrolled algorithm with neural network augmentation can be viewed as a highly structured recurrent architecture as illustrated in Figure 1. There are many traditional algorithms for solving graph recovery problems. We choose AM as our basis because: First, empirically, we tried models built upon other algorithms including G-ISTA, ADMM, etc, but AM-based model gives consistently better performances. Appendix C.10 & C.11 discusses different parameterizations tried. Second, and more importantly, the AM-based architecture has a nice property of maintaining Θ k+1 as a SPD matrix throughout the iterations as long as λ k < ∞. Third, as we prove later in Section 4, the AM algorithm has linear convergence rate, allowing us to use a fixed small number of iterations and still achieve small error margins. Algorithm 1: GLAD To learn the parameters in GLAD architecture, we will directly optimize the recovery objective function rather than using log-determinant objective. A nice property of our deep learning architecture is that each iteration of our model will output a valid precision matrix estimation. This allows us to add auxiliary losses to regularize the intermediate of our GLAD architecture, guiding it to learn parameters which can generate a smooth solution trajectory. Specifically, we will use Frobenius norm in our experiments, and design an objective which has some resemblance to the discounted cumulative reward in reinforcement learning: where (Θ is the output of the recurrent unit GLADcell at k-th iteration, K is number of unrolled iterations, and γ ≤ 1 is a discounting factor. We will use stochastic gradient descent algorithm to train the parameters f in the GLADcell. A key step in the gradient computation is to propagate gradient through the matrix square root in the GLADcell. To do this efficiently, we make use of the property of SPD matrix that X = X 1/2 X 1/2, and the product rule of derivatives to obtain The above equation is a Sylvester's equation for d(X 1/2). Since the derivative dX for X is easy to obtain, then the derivative of d(X 1/2) can be obtained by solving the Sylvester's equation in. The objective function in equation 9 should be understood in a similar way as in;; where deep architectures are designed to directly produce the sparse outputs. For GLAD architecture, a collection of input covariance matrix and ground truth sparse precision matrix pairs are available during training, either coming from simulated or real data. Thus the objective function in equation 9 is formed to directly compare the output of GLAD with the ground truth precision matrix. The goal is to train the deep architecture which can perform well for a family/distribution of input covariance matrix and ground truth sparse precision matrix pairs. The average in the objective function is over different input covariance and precision matrix pairs such that the learned architecture is able to perform well over a family of problem instances. Furthermore, each layer of our deep architecture outputs an intermediate prediction of the sparse precision matrix. The objective function takes into account all these intermediate outputs, weights the loss according to the layer of the deep architecture, and tries to progressively bring these intermediate layer outputs closer and closer to the target ground truth. We note that the designed architecture, is more flexible than just learning the regularization parameters. The component in GLAD architecture corresponding to the regularization parameters are entry-wise and also adaptive to the input covariance matrix and the intermediate outputs. GLAD architecture can adaptively choose a matrix of regularization parameters. This task will be very challenging if the matrix of regularization parameters are tuned manually using cross-validation. A recent theoretical work also validates the choice of GLAD's design. Since GLAD architecture is obtained by augmenting an unrolled optimization algorithm by learnable components, the question is what kind of guarantees can be provided for such learned algorithm, and whether learning can bring benefits to the recovery of the precision matrix. In this section, we will first analyze the statistical guarantee of running the AM algorithm in Eq. and Eq. for k steps with a fixed quadratic penalty parameter λ, and then interpret its implication for the learned algorithm. First, we need some standard assumptions about the true model from the literature: The assumption 2 guarantees that Θ * exists. Assumption 1 just upper bounds the sparsity of Θ * and does not stipulate anything in particular about s. These assumptions characterize the fundamental limitation of the sparse graph recovery problem, beyond which recovery is not possible. Under these assumptions, we prove the linear convergence of AM algorithm (proof is in Appendix B). m, where ρ is the l 1 penalty, d is the dimension of problem and m is the number of samples, the Alternate Minimization algorithm has linear convergence rate for optimization objective defined in. The k th iteration of the AM algorithm satisfies, where 0 < C λ < 1 is a constant depending on λ. From the theorem, one can see that by optimizing the quadratic penalty parameter λ, one can adjust the C λ in the bound. We observe that at each stage k, an optimal penalty parameter λ k can be chosen depending on the most updated value C λ. An adaptive sequence of penalty parameters (λ 1, . . ., λ K) should achieve a better error bound compared to a fixed λ. Since C λ is a very complicated function of λ, the optimal λ k is hard to choose manually. Besides, the linear convergence guarantee in this theorem is based on the sparse regularity parameter ρ log d m. However, choosing a good ρ value in practice is tedious task as shown in our experiments. In summary, the implications of this theorem are: • An adaptive sequence (λ 1, . . ., λ K) should lead to an algorithm with better convergence than a fixed λ, but the sequence may not be easy to choose manually. • Both ρ and the optimal λ k depend on the corresponding error Θ AM − Θ λ F, which make these parameters hard to prescribe manually. • Since, the AM algorithm has a fast linear convergence rate, we can run it for a fixed number of iterations K and still converge with a reasonable error margin. Our learning augmented deep architecture, GLAD, can tune these sequence of λ k and ρ parameters jointly using gradient descent. Moreover, we refer to a recent work by where they considered minimizing the graphical lasso objective with a general nonconvex penalty. They showed that by iteratively solving a sequence of adaptive convex programs one can achieve even better error margins (refer their Algorithm 1 & Theorem 3.5). In every iteration they chose an adaptive regularization matrix based on the most recent solution and the choice of nonconvex penalty. We thus hypothesize that we can further improve our error margin if we make the penalty parameter ρ nonconvex and problem dependent function. We choose ρ as a function depending on the most up-todate solution (Θ k, Σ, Z k), and allow different regularizations for different entries of the precision matrix. Such flexibility potentially improves the ability of GLAD model to recover the sparse graph. In this section, we report several experiments to compare GLAD with traditional algorithms and other data-driven algorithms. The validate the list of desiderata mentioned previously. Especially, it shows the potential of pushing the boundary of traditional graph recovery algorithms by utilizing data. Python implementation (tested on P100 GPU) is available 1. Exact experimental settings details are covered in Appendix C. Evaluation metric. We use normalized mean square error (NMSE) and probability of success (PS) to evaluate the algorithm performance. NMSE is 10 log 10 (E Θ p − Θ * 2 F /E Θ * 2 F) and PS is the probability of correct signed edge-set recovery, i.e., P sign(, where E(Θ *) is the true edge set. Notation. In all reported , D stands for dimension d of the random variable, M stands for sample size and N stands for the number of graphs (precision matrices) that is used for training. Inconsistent optimization objective. Traditional algorithms are typically designed to optimize the 1 -penalized log likelihood. Since it is a convex optimization, convergence to optimal solution is usually guaranteed. However, this optimization objective is different from the true error. Taking ADMM as an example, it is revealed in Figure 2 that, although the optimization objective always converges, errors of recovering true precision matrices measured by NMSE have very different behaviors given different regularity parameter ρ, which indicates the necessity of directly optimizing NMSE and hyperparameter tuning. Expensive hyperparameter tuning. Although hyperparameters of traditional algorithms can be tuned if the true precision matrices are provided as a validation dataset, we want to emphasize that hyperparamter tuning by grid search is a tedious and hard task. Table 1 shows that the NMSE values are very sensitive to both ρ and the quadratic penalty λ of ADMM method. For instance, the optimal NMSE in this table is −9.61 when λ = 0.1 and ρ = 0.03. However, it will increase by a large amount to −2.06 if ρ is only changed slightly to 0.01. There are many other similar observations in this table, where slight changes in parameters can lead to significant NMSE differences, which in turns makes grid-search very expensive. G-ISTA and BCD follow similar trends. For a fair comparison against GLAD which is data-driven, in all following experiments, all hyperparameters in traditional algorithms are fine-tuned using validation datasets, for which we spent extensive efforts (See more details in Appendix C.3, C.6). In contrast, the gradient-based training of GLAD turns out to be much easier. We follow the experimental setting in (; ;) to generate data and perform synthetic experiments on multivariate Gaussians. Each offdiagonal entry of the precision matrix is drawn from a uniform distribution, i.e., Θ * ij ∼ U(−1, 1), and then set to zero with probability p = 1 − s, where s means the sparsity level. Finally, an appropriate multiple of the identity matrix was added to the current matrix, so that the ing matrix had the smallest eigenvalue as 1 (refer to Appendix C.1). We use 30 unrolled steps for GLAD (Figure 3) and compare it to G-ISTA, ADMM and BCD. All algorithms are trained/finetuned using 10 randomly generated graphs and tested over 100 graphs. Convergence and average runtime of different algorithms on Nvidia's P100 GPUs are shown in Figure 4 and Table 2 respectively. GLAD consistently converges faster and gives lower NMSE. Although the fine-tuned G-ISTA also has decent performance, the computation time in each iteration is much longer than GLAD because it requires line search steps. Besides, we could also see a progressive improvement of GLAD across its iterations. As analyzed by , the recovery guarantee (such as in terms of Frobenius norm) of the 1 regularized log-determinant optimization significantly depends on the sample size and other conditions. Our GLAD directly optimizes the recovery objective based on data, and it has the potential of pushing the sample complexity limit. We experimented with this and found the positive. We follow to conduct experiments on GRID graphs, which satisfy the conditions required in . Furthermore, we conduct a more challenging task of recovering restricted but randomly constructed graphs (see Appendix C.7 for more details). The probability of success (PS) is non-zero only if the algorithm recovers all the edges with correct signs, plotted in Figure 5. GLAD consistently outperforms traditional methods in terms of sample complexity as it recovers the true edges with considerably fewer number of samples. Having a good inductive bias makes GLAD's architecture quite data-efficient compared to other deep learning models. For instance, the state-of-the-art'DeepGraph' by is based on CNNs. It contains orders of magnitude more parameters than GLAD. Furthermore, it takes roughly 100, 000 samples, and several hours for training their DG-39 model. In contrast, GLAD learns well with less than 25 parameters, within 100 training samples, and notably less training time. Table 3 also shows that GLAD significantly outperforms DG-39 model in terms of AUC (Area under the ROC curve) by just using 100 training graphs, typically the case for real world settings. Fully connected DL models are unable to learn from such small data and hence are skipped in the comparison. Figure 6 shows that GLAD performs favourably for structure recovery in terms of NMSE on the gene expression data. As the governing equations of the underlying distribution of the SynTReN are unknown, these experiments also emphasize the ability of GLAD to handle non-Gaussian data. Figure 7 visualizes the edge-recovery performance of GLAD models trained on a sub-network of true Ecoli bacteria data. We denote, TPR: True Positive Rate, FPR: False Positive Rate, FDR: False Discovery Rate. The number of simulated training/validation graphs were set to 20/20. One batch of M samples were taken per graph (details in Appendix C.9). Although, GLAD was trained on graphs with D = 25, it was able to robustly recover a higher dimensional graph D = 43 structure. Appendix C.12 contains details of the experiments done on real E.Coli data. The GLAD model was trained using the SynTReN simulator. Appendix C.13 explains our proposed approach to scale for larger problem sizes. We presented a novel neural network, GLAD, for the sparse graph recovery problem based on an unrolled Alternating Minimization algorithm. We theoretically prove the linear convergence of AM algorithm as well as empirically show that learning can further improve the sparse graph recovery. The learned GLAD model is able to push the sample complexity limits thereby highlighting the potential of using algorithms as inductive biases for deep learning architectures. Further development of theory is needed to fully understand and realize the potential of this new direction. Alternating Minimization is performing Taking the gradient of the objective function with respect to Θ to be zero, we have Taking the gradient of the objective function with respect to Z to be zero, we have where Solving the above two equations, we obtain: where B LINEAR CONVERGENCE RATE ANALYSIS m, where ρ is the l 1 penalty, d is the dimension of problem and m is the number of samples, the Alternate Minimization algorithm has linear convergence rate for optimization objective defined in. The k th iteration of the AM algorithm satisfies, where 0 < C λ < 1 is a constant depending on λ. We will reuse the following notations in the appendix: The update rules for Alternating Minimization are: Assumptions: With reference to the theory developed in , we make the following assumptions about the true model. (O P (·) is used to denote bounded in probability.) We now proceed towards the proof: Lemma 2. For any x, y, k ∈ R, k > 0, x = y, Proof. where is the largest eigenvalue of X in absolute value. Proof. First we factorize X using eigen decomposition, X = Q X D X Q X, where Q X and D X are orthogonal matrix and diagonal matrix, respectively. Then we have, Similarly, the above equation holds for Y. Therefore, where we define Q:= Q Y Q X. Similarly, we have, Then the i-th entry on the diagonal of ji. Using the fact that D X and D Y are diagonal, we have, The last step makes use of Similarly, using, we have, Assuming X − Y F > 0 (otherwise trivially holds), using and, we have, Using lemma, we have, Therefore, Lemma 4. Under assumption, the output of the k-th and where 0 < C λ < 1 is a constant depending on λ. Proof. The first part is easy to show, if we observe that in the second update step of AM, η ρ/λ is a contraction under metric d(X, Y) = X − Y F. Therefore we have, Next we will prove the second part. To simplify notation, we let A(X) = X X + 4 λ I. Using the first update step of AM, we have, where The last derivation step makes use of the triangle inequality. Using lemma, we have, Therefore where Λ max (X) is the largest eigenvalue of X in absolute value. The rest is to show that both Λ max (Y λ) and Λ max (Y k+1) are bounded using assumption. For Λ max (Y k+1), we have, Combining and, we have, Therefore, Continuing with, we have, Since Z λ is the minimizer of a strongly convex function, its norm is bounded. And we also have Therefore both Λ max (Y λ) and Λ max (Y k+1) are bounded in, i.e. 0 < C λ < 1 is a constant only depending on λ. m, where ρ is the l 1 penalty, d is the dimension of problem and m is the number of samples, the Alternate Minimization algorithm has linear convergence rate for optimization objective defined in. The k th iteration of the AM algorithm satisfies, where 0 < C λ < 1 is a constant depending on λ. Proof. Error between Θ λ and Θ G Combining the following two equations: Note that by the optimality condition, ∇ z f (Θ λ, Z λ, ρ, λ) = 0, we have the fixed point equation λ and we have: Since G is σ G -strongly convex, where σ G is independent of the sample covariance matrix Σ * as the hessian of G is independent of Σ *. Therefore, Proof. Error between Θ G and Θ * Corollary 5 (Theorem 1. of). Let Θ G be the minimizer for the optimization C EXPERIMENTAL DETAILS This section contains the detailed settings used in the experimental evaluation section. For sections 5.1 and 5.2, the synthetic data was generated based on the procedure described in. A d dimensional precision matrix Θ was generated by initializing a d × d matrix with its off-diagonal entries sampled i.i.d. from a uniform distribution Θ ij ∼ U(−1, 1). These entries were then set to zero based on the sparsity pattern of the corresponding Erdos-Renyi random graph with a certain probability p. Finally, an appropriate multiple of the identity matrix was added to the current matrix, so that the ing matrix had the smallest eigenvalue as 1. In this way, Θ was ensured to be. The top row has the sparsity probability p = 0.5 for the Erdos-Renyi random graph, whereas for the bottom row plots, the sparsity probabilities are uniformly sampled from ∼ U(0.05, 0.15). For finetuning the traditional algorithms, a validation dataset of 10 graphs was used. For the GLAD algorithm, 10 training graphs were randomly chosen and the same validation set was used. C.5 GLAD: ARCHITECTURE DETAILS FOR SECTION(5.2) GLAD parameter settings: ρ nn was a 4 layer neural network and Λ nn was a 2 layer neural network. Both used 3 hidden units in each layer. The non-linearity used for hidden layers was tanh, while the final layer had sigmoid (σ) as the non-linearity for both, ρ nn and Λ nn (refer Figure 3). The learnable offset parameter of initial Θ 0 was set to t = 1. It was unrolled for L = 30 iterations. The learning rates were chosen to be around [0.01, 0.1] and multi-step LR scheduler was used. The optimizer used was'adam'. The best nmse model was selected based on the validation data performance. Figure(Figure 9: We attempt to illustrate how the traditional methods are very sensitive to the hyperparameters and it is a tedious exercise to finetune them. The problem setting is same as described in section(5.3). For all the 3 methods shown above, we have already tuned the algorithm specific parameters to a reasonable setting. Now, we vary the L 1 penalty term ρ and can observe that how sensitive the probability of success is with even slight change of ρ values. values are very sensitive to the choice of t as well. These parameter values changes substantially for a new problem setting. G-ISTA and BCD follow similar trends. Additional plots highlighting the hyperparameter sensitivity of the traditional methods for model selection consistency experiments. Refer figure. Details for experiments in figure. Two different graph types were chosen for this experiment which were inspired from. In the'grid' graph setting, the edge weight for different precision matrices were uniformly sampled from w ∼ U(0.12, 0.25). The edges within a graph carried equal weights. The other setting was more general, where the graph was a random Erdos-Renyi graph with probability of an edge was p = 0.05. The off-diagonal entries of the precision matrix were sampled uniformly from ∼ U[0.1, 0.4]. The parameter settings for GLAD were the same as described in Appendix C.5. The model with the best PS performance on the validation dataset was selected. train/valid/test=10/10/100 graphs were used with 10 sample batches per graph. C.8 GLAD: COMPARISON WITH OTHER DEEP LEARNING BASED METHODS Table shows AUC (with std-err) comparisons with the DeepGraph model. For experiment settings, refer Table 1 of. Gaussian Random graphs with sparsity p = 0.05 were chosen and edge values sampled from ∼ U(−1, 1). GLAD was trained on only 10 graphs with 5 sample batches per graph. The dimension of the problem is D = 39. The architecture parameter choices of GLAD were the same as described in Appendix C.5 and it performs consistently better along all the settings by a significant AUC margin. The SynTReN Van den is a synthetic gene expression data generator specifically designed for analyzing the structure learning algorithms. The topological characteristics of the synthetically generated networks closely resemble the characteristics of real transcriptional networks. The generator models different types of biological interactions and produces biologically plausible synthetic gene expression data enabling the development of data-driven approaches to recover the underlying network. The SynTReN simulator details for section(5.5). For performance evaluation, a connected ErdosRenyi graph was generated with probability as p = 0.05. The precision matrix entries were sampled from Θ ij ∼ U(0.1, 0.2) and the minimum eigenvalue was adjusted to 1 by adding an appropriate multiple of identity matrix. The SynTReN simulator then generated samples from these graphs by incorporating biological noises, correlation noises and other input noises. All these noise levels were sampled uniformly from ∼ U(0.01, 0.1). The figure shows the NMSE comparisons for a fixed dimension D = 25 and varying number of samples M =. The number of training/validation graphs were set to 20/20 and the are reported on 100 test graphs. In these experiments, only 1 batch of M samples were taken per graph to better mimic the real world setting. Figure Unrolled model for ADMM: Algorithm 2 describes the unrolled model ADMMu updates. ρ nn was a 4 layer neural network and Λ nn was a 2 layer neural network. Both used 3 hidden units in each layer. The non-linearity used for hidden layers was tanh, while the final layer had sigmoid (σ) as the non-linearity for both,ρ nn and Λ nn. The learnable offset parameter of initial Θ 0 was set to t = 1. It was unrolled for L = 30 iterations. The learning rates were chosen to be around [0.01, 0.1] and multi-step LR scheduler was used. The optimizer used was'adam'. Figure 10 compares GLAD with ADMMu on the convergence performance with respect to synthetically generated data. The settings were kept same as described in Figure 4. As evident from the plots, we see that GLAD consistently performs better than ADMMu. We had similar observations for other set of experiments as well. Hence, we chose AM based unrolled algorithm over ADMM's as it works better empirically and has less parameters. Although, we are not entirely confident but we hypothesize the reason for above observations as follows. In the ADMM update equations (4 & 5), both the Lagrangian term and the penalty term are intuitively working together as a'function' to update the entries Θ ij, Z ij. Observe that U k can be absorbed into Z k and/or Θ k and we expect our neural networks to capture this relation. We thus expect GLAD to work at least as good as ADMMu. In our formulation of unrolled ADMMu (Algorithm 2) the update step of U is not controlled by neural networks (as the number of parameters needed will be substantially larger) which might be the reason of it not performing as well as GLAD. Our empirical evaluations corroborate this logic that just by using the penalty term we can maintain all the desired properties and learn the problem dependent'functions' with a small neural network. We tried multiple unrolled parameterizations of the optimization techniques used for solving the graphical lasso problem which worked to varying levels of success. We list here a few, in interest for helping researchers to further pursue this recent and novel approach of data-driven algorithm designing. 1. ADMM + ALISTA parameterization: The threshold update for Z AM k+1 can be replaced by ALISTA network. The stage I of ALISTA is determining W, which is trivial in our case as D = I. So, we get W = I. Thus, combining ALISTA updates along with AM's we get an interesting unrolled algorithm for our optimization problem. All the settings are same as the fixed sparsity case described in Figure 4. We see that the AM based parameterization'GLAD' consistently performs better than the ADMM based unrolled architecture'ADMMu'. 2. G-ISTA parameterization: We parameterized the line search hyperparameter c as well as replaced the next step size determination step by a problem dependent neural network of Algorithm in. The main challenge with this parameterization is to main the PSD property of the intermediate matrices obtained. Learning appropriate parameterization of line search hyperparameter such that PSD condition is maintained remains an interesting aspect to investigate. 3. Mirror Descent Net: We get a similar set of update equations for the graphical lasso optimization. We identify some learnable parameters, use neural networks to make them problem dependent and train them end-to-end. 4. For all these methods we also tried unrolling the neural network as well. In our experience we found that the performance does not improve much but the convergence becomes unstable. We use the real data from the'DREAM 5 Network Inference challenge' . This dataset contains 3 compendia that were obtained from microorganisms, some of which are pathogens of clinical relevance. Each compendium consists of hundreds of microarray experiments, which include a wide range of genetic, drug, and environmental perturbations. We test our method for recovering the true E.coli network from the gene expression values recorded by doing actual microarray experiments. The E.coli dataset contains 4511 genes and 805 associated microarray experiments. The true underlying network has 2066 discovered edges and 150214 pairs of nodes do not have an edge between them. There is no data about the remaining edges. For our experiments, we only consider the discovered edges as the ground truth, following the challenge data settings. We remove the genes that have zero degree and then we get a subset of 1081 genes. For our predictions, we ignore the direction of the edges and only consider retrieving the connections between genes. We train the GLAD model using the SynTReN simulator on the similar settings as described in Appendix C.9. Briefly, GLAD model was trained on D=50 node graphs sampled from Erdos-Renyi graph with sparsity probability ∼ U (0.01, 0.1), noise levels of SynTReN simulator sampled from ∼ U (0.01, 0.1) and Θ ij ∼ U (0.1, 0.2)). The model was unrolled for 15 iterations. This experiment also evaluates GLAD's ability to generalize to different distribution from training as well as scaling ability to more number of nodes. We report the AUC scores for E.coli network in Table 4. We can see that GLAD improves over the other competing methods in terms of Area Under the ROC curve (AUC). We understand that it is challenging to model real datasets due to the presence of many unknown latent extrinsic factors, but we do observe an advantage of using data-driven parameterized algorithm approaches. Methods BCD GISTA GLAD AUC 0.548 0.541 0.572 We have shown in our experiments that we can train GLAD on smaller number of nodes and get reasonable for recovering graph structure with considerably larger nodes (AppendixC.12). Thus, in this section, we focus on scaling up on the inference/test part. With the current GPU implementation, we can can handle around 10,000 nodes for inference. For problem sizes with more than 100,000 nodes, we propose to use the randomized algorithm techniques given in. Kindly note that scaling up GLAD is our ongoing work and we just present here one of the directions that we are exploring. The approach presented below is to give some rough idea and may contain loose ends. Randomized algorithms techniques are explained elaborately in. Specifically, we will use some of their key • P1. (Theorem 2.1) We will use the length-squared sampling technique to come up with low-rank approximations • P2. (Theorem 2.5) For any large matrix A ∈ R m×n, we can use approximate it as A ≈ CU R, where C ∈ R m×r, U ∈ R s×r, R ∈ R r×m. • P3. (Section 2.3) For any large matrix A ∈ R m×n, we can get its approximate SVD by using the property E(R T R) = A T A where R is a matrix obtained by length-squared sampling of the rows of matrix A. The steps for doing approximate AM updates, i.e. of equations. Using property P3, we can approximate where V is the right singular vectors of R. Thus, we can combine this approximation with the sketch matrix approximation of Y ≈ CU R to calculate the update in equation. Equation is just a thresholding operation and can be done efficiently with careful implementation. We are looking in to the experimental as well as theoretical aspects of this approach. We are also exploring an efficient distributed algorithm for GLAD. We are investigating into parallel MPI based algorithms for this task (https://stanford.edu/~boyd/admm.html is a good reference point). We leverage the fact that the size of learned neural networks are very small, so that we can duplicate them over all the processors. This is also an interesting future research direction.' and it should match with 'A data-driven learning algorithm based on unrolling the Alternating Minimization optimization for sparse graph recovery.' but not with 'we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.'.

Question: Multilingual Neural Machine Translation (NMT) systems are capable of translating between multiple source and target languages within a single system. An important indicator of generalization within these systems is the quality of zero-shot translation - translating between language pairs that the system has never seen during training. However, until now, the zero-shot performance of multilingual models has lagged far behind the quality that can be achieved by using a two step translation process that pivots through an intermediate language (usually English). In this work, we diagnose why multilingual models under-perform in zero shot settings. We propose explicit language invariance losses that guide an NMT encoder towards learning language agnostic representations. Our proposed strategies significantly improve zero-shot translation performance on WMT English-French-German and on the IWSLT 2017 shared task, and for the first time, match the performance of pivoting approaches while maintaining performance on supervised directions. In recent years, the emergence of sequence to sequence models has revolutionized machine translation. Neural models have reduced the need for pipelined components, in addition to significantly improving translation quality compared to their phrase based counterparts BID35. These models naturally decompose into an encoder and a decoder with a presumed separation of roles: The encoder encodes text in the source language into an intermediate latent representation, and the decoder generates the target language text conditioned on the encoder representation. This framework allows us to easily extend translation to a multilingual setting, wherein a single system is able to translate between multiple languages BID11 BID28.Multilingual NMT models have often been shown to improve translation quality over bilingual models, especially when evaluated on low resource language pairs BID14 BID20. Most strategies for training multilingual NMT models rely on some form of parameter sharing, and often differ only in terms of the architecture and the specific weights that are tied. They allow specialization in either the encoder or the decoder, but tend to share parameters at their interface. An underlying assumption of these parameter sharing strategies is that the model will automatically learn some kind of shared universally useful representation, or interlingua, ing in a single model that can translate between multiple languages. The existence of such a universal shared representation should naturally entail reasonable performance on zero-shot translation, where a model is evaluated on language pairs it has never seen together during training. Apart from potential practical benefits like reduced latency costs, zero-shot translation performance is a strong indicator of generalization. Enabling zero-shot translation with sufficient quality can significantly simplify translation systems, and pave the way towards a single multilingual model capable of translating between any two languages directly. However, despite being a problem of interest for a lot of recent research, the quality of zero-shot translation has lagged behind pivoting through a common language by 8-10 BLEU points BID15 BID24 BID21 BID27. In this paper we ask the question, What is the missing ingredient that will allow us to bridge this gap? Figure 1: The proposed multilingual NMT model along with the two training objectives. CE stands for the cross-entropy loss associated with maximum likelihood estimation for translation between English and other languages. Align represents the source language invariance loss that we impose on the representations of the encoder. While training on the translation objective, training samples (x, y) are drawn from the set of parallel sentences, D x,y. For the invariance losses, (x, y) could be drawn from D x,y for the cosine loss, or independent data distributions for the adversarial loss. Both losses are minimized simultaneously. Since we have supervised data only to and from English, one of x or y is always in English. In BID24, it was hinted that the extent of separation between language representations was negatively correlated with zero-shot translation performance. This is supported by theoretical and empirical observations in domain adaptation literature, where the extent of subspace alignment between the source and target domains is strongly associated with transfer performance BID7 BID8 BID17. Zero-shot translation is a special case of domain adaptation in multilingual models, where English is the source domain and other languages collectively form the target domain. Following this thread of domain adaptation and subspace alignment, we hypothesize that aligning encoder representations of different languages with that of English might be the missing ingredient to improving zero-shot translation performance. In this work, we develop auxiliary losses that can be applied to multilingual translation models during training, or as a fine-tuning step on a pre-trained model, to force encoder representations of different languages to align with English in a shared subspace. Our experiments demonstrate significant improvements on zero-shot translation performance and, for the first time, match the performance of pivoting approaches on WMT English-French-German (en-fr-de) and the IWSLT 2017 shared task, in all zero shot directions, without any meaningful regression in the supervised directions. We further analyze the model's representations in order to understand the effect of our explicit alignment losses. Our analysis reveals that tying weights in the encoder, by itself, is not sufficient to ensure shared representations. As a , standard multilingual models overfit to the supervised directions, and enter a failure mode when translating between zero-shot languages. Explicit alignment losses incentivize the model to use shared representations, ing in better generalization.2 ALIGNMENT OF LATENT REPRESENTATIONS 2.1 MULTILINGUAL NEURAL MACHINE TRANSLATION Let x = (x 1, x 2 ...x m) be a sentence in the source language and y = (y 1, y 2, ... y n) be its translation in the target language. For machine translation, our objective is to learn a model, p(y|x; θ). In modern NMT, we use sequence-to-sequence models supplemented with an attention mechanism BID5 to learn this distribution. These sequence-to-sequence models consist of an encoder, Enc(x) = z = (z 1, z 2, ...z m) parameterized with θ enc, and a decoder that learns to map from the latent representation z to y by modeling p(y|z; θ dec), again parameterized with θ dec. This model is trained to maximize the likelihood of the available parallel data, D x,y. DISPLAYFORM0 In multilingual training we jointly train a single model BID26 to translate from many possible source languages to many potential target languages. When only the decoder is informed about the desired target language, a special token to indicate the target language, < tl >, is input to the first step of the decoder. In this case, D x,y is the union of all the parallel data for each of the supervised translation directions. Note that either the source or the target is always English. For zero-shot translation to work, the encoder needs to produce language invariant feature representations of a sentence. Previous works learn these transferable features by using a weight sharing constraint and tying the weights of the encoders, the decoders, or the attentions across some or all languages BID11 BID24 BID27 BID14. They argue that sharing these layers across languages causes sentences that are translations of each other to cluster together in a common representation space. However, when a model is trained on just the end-to-end translation objective, there is no explicit incentive for the model to discover language invariant representations; given enough capacity, it is possible for the model to partition its intrinsic dimensions and overfit to the supervised translation directions. This would in intermediate encoder representations that are specific to individual languages. We now explore two classes of regularizers, Ω, that explicitly force the model to make the representations in all other languages similar to their English counterparts. We align the encoder representations of every language with English, since it is the only language that gets translated into all other languages during supervised training. Thus, English representations now form an implicit pivot in the latent space. The loss function we then minimize is: DISPLAYFORM0 where L CE is the cross-entropy loss and λ is a hyper-parameter that controls the contribution of the alignment loss Ω. Here we view zero-shot translation through the lens of domain adaptation, wherein English is the source domain and the other languages together constitute the target domain. BID7 and BID30 have shown that target risk can be bounded by the source risk plus a discrepancy metric between the source and target feature distribution. Treating the encoder as a deterministic feature extractor, the source distribution is Enc(x en)p(x en) and the target distribution is Enc(x t)p(x t). To enable zero-shot translation, our objective then is to minimize the discrepancy between these distributions by explicitly optimizing the following domain adversarial loss BID17: DISPLAYFORM0 where Disc is the discriminator and is parametrized by θ disc. D En are English sentences and D T are the sentences of all the other languages. Note that, unlike BID4 BID41, who also train the encoder adversarially with a language detecting discriminator, we are trying to align the distribution of encoder representations of all other languages to that of English and vice-versa. Our discriminator is just a binary predictor, independent of how many languages we are jointly training on. Architecturally, the discriminator is a feed-forward network that acts on the temporally max-pooled representation of the encoder output. We also experimented with a discriminator that made independent predictions for the encoder representation, z i, at each time-step i, but found the pooling based approach to work better. More involved discriminators that consider the sequential nature of the encoder representations may be more effective, but we do not explore them in this work. While adversarial approaches have the benefit of not needing parallel data, they only align the marginal distributions of the encoder's representations. Further, adversarial approaches are hard to optimize and are often susceptible to mode collapse, especially when the distribution to be modeled is multi-modal. Even if the discriminator is fully confused, there are no guarantees that the two learned distributions will be identical BID2.To resolve these potential issues, we attempt to make use of the available parallel data, and enforce an instance level correspondence between the pairs (x, y) ∈ D x,y, rather than just aligning the marginal distributions of Enc(x)p(x) and Enc(y)p(y) as in the case of domain-adversarial training. Previous work on multi-modal and multi-view representation learning has shown that, when given paired data, transferable representations can be learned by improving some measure of similarity between the corresponding views from each mode. Various similarity measures have been proposed such as Euclidean distance BID22, cosine distance BID16, correlation BID1 etc. In our case, the different views correspond to equivalent sentences in different languages. Note that Enc(x) and Enc(y) are actually a pair of sequences, and to compare them we would ideally have access to the word level correspondences between the two sentences. In the absence of this information, we make a bag-of-words assumption and align the pooled representation similar to BID19; BID10. Empirically, we find that max pooling and minimizing the cosine distance between the representations of parallel sentences similar to works well. We now minimize the distance function: DISPLAYFORM0 A multilingual model with a single encoder and a single decoder similar to BID24 is our baseline. This setup maximally enforces the parameter sharing constraint that previous works rely on to promote cross-lingual transfer. We first train our model solely on the translation loss until convergence, on all languages to and from English. This is our baseline multilingual model. We then fine-tune this model with the proposed alignment losses, in conjunction with the translation objective. We then compare the performance of the baseline model against the aligned models on both the supervised and the zero-shot translation directions. We also compare our zero-shot performance against the pivoting performance using the baseline model. For our en↔{fr, de} experiments, we train our models on the standard en→fr (39M) and en→de (4.5M) training datasets from WMT'14. We pre-process the data by applying the standard Moses pre-processing 1. We swap the source and target to get parallel data for the fr→en and de→en directions. The ing datasets are merged by oversampling the German portion to match the size of the French portion. This in a total of 158M sentence pairs. We get word counts and apply 32k BPE BID33 ) to obtain subwords. The target language < tl > tokens are also added to the vocabulary. We use newstest-2012 as the dev set and newstest-2013 as the test set. Both of these sets are 3-way parallel and have 3003 and 3000 sentences respectively. We run all our experiments with Transformers BID36, using the TransformerBase config. We train our model with a learning rate of 1.0 and 4000 warmup steps. Input dropout is set to 0.1. We use synchronized training with 16 Tesla P100 GPUs and train the model for 500k steps. The model is instructed on which language to translate a given input sentence into, by feeding in a unique < tl > token per target language. In our implementation, this token is pre-pended into the source sentence, but it could just as easily be fed into the decoder to the same effect. For the alignment experiments, we fine-tune a pre-trained multilingual model by jointly training on both the alignment and translation losses. For adversarial alignment, the discriminator is a feedforward network with 3 hidden layers of dimension 2048 using the leaky ReLU(α = 0.1) nonlinearity. λ was tuned to 1.0 for both the adversarial and the cosine alignment losses. Simple fine-tuning with SGD using a learning rate of 1e-4 works well and we do not need to train from scratch. We observe that the models converge within a few thousand updates. de → f r f r → de en → f r en → de f r → Table 1: Zero-shot with baseline and aligned models compared against pivoting. Zero-Shot are marked zs. Pivoting through English is performed using the baseline multilingual model. Our , in Table 1, demonstrate that both our approaches to align representations in large improvements in zero-shot translation quality for both directions, effectively closing the gap to the performance of the strong pivoting baseline. We didn't notice any significant differences between the performance of the two proposed alignment methods. Importantly, these improvements come at no cost to the quality in the supervised directions. While both the proposed approaches aren't significantly different in terms of final quality, we noticed that the adversarial regularizer was very sensitive to the initialization scheme and the choice of hyper-parameters. In comparison, the cosine distance loss was relatively stable, with λ being the only hyper-parameter controlling the weight of the alignment loss with respect to the translation loss. We further analyze the outputs of our baseline multilingual model in order to understand the effect of alignment on zero-shot performance. We identify the major effects that contribute to the poor zeroshot performance in multilingual models, and investigate how an explicit alignment loss resolves these pathologies. 94% 0% f r references 4% 0% 96% de references 4% 96% 0% Table 2: Percentage of sentences by language in reference translations and the sentences decoded using the baseline model (newstest2012)While investigating the high variance of the zero-shot translation score during multilingual training in the absence of alignment, we found that a significant fraction of the examples were not getting translated into the desired target language at all. Instead, they were either translated to English or simply copied. This phenomenon is likely a consequence of the fact that at training time, German and French source sentences were always translated into English. Because of this, the model never learns to properly attribute the target language to the < tl > token, and simply changing the < tl > token at test time is not effective. We count the number of sentences in each language using an automatic language identification tool and report the in Table 2.Further, we find that for a given sentence, all output tokens tend to be in the same language, and there is little to no code-switching. This was also observed by BID24, where it was explained as a cascading effect in the decoder: Once the decoder starts emitting tokens in one language, the conditional distribution p(y i |y i−1, ..., y 1) is heavily biased towards that particular language. With explicit alignment, we remove the target language information encoded into the source token representations. In the absence of this confounding information, the < tl > target token gives us more control to set the translation direction. Table 3: BLEU on subset of examples predicted in the right language by the direct translation using the baseline system (newstest2012)Here we try to isolate the gains our system achieves due to improvements in the learning of transferable features, from those that can be attributed to decoding to the desired language. We discount the errors that could be attributed to incorrect language errors and inspect the translation quality on the subset of examples where the baseline model decodes in the right language. We re-evaluate the BLEU scores of all systems and show the in Table 3. We find that the vanilla zero-shot translation system (Baseline) is much stronger than expected at first glance. It only lags the pivoting baseline by 0.5 BLEU points on French to German and by 2.7 BLEU points on German to French. We can now see that, even on this subset which was chosen to favor the baseline model, the representation alignment of our adapted model contributes to improving the quality of zero-shot translation by 0.7 and 2.2 BLEU points on French to German and German to French, respectively. We design a simple experiment to determine whether representations learned while training a multilingual translation model are truly cross-lingual. We probe our baseline and aligned multilingual models with 3-way aligned data to determine the extent to which their representations are functionally equivalent, during different stages in model training. Because source languages can have different sequence lengths and word orders for equivalent sentences, it is not possible to directly compare encoder output representations. However, it is possible to directly compare the representations extracted by the decoder from the encoder outputs for each language. Suppose we want to compare representations of semantically equivalent English and German sentences when translating into French. At time-step i in the decoder, we use the model to predict p(y i |Enc(x en), y 1:(i−1) ) and p(y i |Enc(x de), y 1:(i−1) ). However, in the seq2seq with attention formulation, these problems reduce to predicting p(y i |c We use a randomly sampled set of 100 parallel en-de-fr sentences extracted from our dev set, newstest2012, to perform this analysis. For each set of aligned sentences, we obtain the sequence of aligned context vectors (c en i, c de i) and plot the mean cosine distances for our baseline training run, and the incremental runs with alignment losses in FIG1. Our indicate that the vanilla multilingual model learns to align encoder representations over the course of training. However, in the absence of an external incentive, the alignment process arrests as training progresses. Incrementally training with the alignment losses in a more language-agnostic representation, which contributes to the improvements in zero-shot performance. Given the good on WMT en-fr-de, we now extend our experiments, to test the scalability of our approach to multiple languages. We work with the IWSLT-17 dataset which has transcripts of Ted talks in 5 languages: English (en), Dutch (nl), German (de), Italian (it), and Romanian (ro). The original dataset is multi-way parallel with approximately 220 thousand sentences per language, but for the sake of our experiments we only use the to/from English directions for training. The dev and test sets are also multi-way parallel and comprise around 900 and 1100 sentences per language pair respectively. We again use the transformer base architecture. We set the learning rate to 2.0 and the number of warmup steps to 8k. A dropout rate of 0.2 was applied to all connections of the transformer. We use the cosine loss with λ set to 0.001 because of how easy it is to tune. Our baseline model's scores on IWSLT-17 are suspiciously close to that of bridging, as seen in TAB3. We suspect this is because the data that we train on is multi-way parallel, and the English sentences are shared across the language pairs. This may be helping the model learn shared representations with the English sentences acting as pivots. Even so, we are able to gain 1 BLEU over the strong baseline system and demonstrate the applicability of our approach to larger groups of languages.5 RELATED WORK Multilingual NMT models were first proposed by BID11 and have since been explored in BID14; BID9 and several other works. While zero-shot translation was the direct goal of BID14, they were only able to achieve'zero-resource translation', by using their pre-trained multi-way multilingual model to generate pseudo-parallel data for fine-tuning. BID24 were the first to show the possibility of zero-shot translation by proposing a model that shared all the components and used a token to indicate the target language. BID32 propose a novel way to modulate the amount of sharing between languages, by using a parameter generator to generate the parameters for either the encoder or the decoder of the multilingual NMT system based on the source and target languages. They also report higher zero-shot translation scores with this approach. Learning coordinated representations with the use of parallel data has been explored thoroughly in the context of multi-view and multi-modal learning BID6. These often involve either auto-encoder like networks with a reconstruction objective, or paired feed-forward networks with a similarity based objective. This function used to encourage similarity may be Euclidean distance BID22, cosine distance BID16, partial order BID37, correlation BID1, etc. More recently a vast number of adversarial approaches have been proposed to learn domain invariant representations, by ensuring that they are indistinguishable by a discriminator network BID17.The use of aligned parallel data to learn shared representations is common in the field of crosslingual or multilingual representations, where work falls into three main categories. Obtaining representations from word level alignments -bilingual dictionaries or automatically generated word alignments -is the most popular approach BID13 BID42. The second category of methods try to leverage document level alignment, like parallel Wikipedia articles, to generate cross-lingual representations BID34 BID38. The final category of methods often use sentence level alignments, in the form of parallel translation data, to obtain cross-lingual representations BID23 BID18 BID29 BID0. Recent work by BID12 showed that the representations learned by a multilingual NMT system are widely applicable across tasks and languages. Parameter sharing based approaches have also been tried in the context of unsupervised NMT, where learning a shared latent space BID3 was believed to improve translation quality. Some approaches explore applying adversarial losses on the encoder, to ensure that the representations are language agnostic. However, recent work has shown that enforcing a shared latent space is not important for unsupervised NMT BID25, and the cycle consistency loss suffices by itself. In this work we propose explicit alignment losses, as an additional constraint for multilingual NMT models, with the goal of improving zero-shot translation. We view the zero-shot NMT problem in the light of subspace alignment for domain adaptation, and propose simple approaches to achieve this. Our experiments demonstrate significantly improved zero-shot translation performance that are, for the first time, comparable to strong pivoting based approaches. Through careful analyses we show how our proposed alignment losses in better representations, and thereby better zeroshot performance, while still maintaining performance on the supervised directions. Our proposed methods have been shown to work reliably on two public benchmarks datasets: WMT EnglishFrench-German and the IWSLT 2017 shared task.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Simple similarity constraints on top of multilingual NMT enables high quality translation between unseen language pairs for the first time.

Represent the TLDR for finding the corresponding scientific article

Conditional VAE on top of latent spaces of pre-trained generative models that enables  transfer between drastically different domains while preserving locality and semantic alignment.