{'fraction_non_alphanumeric': 0.04080703883495146, 'fraction_numerical': 0.015738774271844662, 'mean_word_length': 4.5107628004179725, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Interpretable Deep Representation Learning from Temporal Multi-view Data

In many scientific problems such as video surveillance, modern genomics, and finance, data are often collected from diverse measurements across time that exhibit time-dependent heterogeneous properties. Thus, it is important to not only integrate data from multiple sources (called multi-view data), but also to incorporate time dependency for deep understanding of the underlying system. We propose a generative model based on variational autoencoder and a recurrent neural network to infer the latent dynamics for multi-view temporal data. This approach allows us to identify the disentangled latent embeddings across views while accounting for the time factor. We invoke our proposed model for analyzing three datasets on which we demonstrate the effectiveness and the interpretability of the model.


Introduction
Multi-view data is prevalent in real-world applications. For instance, a photo can be taken at different angles, the human motion can be described by different gestures, and medical scenarios where each observed clinical outcome of a patient can correspond to a specific medical test. These views often represent diverse and complementary information of the same data. Integrating multi-view data has the potential to yield more generalizable representations and is helpful in boosting the performance of data mining tasks.
Temporal multi-view data arise in a wide variety of fields, such as biomedical research, sociology, finance, computer vision and many others (Yang et al., 2017) in which datasets are collected repeatedly over time for each individual. Analyzing temporal multi-view data in such studies, with the objective of delivering interpretable learning, is challenging.
Many popular multi-view learning methods have been developed based on group factor analysis (Sridharan and Kakade, 2008;Klami et al., 2015;Zhao et al., 2016;Leppaaho et al., 2017), where each group corresponds to a specific data view. The group factor analysis generates a common linear mapping between the latent and observed groups of variables (multiple views). In order to further extract interpretable information, most of the methods exploit the idea of using sparse linear factor models. In particular, the resulting latent factor is restricted to contribute to variation in only a subset of the observed features. For example, sparse factor loadings in gene expression data analysis can be interpreted as non-disjoint clusters of co-regulated genes (Pournara and Wernisch, 2007;Lucas et al., 2010;Gao et al., 2013). Limitations of those existing sparse methods for the application in real data scenarios are scalability and the inability to handle nonlinear time-dependent complex structures (Ainsworth et al., 2018).
The variational auotencoders (VAE) is a powerful deep generative learning technique for efficient inference (Kingma and Welling, 2014;Rezende et al., 2014). VAE learns a lower dimensional representation of the data through an encoder, and then with a decoder transforms the latent representation back to the original data space. VAE is flexible and can account for any kind of data. However, the original VAE ignores the temporal correlations across latent dimensions.
Contributions Our motivation lies in the study of high-dimensional temporal multiview data. We seek to infer trajectories of latent variables that provide insights into the latent, lower-dimensional structure derived from the dynamics of the observed data space. Motivated by the success of variational recurrent neural network (VRNN) for modeling temporal sequence data, we propose a new modeling strategy that integrates VRNN into sparse group factor analysis. We label this model as the interpretable representation learning for temporal multi-view data (ITM-VAE). The resulting model, thus serves as a nonlinear factor model for multi-view data observed across time. Our main contributions can be summarized as follows: • We build a novel interpretable model that can perform sensible disentanglement for temporal multi-view data. The ability of ITM-VAE to perform sensible disentanglement is through the introduction of a view and time-specific transformation W (introduced in Section 4) which is built on a sparsity inducing prior between the time-specific latent representation and the view and time-specific neural generator.
• We derive an efficient timestep-wise variational inference scheme for learning temporal multi-view data.
• We show that ITM-VAE can learn dynamic dependency among views. This is an appealing feature of ITM-VAE because most of the complex systems depend on a temporal component, and such a component contributes to the development of variable interactions gradually. The ability to access the dynamic relationship of groups of variables will help us gain insight for downstream analysis of the complex data.

Related Work
A few extensions have been proposed for VAE to model the correlations in the latent space. The conditional VAE (CVAE) (Sohn et al., 2015) is a graphical model, and its input observations modulate the prior on Gaussian latent variables that generate the outputs. However, CVAE cannot model the individual sample-specific temporal structure (Sohn et al., 2015). GPPVAE (Casale et al., 2018) combines the VAE and the Gaussian process (GP) prior over the latent space to model the temporal dependencies between samples. Due to the restrictive nature of the view-object GP product kernel, GPPVAE cannot capture the individual-specific temporal structure. An extension of GPPVAE, GP-VAE (Fortuin et al., 2020), is designed specifically for data imputation. GP-VAE places an independent GP prior on each individual sample's time-series to relax the inference technique. A limitation of GP-VAE is that GP-VAE cannot capture the shared temporal structure across all data points. DP-GP-LVM is a nonparametric Bayesian latent variable model that aims to learn the dependency structures of multimodal data by the GP prior (Lawrence et al., 2019). GP prior is shown to be well suited for time series modeling, however, it comes at the cost of inverting the kernel matrix, which has a time complexity of O(d 3 ), where d is the dimensionality of the data. Moreover, it is often a challenge to design a kernel function that can accurately capture both the correlation in feature space and in a temporal dimension. The recurrent neural networks (RNNs) (Martens and Sutskever, 2011;Hermans and Schrauwen, 2013;Pascanu et al., 2013;Graves, 2013) have shown good performance in modeling sequence data, where the latent random variables in the RNN function serve as "memory" of the past sequence. RNN can be further extended to integrate the dependencies between the latent random variables at neighboring time steps, called variational recurrent neural network (VRNN) (Chung et al., 2015) in the context of VAE. VRNN can handle complex nonlinear and highly structured sequential data, Output interpretable VAE (oi-VAE) is designed for non-temporal grouped data with a structured VAE comprised of group-specific generators (Ainsworth et al., 2018). The latent variables are shared across all groups and are assumed to be iid for each data point. Because of the group design of oi-VAE, its model interpretation is limited to factor level. Our proposed ITM-VAE is a temporal extension of oi-VAE with a feature level interpretation.

Background
Generative Model: In generative models as shown in Fig. 1, the class of VAEs are popular for efficient approximate inference and learning (Kingma and Welling, 2014). VAE approximates intractable posterior distributions over latent representations that are parameterized by a deep neural network, which maps observations to a distribution over latent variables.
For non-sequential data, VAE has become one of the most popular approaches for efficiently recovering complex multimodal distributions. Recently, VAE has been extended to dynamic systems (Archer et al., 2015). Briefly, VAE provides a mapping from the observations to a distribution on their latent representation. The resulting simpler latent subspace can be used to describe the underlying complex system. Mathematically, let x ∈ R d denote a d-dimensional observation and z ∈ R k denote a vector of latent random variables of fixed dimension k with k < d. The generative process of VAE can be represented as: where I is the identity matrix, D is a d×d diagonal matrix whose diagonals are the marginal variances of each component of x, and µ x is the mean of the Gaussian likelihood which is produced by a neural network with parameters θ taking z as an input. Then the joint distribution is defined as: p(x, z; θ) = p(x | z; θ)p(z).
(2) Learning Inference: One unique feature of VAE is that it allows the conditional distribution p(x | z) to be a potentially highly nonlinear mapping from z to x.
The likelihood is then parameterized with a generative network (called decoder). VAE uses q(z|x) with an inference network (called encoder) to approximate the posterior distribution of z. For example, q(z|x) can be a Gaussian N (µ, σ 2 I), where both µ and σ 2 are parameterized by a neural network: [µ, log σ 2 ] = f φ (x), where f φ is a neural network with parameters φ. The parameters for both generative and inference networks are learned through variational inference, and Jensen's inequality yields the evidence lower bound (ELBO) on the marginal likelihood of the data: where KL(Q P ) is the Kullback-Leibler divergence between two distributions Q and P . q(z; φ) is a tractable variational distribution meant to approximate the intractable posterior distribution p(z | x); it is controlled by some parameters φ. We want to choose φ that makes the bound in Eq. (3) as tight as possible, φ * arg max φ L(x; θ, φ).
One can train a feedforward inference network to find good variational parameters φ(x) for a given x, where φ(x) is the output of a neural network with parameters φ that are trained to maximize L(x; θ, φ(x)) (Kingma and Welling, 2014).

ITM-VAE model
Let x 1:T = (x 1 , ..., x T ) denote the sequence data observed at T timesteps. We then rewrite x t , which is the sequence data at timestep t, to incorporate the view information as denote the data from the gth view at time t and G is the number of views, each view has the dimension of d g .

Modeling Framework
Prior Given a temporal sequence of vectors x 1:T = (x 1 , ..., x T ), x t ∈ R d , the conventional VAEs assume an independent latent variable z for each timestep t: z ∼ N (0, I). To encode temporal variability, we propose to allow the latent variable z t at timestep t to depend on the state variable h t−1 of an RNN though the following distribution: where both µ 0,t and σ 0,t are produced by a distinct neural network that approximates the time-dependent prior distribution (Chung et al., 2015). More specifically, [µ 0,t , σ 2 0,t ] = ϕ prior τ (h t−1 ), and ϕ prior τ (h t−1 ) denotes a neural network taking the previous hidden state h t−1 as input.
Encoder Similar to the VAEs, we need to define an approximate posterior q(z|x). We propose to let z t capture the shared variability among views at each timestep by allowing q(z|x) as a function of both and h t−1 as: Decoder The generation of x t will depend on both z t and h t−1 . In addition, we propose to model different views of data independently while allowing the latent variable z t to be shared across G views at timestep t. The corresponding generative distribution will be: where D (g) x,t is a diagonal matrix. We then introduce a sequence of latent matrices W (g) t ∈ R dg×k , for t = 1 : T, g = 1 : G. W will help with model interpretation by placing a column-wise sparsity prior which will be introduced in section 4.1: Model Interpretability. Both parameters µ (g) x,t and D (g) x,t will be conditioned on W (g) t , z t and h t−1 through: where ϕ dec θt,g denotes a neural network with parameters θ t,g , and diag(D) denotes the diagonal elements of the matrix D.
Recurrence The hidden state h t is updated by conditioning on z t in a recurrent way: , where S is the transition function which can be implemented with gated activation functions such as long short-term memory or gated recurrent unit (Cho et al., 2014;Hochreiter and Schmidhuber, 1997). VRNN demonstrates that including feature extractors in the recurrent equation is important for learning complex data: where ϕ x τ and ϕ z τ are two neural networks for feature extraction from x t and z t , respectively. By the above model specifications, the generative distribution can be factorized as: The ITM-VAE model structure is depicted in Fig. 2.  5); (c) Updating RNN hidden states using Eq. (9), and (d) Generating function using Eqs. (7) and (8).
Model Interpretability Similar to oi-VAE (Ainsworth et al., 2018), we use a spare prior (Kyung et al., 2010) on the weight matrix to achieve interpretable results. We place a column-wise sparsity prior for W (g) t to ensure the interpretability of the model: where d g denotes the number of features for each view g. Note that our prior specification is different from oi-VAE, which assumes each view's input dimensions are the same, and that is the reason why oi-VAE is limited in factor level interpretation and cannot track the feature sparsity for downstream analysis. The parameter λ controls the model sparsity. More specifically, a larger value of λ implies more column-wise sparsity in W (g) t:,j . Marginalizing over γ 2 gjt induces group sparsity over the columns of W (g) t:,j . Hence, the model automatically tracks the sparse features among groups through time.

Timestep-wise Learning
The traditional VAEs are learned by optimizing the ELBO using stochastic gradient methods. We are more interested in the sparsity of the learned W (g) t:,j for model interpretability. The sparsity inducing prior on W (g) t:,j is marginally equivalent to the convex group lasso penalty. Hence, we propose to adapt the idea of collapsed variational inference (Ainsworth et al., 2018) to obtain the true sparsity of the columns W 1:T , · · · , W 1:T , γ 2 = γ 2 1:G,1:K,1:T , x = x 1:T , and z = z 1:T .

Optimization
Parikh and Boyd (2014) proposed the proximal gradient descent algorithms which are a broad class of optimization techniques for separable objectives with both differentiable and potentially non-differentiable components, where g(x) is differentiable and h(x) is potentially non-smooth or non-differentiable. oi-VAE (Ainsworth et al., 2018) stated that collapsed variational inference with proximal updates provided faster convergence and succeeded in identifying sparser models than other techniques. We chose to use proximal gradient descent updates on our temporal latent-togroup matrices W (g) t:,j for timestep-wise learning like oi-VAE did on their latent-to-group matrices W (g) .j . In our scenario, we define where s denotes the sth iteration, λ s > 0 is a step size, and prox f (x) is the proximal operator for the function f . Expanding the definition of prox λ s g , we can show that the proximal step corresponds to minimizing g(x) plus a quadratic approximation to g(x) centered on x s . For f : R n → R and g : R n → R ∪ {+∞} are closed proper convex and f is differentiable. For g(W

Methods Considered
In addition to the ITM-VAE 1 , we also consider VAE (Kingma and Welling, 2014), adapted conditional VAE (CVAE) (Sohn et al., 2015), oi-VAE (Ainsworth et al., 2018), VRNN (Chung et al., 2015), and GP-VAE (Fortuin et al., 2020). VAE and oi-VAE: We concatenate data across different timesteps and treat the concatenated data independent. CVAE: CVAE requires the label information as input to both the encoder and decoder networks. For our unsupervised problem, we assign different t for the data as its label information. More specifically, the input for encoder is [X t ,t] and the input for decoder is [Z t ,t]. Evaluation metrics To check and validate how well the disentanglement is achieved, we propose to visualize the W (g) t:,j matrix at different timesteps t and quantitatively compare the MSE[test] (mean squared error on the test data) with alternative methods.
Selection on λ and k The parameter λ controls the model sparsity, larger λ will imply more column-wise sparsity in W (g) t:,j , we propose to select λ based on the learned W (g) t:,j (Fig. 3a) to check the sparsity and the MSE[test] (Fig. 3b). The latent dimension k is chosen based on interpretation purpose.

Artificial Data
Setup In order to visualize the performance, we generate one-bar images. The row position of the bar was taken as different time point labels, starting from the first row as time point t = 1 to the last row as time point t = 8. Thus, there are in total T = 8 time points.
Dataset Generation We generate 2000 8×8 one-bar images and we add normal random noises with mean 0 and standard deviation 0.05 to the entire image. We randomly select 80% (n=1600) of the image for training, 10% (n=200) for validation and 10% (n=200) for testing. For each batch, we use batch size 64. Therefore, the data structure is of 8 × 64 × 64 for ITM-VAE. In order to associate each dimension of z with a unique row in the image, we chose k = 8.
Results For ITM-VAE and CVAE, we calculate the MSE[test] on the concatenated time points of test data which is the same for the rest of the experiments. The results are in Fig. 4 and ITM-VAE yields the lowest MSE[test] comparing to other competitors Table. 1. VRNN and CVAE perform similarly and close to ITM-VAE. However, GP-VAE and oi-VAE do not perform very well. VAE has the largest MSE. These results indicate that ITM-VAE can learn multivariate time series data and reconstruct them very well. Meanwhile, we randomly select 64 images for each batch and replicate each image 20 times (T = 20) to represent the perfect time series structure, the data structure is of 20 × 64 × 64. We showed the learned W

Motion Capture Data
Setup We consider the motion capture data obtained from CMU (http://mocap.cs.cmu.edu) to evaluate ITM-VAE's ability to handle complex longitudinal multivariate data. We use subject 7 data, which contains 11 trials of standard walking and one brisk walking recordings from the same person. For each trial, it contains different time frames of the person's moving skeleton, and it measures 59 joint angles split across 29 distinct joints. In this setting, we treat each distinct joint as a view, and each joint has 1 to 3 observed degrees of freedom to represent the different group dimensions. The task is to evaluate the model's ability for sensible dynamic disentanglement, model interpretability and generalization ability.
Data For model training, we use the data from 1 to 10 trials. In total, the 10 trials training data have 3776 frames. For testing, we use the 11th trial data which has 315 frames. We set T = 32 to train the model with batch size 32. For each batch, the data structure is of 32 × 32 × 59 for ITM-VAE.
Results To check different latent dimensional effects of z, we train ITM-VAE on k = 4, 8, and 16. Fig. 5 shows the results for k = 8. ITM-VAE displays lower MSE[test] than all the other competitors in Table. 1. From the MSE[test], we can see that CVAE, VRNN, and oi-VAE perform very similarly with each other. This demonstrates that even oi-VAE is not a temporal model, its model structure can still capture the complex variations and our ITM-VAE is a temporal extension of oi-VAE.
To evaluate the generative ability of ITM-VAE, we show the reconstructed images of trial 11 in Fig. 5 bottom row. The hidden dynamic information extracted from ITM-VAE generates very natural poses of human walking. In fact, there is clearly a moving pattern from the head to foot between neighboring timesteps. On the other hand, the results obtained from oi-VAE, which treats each time frame data independently, are very similar among each other, and there is no obvious trend in CVAE either. Since GP-VAE cannot capture the shared temporal structure, it fails to capture some details of the trend in foot and hand. We further compare the test-loglikehood between ITM-VAE and oi-VAE on trial 11 and trial 12, which is the brisk walk data. Table. 3 records the log-likelihood for both ITM-VAE and oi-VAE models on two testing trials with k = 4, 8. ITM-VAE has higher test log-likelihood and both methods achieve higher test log-likelihood when the latent dimension k is larger. This indicates that ITM-VAE can achieve better generalization than oi-VAE, because the brisk walking trial is very different from the training walking trials. Fig. 6 shows that the factors change across different time points. For example, from time point 1 to 3, the first factor (first column of the left and middle images) changes from lfoot (left foot) to rfoot (right foot), factor 2 changes from rwrist (right wrist) to thorax, and factor 7 changes from rwrist to rtibia (right tibia). These changes are indeed reasonable because when we start to walk with the foot, the tibia and the thorax move accordingly (Versichele et al., 2012). The above observation demonstrates that the learned latent representation from ITM-VAE has an intuitive anatomical interpretation for different time points. We also provided a detailed list of the joints per latent variable dimension that are most strongly influenced by each factor in Table. 2. For example, factor 1 represents foot and lower back, factor 2 represents wrist, thorax and upper back, and factor 8 represents wrist, foot and hand. All these observations demonstrate that ITM-VAE can track the dynamic latent embeddings and provide meaningful interpretation.

Metabolomic Data
Setup In this section, we propose to analyze the data obtained from a longitudinal study (Jozefczuk et al., 2010), where one of the objectives is to compare metabolic changes of E.coli response to five different perturbations: cold, heat, oxidative stress, lactose diauxie, and stationary phase. The task is to evaluate the model on limited sample size studies which is common in the life sciences field. Data The dataset contains 196 metabolite expression values measured for 8 subjects at 12 different time points under five stress conditions. We treat each condition as a group and randomly select 6 subjects as the training set and the remaining 2 subjects as the test  set. We use batch size n = 2, and for each batch, the data structure is of 12 × 2 × 980 for ITM-VAE.
Results ITM-VAE has the lowest MSE[test] in Table. 1 (last column). GP-VAE and VRNN perform better than CVAE and other non-temporal competitors. This result further demonstrates that the generative model structure of ITM-VAE and oi-VAE can handle the limited sample size problems better than traditional VAEs, because oi-VAE is non-temporal model so it performs worse than VRNN and GP-VAE. The learned group-weights W (g) t:,j from ITM-VAE are shown in Fig. 7a-b. For ITM-VAE, it is clear that at time t = 3, most of the factors' variations are explained by cold and heat groups, at time t = 5, cold group explains most of the variations. These results are consistent with the findings in the original paper (Jozefczuk et al., 2010). Another important downstream analysis is the inspection of top features with largest weight: The loadings can give insights into the biological process underlying the heterogeneity captured by a latent factor. There might be scale differences among groups, the weights of different views are not directly comparable. For simplicity, we scale each weight vector by its absolute value. In Fig. 7c and d, we plotted the top 5 metabolites with largest absolute weight of two interesting factors. Like the top feature Norvaline in Fig. 7c is known to promote tissue regeneration and muscle growth (Ming et al., 2009), and to become a precursor in the penicillin biosynthetic pathway. We leave more detailed interpretation for biological interest.

Discussion
We develop an interpretable nonlinear framework for temporal multi-view data, namely ITM-VAE, with the goal of disentangling the dynamically shared latent embeddings for the complex multi-view varations. One key feature of ITM-VAE is its ability to integrate the VRNN to the shared latent variables among different groups in order to model the complex sequence data and extract the dependency relationships.
Our empirical analyses on both motion capture and metabolomics data demonstrate that ITM-VAE can successfully extract the hidden time dependence structures. More importantly, the achieved model efficiency and interpretability does not occur at the cost of model generalization. Because ITM-VAE can model complex temporal multi-view data and result in interpretable results, we believe ITM-VAE will have wide applications in different fields.
After feature extraction from x t and z t , then, we stack x t and z t with h t−1 together for the inference and generative model respectively.

Figure 8:
Additional results of ITM-VAE on artificial data. k = 8, λ = 5, iteration = 10000, batch-size = 64, t = 8. Here, we didn't assign the t label to each image since we want to check the model sparsity induced by λ. true image: the training image, Z: the sampled z values from encoder, X mu: the decoder mean, X sigma: the decoder sigma, sample: the reconstructed image from decoder mean and sigma, learned weights: the learned weights from the model.

A.2 Experimental Details
We ran Adam for the inference and generative net parameters optimization with learning rate 1e-3. Proximal gradient descent was run on W t with learning rate 1e-4. Artificial data: We chose λ = 5. For the first part of experiment, we want to select λ so we randomly selected 64 images at each iteration and replicate each image 20 times as one batch, we ran for 10,000 iterations. The data structure is 20 × 64 × 64. For the second part of experiment, we assign the row position of bar as the time label, so in total we have t = 8 different types of images, the data structure for each batch is 8×64×64. Motion capture data: We chose λ = 5, and we used T = 32 frames and replicate each frame 32 times to stack as one batch ( 32 × 32 × 59) to train our model, optimization was run for 100 epochs. Metabolomic data: We chose λ = 10, we randomly selected n = 2 as one batch, the data structure for each batch is 12 × 2 × 980, we ran 10,000 epochs.