Abstract
Abstract-In this contribution we present a method that extends the Canonical Correlation Analysis for two groups of variables to the case of multiple conditions. Contrary to the extensions in literature based on augmenting the number of variable groups, the addition of conditions allows for a more robust estimate of the canonical correlation structure inherently present in the data. Algorithms to solve the estimation problem are based on joint approximate diagonalization algorithms for matrix sets. Simulations show the performance of the proposed method under two different scenarios: the calculation of a latent canonical structure and the estimation of a bilinear mixture model.
I. INTRODUCTION
In this work we present a method to extend the classical Canonical Correlation Analysis (CCA) of Hotelling for a single condition on two groups of variables [5] to the case of multiple conditions for two groups of variables. While a lot of work has focused on the extension of Hotelling's original proposal to multiple groups of variables, only little work has been carried out with respect to the multiplication of the conditions. However, many applications in engineering behave under this model, such as simultaneous recordings in two heterogeneous measurement spaces. Examples may be the simultaneous recordings of extracranial and intracranial electroencephalographic or electrocardiographic data or even the simultaneous recording of electroencephalographic data and the GAZE direction. In this contribution we aim at sketching the major contributions to CCA during the last decades and we propose a variant as to include the multiple conditions. Simulation studies will show the performance of the proposed algorithm under varying conditions.
II. METHODS

A. Canonical Correlation Analysis
Canonical Correlation Analysis (CCA) is one of the most useful methods for describing linear relationships between (the scores of) two groups of variables. Originally proposed in Hotelling's papers [5] , it has ever since received much attention in psychometrics, chemometrics and other scientific domains related to the search for explanatory, latent variables underlying multiple observations. The basic CCA model for 
A. Canonical Correlation Analysis
where P i is an orthogonal, unit L 2 -norm basis for X i , which can be obtained as P i = W T i X i through, e.g., a singular value decomposition of X i . In other words, the CCA model strives to find those weighting vectors t i that maximise the correlation
A. Canonical Correlation Analysis
From the seminal work of Hotelling on, many efforts have been conducted to generalize model (1) to multiple observation sets. Horst [4] maximized the average correlation coefficient between linear combinations, i.e.,
A. Canonical Correlation Analysis
and in the works of Carroll [2] we find an explicit maximization of the correlation between P i t i and the auxiliary variable z by maximizing the function
A. Canonical Correlation Analysis
In fact, the maximizer of the above is the eigenvector
A. Canonical Correlation Analysis
−1 X i associated with its largest eigenvalue. An effort to regroup the above and similar algorithms using a limited number of representative functions and a single algorithmic solution can be found in [3] . But all of the above methods deal with the presence of multiple groups of data and only little attention has been given to the observation of two groups of variables under several conditions.
A. Canonical Correlation Analysis
In addition, all of the previously introduced algorithms use a summing over the different correlation matrices. Unfortunately, by summing over the different correlation structures, information may go lost. Recent advances in (multi-)linear algebra and signal processing have shown that the joint diagonalization of the matrix set containing all correlation structures offers a more stable solution. This can be seen from the fact that the joint diagonalization of the matrix set is less biased than the diagonalization of the sum, since a perfect diagonalization of the sum R also diagonalizes the imperfections due to the integrated noise. A simple example may illustrate this phenomenon:
A. Canonical Correlation Analysis
Example 1 (Matrix sum diagonalization): Take two (semi-positive definite) random symmetric matrices A, B ∈ R 2×2 and calculate a diagonalization of their sum. Now, by simple calculations we see that
A. Canonical Correlation Analysis
, where w i is the i-th column of W. But the inverse does not generally hold true! Actually, the conditions on the sum are necessary but not sufficient for the conditions on the individual terms to hold.
A. Canonical Correlation Analysis
The diagonalization of a single matrix R has another important drawback, namely, the rotational ambiguity. It can be easily proved that if W diagonalizes R as Λ = W T RW, then so do all T = WQ, where Q is an arbitrary orthogonal matrix. For the above example, this means that there does exist a matrix T = WQ that diagonalizes the sum (A + B) and A and by consequence also B. However, this does not extend to the case of more than two matrices, since it cannot be guaranteed that the eigenvalues remain real. This is one of the reasons why joint approximate diagonalization of a matrix set has received that much attention in the signal processing literature over the last decades. In what follows, we will show how the joint approximate diagonalization can also be employed for canonical correlation analysis when multiple observations are available of the same two groups of variables under various conditions.
B. Multiple Condition Canonical Correlation Analysis
Remark that maximizing the correlation of the observed sets with an auxiliary variable z does no longer make sense when we consider multiple conditions. Indeed, the response of a variable may change from condition to condition and it is no longer of primordial interest to maximize this correlation. Denote by R yy by requiring that the sets {W
B. Multiple Condition Canonical Correlation Analysis
yy W y } k are approximately diagonal. However, this does not guarantee the approximate joint diagonality of the set {W x R (k) xy W y } k . As a consequence, we must impose the following (∀k):
B. Multiple Condition Canonical Correlation Analysis
xy the appropriate diagonal matrices. In what follows, we will drop the superscript · (k) to facilitate reading. All expressions should be understood ∀k.
B. Multiple Condition Canonical Correlation Analysis
Defining The joint approximate diagonalization may now alternatingly be carried out on the two sets {R
B. Multiple Condition Canonical Correlation Analysis
yy }. To that purpose one can choose one of the available joint approximate diagonalization algorithms readily available in the literature such as [1] , [8] , [6] . We propose to alternatingly run one step of the chosen algorithm on each of the sets to avoid getting stuck in local minima.
C. Weighted Multiple Condition Canonical Correlation Analysis
Actually, the above trick is reminiscent to the trick used in [7] for spatio-temporal blind source separation. Let us, in analogy to [7] , introduce the weighting value α, which balances between an independent joint approximate diagonalization in each of the measurement spaces and the diagonalization of the cross-covariance matrices. In other words, the sets of diagonalization matrices take the form {αR
C. Weighted Multiple Condition Canonical Correlation Analysis
yy }. Remark that the above paragraph considered the form α = 0.5.
D. A Welcome Byproduct
Suppose we consider the measurement spaces as the two dimensions in which we recorded e.g. an event-related potential (ERP). The different conditions then simply reduce to the different trials under which we have recorded the ERP. Our signals may be represented as X (k) = R xy , where x accounts for the observations along the spatial dimension and y for the observations along the dimension of relative time after stimulation onset. The matrices R xx and R yy can then simply be obtained by marginalisation over the temporal, respectively the spatial dimension as R xx = E{XX T } and R yy = E{X T X}. The joint diagonalization of R yy , R xx and R xy is then closely related to an approximate singular value decomposition of the matrix set {X (k) }, possibly with non-orthogonal changes of bases (if we choose a nonorthogonal joint diagonalization algorithm, e.g. [8] , [6] ).
III. SIMULATIONS
Since we would like to display a wide variety of possible applications, we have chosen two scenarios as the bases for our simulations. All simulation suppose M = 3, N = 5 and K = 25. For each observation k we have a population of 10 3 samples. We used a non-orthogonal fast approximate joint diagonalization algorithm of Tichavský and Yeredor [8] alternating the iterations between the two sets {(1 − α)R xx , αR xx } and {(1 − α)R yy , αR yy } and repeated the experiment over 100 Monte Carlo realisations.
III. SIMULATIONS
In both the scenarios we investigate the diagonality (up to permutation and scale) of the final matrices W T x R xx W x , W T y R yy W y and W T x R xy W y using the Moreau-Amari performance index for a matrix G ∈ R D×D defined as
III. SIMULATIONS
For the rectangular matrices W T x R xy W y we calculate only the term corresponding to its smallest dimension, i.e., for
A. Scenario 1
The first scenario is the case where the observed variables behave just as in the model, i.e., variable (groups) x and y are created such that R xx and R yy are the identity matrices. The variables are then observed by the user through a linear mixture up to some additive noise as
A. Scenario 1
x , where n x is a vector of the dimension of x containing unit variance Gaussian noise and U x ∈ R M ×M is a non-degenerate mixing matrix. The observations
A. Scenario 1
y are created analogously and we calculate for each k the corresponding covariance and cross-covariance matrices. The results are given in Figure 1 .
A. Scenario 1
It is clear from the measure P I(W T x R xy W y ) that the influence of α cannot be neglected. Indeed, when α = 0 -i.e. neglecting the cross-covariance structure -, the diagonalization of W T x R xy W y in the noiseless case is only approximate, whereas for α > 0 it is quasi-exact.
B. Scenario 2
In this scenario we consider the observations R
B. Scenario 2
and calculate the marginalized covariances as described in paragraph II-D. The matrices U x and U y are also called the mixing matrices and its entries are drawn according to the normal distribution with zero mean and unit variance. The matrix ∆ (k) is a diagonal matrix with entries drawn from the same normal distribution. Next to the Moreau-Amari performance index on the resulting approximately diagonalized matrices we also calculate the index for the matrices W T x U x and W T y U y , which describes how well we can estimation our model. The results are given in Figure 2 .
B. Scenario 2
It is particularly interesting that in scenario 2, except for low noise, it seems that the influence of parameter α is inverted with respect to scenario 1. For higher noise levels, it seems that the exclusion of R xy -both in terms of diagonalization as in terms of model estimation -slightly augments the performance. This is contradictory to what is generally accepted, in that the singular value decomposition of the signal is judged more stable than the left or right eigenvalue decompositions of the covariance matrices. 
IV. CONCLUSIONS
The extension of the canonical correlation analysis to multiple observations of two groups of variables seems a promising approach for signal processing applications. When estimating the latent canonical structure in the data, multiple observations may lower the impact of noise on the estimates. The proposed method allows for the estimation of the nonorthogonal matrices exposing the inherent correlation structure without a required pre-whitening of the observations. In addition, by introducing a weighting parameter, one may balance between the covariance and the cross-covariance diagonalization. At last, when the diagonalization of an observed matrix structure and its marginalized covariances is envisaged, the method is reminiscent to an approximately joint singular value decomposition.
IV. CONCLUSIONS
Future works should focus on the application of the algorithm to joint recordings of GAZE and EEG signals, withR ∈ R N ×N an orthogonal matrix. The entriesR ij contain the values E{x i y j } and since E{x i x j } = 0, E{y i y j } = 0∀i = j. We have that each row and each column ofR can contain at maximum one entry |R ij | > cos(π/4) = 1/ √ 2. Suppose now that ∃j 1 : |E{x m y j1 }| ≈ |E{x m ′ y j1 }| ≈ 1/ √ 2, then all other entries E{x i y j1 } ≈ 0, ∀i = m, m ′ . TakingRR T = Λ, we see that Λ mm > 1/2 and Λ mm ′ > 1/2. Analogous reasoning brings us to ∃i 1 : |E{x i1 y n }| ≈ |E{x i1 y n ′ }| ≈ 1/ √ 2 ⇒ E{x i1 y j } ≈ 0, ∀j = n, n ′ .Thus, Λ is not diagonal under the above assumptions and at least one of the assumed entries must be much smaller than 1/ √ 2.

{'paper_id': '11523541', 'title': 'An extension of the Canonical Correlation Analysis to the case of multiple observations of two groups of variables', 's2_url': 'https://api.semanticscholar.org/CorpusID:11523541', 'source': 'S2ORC'}