A Hierarchical Latent Variable Model for Data Visualization.
AbstractVisualization has proven to be a powerful and widely-applicable tool for the analysis and interpretation of multivariate data. Most visualization algorithms aim to find a projection from the data space down to a two-dimensional visualization space. However, for complex data sets living in a high-dimensional space, it is unlikely that a single two-dimensional projection can reveal all of the interesting structure. We therefore introduce a hierarchical visualization algorithm which allows the complete data set to be visualized at the top level, with clusters and subclusters of data points visualized at deeper levels. The algorithm is based on a hierarchical mixture of latent variable models, whose parameters are estimated using the expectation-maximization algorithm. We demonstrate the principle of the approach on a toy data set, and we then apply the algorithm to the visualization of a synthetic data set in 12 dimensions obtained from a simulation of multiphase flows in oil pipelines, and to data in 36 dimensions derived from satellite images. A Matlab software implementation of the algorithm is publicly available from the World Wide Web.
Introduction
Many algorithms for data visualization have been proposed by both the neural computing
and statistics communities, most of which are based on a projection of the data onto a two-dimensional
visualization space. While such algorithms can usefully display the structure
of simple data sets, they often prove inadequate in the face of data sets which are more
complex. A single two-dimensional projection, even if it is non-linear, may be insufficient
to capture all of the interesting aspects of the data set. For example, the projection which
best separates two clusters may not be the best for revealing internal structure within
one of the clusters. This motivates the consideration of a hierarchical model involving
multiple two-dimensional visualization spaces. The goal is that the top-level projection
should display the entire data set, perhaps revealing the presence of clusters, while lower-level
projections display internal structure within individual clusters, such as the presence
of sub-clusters, which might not be apparent in the higher-level projections.
Once we allow the possibility of many complementary visualization projections, we can
consider each projection model to be relatively simple, for example based on a linear
projection, and compensate for the lack of flexibility of individual models by the overall
flexibility of the complete hierarchy. The use of a hierarchy of relatively simple models
offers greater ease of interpretation as well as the benefits of analytical and computational
simplification. This philosophy for modelling complexity is similar in spirit to the "mixture
of experts" approach for solving regression problems [1].
The algorithm discussed in this paper is based on a form of latent variable model which
is closely related to both principal component analysis (PCA) and factor analysis. At
the top level of the hierarchy we have a single visualization plot corresponding to one
such model. By considering a probabilistic mixture of latent variable models we obtain
a soft partitioning of the data set into 'clusters', corresponding to the second level of the
hierarchy. Subsequent levels, obtained using nested mixture representations, provide successively
refined models of the data set. The construction of the hierarchical tree proceeds
top down, and can be driven interactively by the user. At each stage of the algorithm
the relevant model parameters are determined using the expectation-maximization (EM)
algorithm.
In the next section we review the latent-variable model, and in Section 3 we discuss the
extension to mixtures of such models. This is further extended to hierarchical mixtures in
Section 4, and is then used to formulate an interactive visualization algorithm in Section 5.
We illustrate the operation of the algorithm in Section 6 using a simple toy data set. Then
we apply the algorithm to a problem involving the monitoring of multi-phase flows along
oil pipes in Section 7 and to the interpretation of satellite image data in Section 8. Finally,
extensions to the algorithm, and the relationships to other approaches, are discussed in
Section 9.
Latent Variables
We begin by introducing a simple form of linear latent variable model and discuss its
application to data analysis. Here we give an overview of the key concepts, and leave the
detailed mathematical discussion to Appendix A. The aim is to find a representation of
a multi-dimensional data set in terms of two latent (or 'hidden') variables. Suppose the
data space is d-dimensional with coordinates y and that the data set consists of
a set of d-dimensional vectors ft n g where . Now consider a two-dimensional
latent space together with a linear function which maps the latent space
into the data space
where W is a d \Theta 2 matrix and - is a d-dimensional vector. The mapping (1) defines
a two-dimensional planar surface in the data space. If we introduce a prior probability
distribution p(x) over the latent space given by a zero-mean Gaussian with a unit covariance
matrix, then (1) defines a singular Gaussian distribution in data space with mean
- and covariance matrix h(y . Finally, since we do not expect
the data to be confined exactly to a two-dimensional sheet, we convolve this distribution
with an isotropic Gaussian distribution p(tjx; oe 2 ) in data space having a mean of zero and
covariance oe 2 I where I is the unit matrix. Using the rules of probability, the final density
model is obtained from the convolution of the noise model with the prior distribution over
latent space in the form
Z
Since this represents the convolution of two Gaussians, the integral can be evaluated
analytically, resulting in a distribution p(t) which corresponds to a d-dimensional Gaussian
with mean - and covariance matrix WW T
If we had considered a more general model in which conditional distribution p(tjx) is given
by a Gaussian with a general covariance matrix (having d independent parameters) then
we would obtain standard linear factor analysis [2, 3]. In fact our model is more closely
related to principal component analysis, as we now discuss.
The log likelihood function for this model is given by
likelihood can be used to fit the model to the data and hence determine values for the
parameters -, W and oe 2 . The solution for - is just given by the sample mean. In the
case of the factor analysis model, the determination of W and oe 2 corresponds to a non-linear
optimization which must be performed iteratively. For the isotropic noise covariance
matrix, however, it was shown by Tipping and Bishop [4, 5] that there is an exact closed
form solution as follows. If we introduce the sample covariance matrix given by
then the only non-zero stationary points of the likelihood occur for:
where the two columns of the matrix U are eigenvectors of S, with corresponding eigen-values
in the diagonal matrix  , and R is an arbitrary 2 \Theta 2 orthogonal rotation matrix.
Furthermore, it was shown that the stationary point corresponding to the global maximum
of the likelihood occurs when the columns of U comprise the two principal eigenvectors
of S (i.e. the eigenvectors corresponding to the two largest eigenvalues) and that all other
combinations of eigenvectors represent saddle-points of the likelihood surface. It was also
shown that the maximum-likelihood estimator of oe 2 is given by
d
which has a clear interpretation as the variance 'lost' in the projection, averaged over the
lost dimensions.
Unlike conventional PCA, however, our model defines a probability density in data space,
and this is important for the subsequent hierarchical development of the model. The choice
of a radially symmetric rather than a more general diagonal covariance matrix for p(tjx)
is motivated by the desire for greater ease of interpretability of the visualization results,
since the projections of the data points onto the latent plane in data space correspond (for
small values of oe 2 ) to an orthogonal projection as discussed in Appendix A.
Although we have an explicit solution for the maximum-likelihood parameter values, it was
shown by Tipping and Bishop [4, 5] that significant computational savings can sometimes
be achieved by using the following EM (expectation-maximization) algorithm [6, 7, 8].
Using (2) we can write the log likelihood function in the form
Z
in which we can regard the quantities x n as missing variables. The posterior distribution of
the x n , given the observed t n and the model parameters, is obtained using Bayes' theorem
and again consists of a Gaussian distribution. The E-step then involves the use of 'old'
parameter values to evaluate the sufficient statistics of this distribution in the form
I is a 2 \Theta 2 matrix, and h i denotes the expectation computed with
respect to the posterior distribution of x. The M-step then maximizes the expectation of
the complete-data log likelihood to give
f
e
Nd
Whx
in which e denotes 'new' quantities. Note that the new value for f
W obtained from (9) is
used in the evaluation of oe 2 in (10). The model is trained by alternately evaluating the
sufficient statistics of the latent-space posterior distribution using (7) and (8) for given
oe 2 and W (the E-step), and re-evaluating oe 2 and W using (9) and (10) for given hx n i
and hx n x T
(the M-step). It can be shown that, at each stage of the EM algorithm,
the likelihood is increased unless it is already at a local maximum, as demonstrated in


Appendix

For N data points in d dimensions, evaluation of the sample covariance matrix requires
any approach to finding the principal eigenvectors based on an
explicit evaluation of the covariance matrix must have at least this order of computational
complexity. By contrast, the EM algorithm involves steps which are only O(Nd). This
saving of computational cost is a consequence of having a latent space whose dimensionality
(which, for the purposes of our visualization algorithm, is fixed at 2) does not scale with
d.
If we substitute the expressions for the expectations given by the E-step equations (7) and
(8) into the M-step equations we obtain the following re-estimation formulae
f
e
d
which shows that all of the dependence on the data occurs through the sample covariance
matrix S. Thus the EM algorithm can be expressed as alternate evaluations of (11) and
(12). (Note that (12) involves a combination of 'old' and `new' quantities.) This form of
the EM algorithm has been introduced for illustrative purposes only, and would involve
cost due to the evaluation of the covariance matrix.
We have seen that each data point t n induces a Gaussian posterior p(x n jt n ) distribution in
the latent space. For the purposes of visualization, however, it is convenient to summarize
each such distribution by its mean, given by hx n i, as illustrated in Figure 1. Note that
x
prior
posterior

Figure

1: Illustration of the projection of a data point onto the mean of the posterior
distribution in latent space.
these quantities are obtained directly from the output of the E-step (7). Thus a set of data
points projected onto a corresponding set of points fhx n ig in
the 2-dimensional latent space.
3 Mixtures of Latent Variable Models
We can perform an automatic soft clustering of the data set, and at the same time obtain
multiple visualization plots corresponding to the clusters, by modelling the data with a
mixture of latent variable models of the kind described in Section 2. The corresponding
density model takes the form
where M 0 is the number of components in the mixture, and the parameters - i are the
mixing coefficients, or prior probabilities, corresponding to the mixture components p(tji).
Each component is an independent latent variable model with parameters - i , W i and oe 2
This mixture distribution will form the second level in our hierarchical model.
The EM algorithm can be extended to allow a mixture of the form (13) to be fitted to the
data (see Appendix B for details). To derive the EM algorithm we note that, in addition
to the fx n g, the missing data now also includes labels which specify which component
is responsible for each data point. It is convenient to denote this missing data by a set
of variables z ni where z generated by model i (and zero otherwise). The
prior expectations for these variables are given by the - i and the corresponding posterior
probabilities, or responsibilities, are evaluated in the extended E-step using Bayes' theorem
in the form
Although a standard EM algorithm can be derived by treating the fx n g and the z ni
jointly as missing data, a more efficient algorithm can be obtained by considering a two-stage
form of EM. At each complete cycle of the algorithm we commence with an 'old'
set of parameter values - i , - i , W i and oe 2
i . We first use these parameters to evaluate
the posterior probabilities R ni using (14). These posterior probabilities are then used to
obtain 'new' values e
using the following re-estimation formulae
e
R ni (15)
e
The new values e
are then used in evaluation of the sufficient statistics for the posterior
distribution for x ni
Finally, these statistics are used to evaluate 'new' values f
and e oe 2
using
f
R ni hx ni x T
e
d
R ni kt
\Gamma2
R ni hx T
R ni
f
which are derived in Appendix B.
As for the single latent variable model, we can substitute the expressions for hx ni i and
ni i, given by (17) and (18) respectively, into (19) and (20). We then see that the
re-estimation formulae for f
i take the form
f
d
f
where all of the data dependence been expressed in terms of the quantities
and we have defined
R ni . The matrix S i can clearly be interpreted as a
responsibility-weighted covariance matrix. Again, for reasons of computational efficiency,
the form of EM algorithm given by (17) to (20) is to be preferred if d is large.
Hierarchical Mixture Models
We now extend the mixture representation of Section 3 to give a hierarchical mixture
model. Our formulation will be quite general and can be applied to mixtures of any
parametric density model.
So far we have considered a two-level system consisting of a single latent variable model
at the top level and a mixture of M 0 such models at the second level. We can now extend
the hierarchy to a third level by associating a group G i of latent variable models with each
model i in the second level. The corresponding probability density can be written in the
where p(tji; latent variable models, and - jji correspond to
sets of mixing coefficients, one for each i, which satisfy
1. Thus each level of
the hierarchy corresponds to a generative model, with lower levels giving more refined and
detailed representations. This model is illustrated in Figure 2.
Determination of the parameters of the models at the third level can again be viewed as
a missing data problem in which the missing information corresponds to labels specifying
which model generated each data point. When no information about the labels is provided
COPY
Second Level
Mixture Model
en t Mode
Third Level
Mixture Model

Figure

2: The structure of the hierarchical model.
the log likelihood for the model (24) would take the form
If, however, we were given a set of indicator variables z ni specifying which model i at the
second level generated each data point t n then the log likelihood would become
z ni ln
In fact we only have partial, probabilistic, information in the form of the posterior responsibilities
R ni for each model i having generated the data points t n , obtained from
the second level of the hierarchy. Taking the expectation of (26) we then obtain the log
likelihood for the third level of the hierarchy in the form
R ni
in which the R ni are constants. In the particular case in which the R ni are all 0 or 1,
corresponding to complete certainty about which model in the second level is responsible
for each data point, the log likelihood (27) reduces to the form (26).
Maximization of (27) can again be performed using the EM algorithm, as discussed in


Appendix

C. This has the same form as the EM algorithm for a simple mixture, discussed
in Section 3, except that in the E-step, the posterior probability that model (i; generated
data point t n is given by
in which
Note that R ni are constants determined from the second level of the hierarchy, and R njji
are functions of the 'old' parameter values in the EM algorithm. The expression (29)
automatically satisfies the relation
so that the responsibility of each model at the second level for a given data point n is
shared by a partition of unity between the corresponding group of offspring models at the
third level.
The corresponding EM algorithm can be derived by a straightforward extension of the
discussion given in Section 3 and Appendix B, and is outlined in Appendix C. This shows
that the M-step equations for the mixing coefficients and the means are given by
e
e
The posterior expectations for the missing variables z ni;j are then given by
Finally, the W i;j and oe 2
i;j are updated using the M-step equations
f
R ni;j hx ni;j x T
e
d
R ni;j kt
\Gamma2
R ni;j hx T
R ni;j
f
Again, we can substitute the E-step equations into the M-step equations to obtain a set
of update formulae of the form
f
e
d
f
where all of the summations over n have been expressed in terms of the quantities
in which we have defined
. The S i;j can again be interpreted as responsibility-
weighted covariance matrices.
It is straightforward to extend this hierarchical modelling technique to any desired number
of levels, for any parametric family of component distributions.
5 The Visualization Algorithm
So far we have described the theory behind hierarchical mixtures of latent variable models,
and have illustrated the overall form of the visualization hierarchy in Figure 2. We now
complete the description of our algorithm by considering the construction of the hierarchy,
and its application to data visualization.
Although the tree structure of the hierarchy can be pre-defined, a more interesting possi-
bility, with greater practical applicability, is to build the tree interactively. Our multi-level
visualization algorithm begins by fitting a single latent variable model to the data set, in
which the value of - is given by the sample mean. For low values of the data space dimensionality
d we can find W and oe 2 directly by evaluating the covariance matrix and
applying (4) and (5). However, for larger values of d it may be computationally more
efficient to apply the EM algorithm, and a scheme for initializing W and oe 2 is given in


Appendix

D. Once the EM algorithm has converged, the visualization plot is generated
by plotting each data point t n at the corresponding posterior mean hx n i in latent space.
On the basis of this plot the user then decides on a suitable number of models to fit at
the next level down, and selects points x (i) on the plot corresponding, for example, to
the centres of apparent clusters. The resulting points y (i) in data space, obtained from
(1), are then used to initialize the means - i of the respective sub-models. To initialize
the remaining parameters of the mixture model we first assign the data points to their
nearest mean vector - i and then either compute the corresponding sample covariance
matrices and apply a direct eigenvector decomposition, or use the initialization scheme of


Appendix

D followed by the EM algorithm.
Having determined the parameters of the mixture model at the second level we can then
obtain the corresponding set of visualization plots, in which the posterior means hx ni i are
again used to plot the data points. For these it is useful to plot all of the data points
on every plot, but to modify the density of 'ink' in proportion to the responsibility which
each plot has for that particular data point. Thus, if one particular component takes most
of the responsibility for a particular point, then that point will effectively be visible only
on the corresponding plot. The projection of a data point onto the latent spaces for a
mixture of two latent variable models is illustrated schematically in Figure 3.
The resulting visualization plots are then used to select further sub-models, if desired,

Figure

3: Illustration of the projection of a data point onto the latent spaces of a mixture
of two latent variable models.
with the responsibility weighting of (28) being incorporated at this stage. If it is decided
not to partition a particular model at some level, then it is easily seen from (30) that the
result of training is equivalent to copying the model down unchanged to the next level.
Equation (30) further ensures that the combination of such copied models with those
generated through further sub-modelling defines a consistent probability model, such as
that represented by the lower three models in Figure 2. The initialization of the model
parameters is by direct analogy with the second-level scheme, with the covariance matrices
now also involving the responsibilities R ni as weighting coefficients, as in (23). Again, each
data point is in principle plotted on every model at a given level, with a density of 'ink'
proportional to the corresponding posterior probability, given for example by (28) in the
case of the third level of the hierarchy.
Deeper levels of the hierarchy involve greater numbers of parameters and it is therefore
important to avoid over-fitting and to ensure that the parameter values are well-determined
by the data. If we consider principal component analysis then we see that three (non-
data points are sufficient to ensure that the covariance matrix has rank two and
hence that the first two principal components are defined, irrespective of the dimensionality
of the data set. In the case of our latent variable model, four data points are sufficient
to determine both W and oe 2 . From this we see that we do not need excessive numbers
of data points in each leaf of the tree, and that the dimensionality of the space is largely
irrelevant.
Finally, it is often also useful to be able to visualize the spatial relationship between a
group of models at one level and their parent at the previous level. This can be done
by considering the orthogonal projection of the latent plane in data space onto the corresponding
plane of the parent model, as illustrated in Figure 4. For each model in the
hierarchy (except those at the lowest level) we can plot the projections of the associated
models from the level below.
In the next section, we illustrate the operation of this algorithm when applied to a simple
toy data set, before presenting results from the study of more realistic data in Sections 7
and 8.
Figure

4: Illustration of the projection of one of the latent planes onto its parent plane.
6 Illustration using Toy Data
We first consider a toy data set consisting of 450 data points generated from a mixture of
three Gaussians in a three-dimensional space. Each Gaussian is relatively flat (has small
variance) in one dimension, and all have the same covariance but differ in their means.
Two of these pancake-like clusters are closely spaced, while the third is well separated
from the first two. The structure of this data set has been chosen order to illustrate the
interactive construction of the hierarchical model.
To visualize the data, we first generate a single top-level latent variable model, and plot
the posterior mean of each data point in the latent space. This plot is shown at the top of

Figure

5, and clearly suggests the presence of two distinct clusters within the data. The
user then selects two initial cluster centres within the plot, which initialize the second-
level. This leads to a mixture of two latent variable models, the latent spaces of which
are plotted at the second level in Figure 5. Of these two plots, that on the right shows
evidence of further structure, and so a sub-model is generated, again based on a mixture
of two latent variable models, which illustrates that there are indeed two further distinct
clusters.
At this third step of the data exploration, the hierarchical nature of the approach is evident
as the latter two models only attempt to account for the data points which have already
been modelled by their immediate ancestor. Indeed, a group of offspring models may be
combined with the siblings of the parent and still define a consistent density model. This
is illustrated in Figure 5, in which one of the second level plots has been 'copied down'
(shown by the dotted line) and combined with the other third-level models. When offspring
plots are generated from a parent, the extent of each offspring latent space (i.e. the axis
limits shown on the plot) is indicated by a projected rectangle within the parent space,
using the approach illustrated in Figure 4, and these rectangles are numbered sequentially
such that the leftmost sub-model is '1'. In order to display the relative orientations of the
latent planes, this number is plotted on the side of the rectangle which corresponds to the
top of the corresponding offspring plot. The original three clusters have been individually
coloured and it can be seen that the red, yellow and blue data points have been almost
perfectly separated in the third level.

Figure

5: A summary of the final results from the toy data set. Each data point is
plotted on every model at a given level, but with a density of ink which is
proportional to the posterior probability of that model for the given data
point.
7 Oil Flow Data
As an example of a more complex problem we consider a data set arising from a non-invasive
monitoring system used to determine the quantity of oil in a multi-phase pipeline
containing a mixture of oil, water and gas [9]. The diagnostic data is collected from
a set of three horizontal and three vertical beam-lines along which gamma rays at two
different energies are passed. By measuring the degree of attenuation of the gammas, the
fractional path length through oil and water (and hence gas) can readily be determined,
giving 12 diagnostic measurements in total. In practice the aim is to solve the inverse
problem of determining the fraction of oil in the pipe. The complexity of the problem
arises from the possibility of the multi-phase mixture adopting one of a number of different
geometrical configurations. Our goal is to visualize the structure of the data in the original
12-dimensional space. A data set consisting of 1000 points is obtained synthetically by
simulating the physical processes in the pipe, including the presence of noise dominated
by photon statistics. Locally, the data is expected to have an intrinsic dimensionality
of 2 corresponding to the two degrees of freedom given by the fraction of oil and the
fraction of water (the fraction of gas being redundant). However, the presence of different
flow configurations, as well as the geometrical interaction between phase boundaries and
the beam paths, leads to numerous distinct clusters. It would appear that a hierarchical
approach of the kind discussed here should be capable of discovering this structure. Results
from fitting the oil flow data using a 3-level hierarchical model are shown in Figure 6.
Homogeneous
Annular
Laminar

Figure

Results of fitting the oil data. Colours denote different multi-phase flow configurations
corresponding to homogeneous (red), annular (blue) and laminar
(yellow).
In the case of the toy data discussed in Section 6, the optimal choice of clusters and sub-clusters
is relatively unambiguous and a single application of the algorithm is sufficient
to reveal all of the interesting structure within the data. For more complex data sets, it
is appropriate to adopt an exploratory perspective and investigate alternative hierarchies,
through the selection of differing numbers of clusters and their respective locations. The
example shown in Figure 6 has clearly been highly successful. Note how the apparently
single cluster, number 2, in the top level plot is revealed to be two quite distinct clusters
at the second level, and how data points from the 'homogeneous' configuration have been
isolated and can be seen to lie on a two-dimensional triangular structure in the third level.
8 Satellite Image Data
As a final example, we consider the visualization of a data set obtained from remote-sensing
satellite images. Each data point represents a 3x3 pixel region of a satellite land image,
and for each pixel there are four measurements of intensity taken at different wavelengths
(approximately red and green in the visible spectrum, and two in the near infra-red). This
gives a total of 36 variables for each data point. There is also a label indicating the type
of land represented by the central pixel. This data set has previously been the subject of
a classification study within the Statlog project [10].
We applied the hierarchical visualization algorithm to 600 data points, with 100 drawn
at random of each of six classes in the 4435-point data set. The result of fitting a 3-level
hierarchy is shown in Figure 7. Note that the class labels are used only to colour the data
red soil
cotton crop
grey soil
damp grey soil
soil with vegetation stubble
very damp grey soil

Figure

7: Results of fitting the satellite image data.
points and play no role in the maximum likelihood determination of the model parameters.

Figure

7 illustrates that the data can be approximately separated into classes, and the
'very damp grey soil' continuum is clearly evident in
component 3 at the second level. One particularly interesting additional feature is that
there appear to be two distinct and separated clusters of 'cotton crop' pixels, in mixtures
1 and 2 at the second level, which are not evident in the single top-level projection. Study
of the original image [10] indeed indicates that there are two separate areas of 'cotton
crop'.
9 Discussion
We have presented a novel approach to data visualization which is both statistically principled
and which, as illustrated by real examples, can be very effective at revealing structure
within data. The hierarchical summaries of Figures 5, 6 and 7 are relatively simple to in-
terpret, yet still convey considerable structural information.
It is important to emphasize that in data visualization there is no objective measure of
quality, and so it is difficult to quantify the merit of a particular data visualization tech-
nique. This is one reason, no doubt, why there is a multitude of visualization algorithms
and associated software available. While the effectiveness of many of these techniques is
often highly data-dependent, we would expect the hierarchical visualization model to be a
very useful tool for the visualization and exploratory analysis of data in many applications.
In relation to previous work, the concept of sub-setting, or isolating, data points for further
investigation can be traced back to Maltson and Dammann [11], and was further developed
by Friedman and Tukey [12] for exploratory data analysis in conjunction with projection
pursuit. Such sub-setting operations are also possible in current dynamic visualization
software, such as 'XGobi' [13]. However, in these approaches there are two limitations.
First, the partitioning of the data is performed in a hard fashion, while the mixture of
latent variable models approach discussed in this paper permits a soft partitioning in
which data points can effectively belong to more than one cluster at any given level.
Second, the mechanism for the partitioning of the data is prone to sub-optimality as the
clusters must be fixed by the user based on a single two-dimensional projection. In the
hierarchical approach advocated in this paper, the user selects only a 'first guess' for the
cluster centres in the mixture model. The EM algorithm is then utilized to determine the
parameters which maximize the likelihood of the model, thus allowing both the centres
and the widths of the clusters to adapt to the data in the full multi-dimensional data
space. There is also some similarity between our method and earlier hierarchical methods
in script recognition [14] and motion planning [15] which incorporate the Kohonen Self-Organizing
Feature Map [16] and so offer the potential for visualization. As well as again
performing a hard clustering, a key distinction in both of these approaches is that different
levels in the hierarchies operate on different subsets of input variables and their operation
is thus quite different from the hierarchical algorithm described in this paper.
Our model is based on a hierarchical combination of linear latent variable models. A
related latent variable technique called the generative topographic mapping (GTM) [17]
uses a non-linear transformation from latent space to data space and is again optimized
using an EM algorithm. It is straightforward to incorporate GTM in place of the linear
latent variable models in the current hierarchical framework.
As described, our model applies to continuous data variables. We can easily extend the
model to handle discrete data as well as combinations of discrete and continuous vari-
ables. In case of a set of binary data variables y k 2 f0; 1g we can express the conditional
distribution of a binary variable, given x, using a binomial distribution of the form
is the
logistic sigmoid function, and w k is the k th column of W. For data having a 1-of-D coding
scheme we can represent the distribution of data variables using a multi-nomial distribution
of the form
are defined by a softmax, or normalized
exponential, transformation of the form
If we have a data set consisting of a combination of continuous, binary and categorical
variables, we can formulate the appropriate model by writing the conditional distribution
p(tjx) as a product of Gaussian, binomial and multi-nomial distributions as appropriate.
The E-step of the EM algorithm now becomes more complex since the marginalization
over the latent variables, needed to normalize the posterior distribution in latent space,
will in general be analytically intractable. One approach is to approximate the integration
using a finite sample of points drawn from the prior [17]. Similarly, the M-step is more
complex, although it can be tackled efficiently using the iterative re-weighted least squares
One important consideration with the present model is that the parameters are determined
by maximum likelihood, and this criterion need not always lead to the most interesting
visualization plots. We are currently investigating alternative models which optimize other
criteria such as the separation of clusters. Other possible refinements include algorithms
which allow a self-consistent fitting of the whole tree, so that lower levels have the opportunity
to influence the parameters at higher levels. While the user-driven nature of the
current algorithm is highly appropriate for the visualization context, the development of
an automated procedure for generating the hierarchy would clearly also be of interest.
A software implementation of the probabilistic hierarchical visualization algorithm in
Matlab is available from:
http://www.ncrg.aston.ac.uk/PhiVis

Acknowledgements

This work was supported by EPSRC grant GR/K51808: Neural Networks for Visualization
of High-Dimensional Data. We are grateful to Michael Jordan for useful discussions, and
we would like to thank the Isaac Newton Institute in Cambridge for their hostpitality.



--R

"Hierarchical mixtures of experts and the EM algo- rithm.,"
An Introduction to Latent Variable Models.
Multivariate Analysis Part 2: Classification
"Mixtures of principal component analysers,"
"Mixtures of probabilistic principal component analysers,"
"Maximum likelihood from incomplete data via the EM algorithm,"
"EM algorithms for ML factor analysis,"
Neural Networks for Pattern Recognition.
"Analysis of multiphase flows using dual-energy gamma densitometry and neural networks,"
Neural and Statistical Classification.
"A technique for determining and coding sub-classes in pattern recognition problems,"
"A projection pursuit algorithm for exploratory data analysis,"
"Interactive high-dimensional data visualiza- tion,"
"Script recognition with hierarchical feature maps,"
"Learning fine motion by using the hierarchical extended Kohonen map,"

"GTM: the generative topographic mapping,"
Chapman and Hall
--TR

--CTR
Peter Tino , Ian Nabney, Hierarchical GTM: Constructing Localized Nonlinear Projection Manifolds in a Principled Way, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.5, p.639-656, May 2002
Michalis K. Titsias , Aristidis Likas, Mixture of experts classification using a hierarchical mixture model, Neural Computation, v.14 n.9, p.2221-2244, September 2002
Tien-Lung Sun , Wen-Lin Kuo, Visual exploration of production data using small multiples design with non-uniform color mapping, Computers and Industrial Engineering, v.43 n.4, p.751-764, September 2002
Neil D. Lawrence , Andrew J. Moore, Hierarchical Gaussian process latent variable models, Proceedings of the 24th international conference on Machine learning, p.481-488, June 20-24, 2007, Corvalis, Oregon
Alexei Vinokourov , Mark Girolami, A Probabilistic Framework for the Hierarchic Organisation and Classification of Document Collections, Journal of Intelligent Information Systems, v.18 n.2-3, p.153-172, March-May 2002
Daniel Boley, Principal Direction Divisive Partitioning, Data Mining and Knowledge Discovery, v.2 n.4, p.325-344, December 1998
Hiroshi Mamitsuka, Essential Latent Knowledge for Protein-Protein Interactions: Analysis by an Unsupervised Learning Approach, IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB), v.2 n.2, p.119-130, April 2005
Ting Su , Jennifer G. Dy, Automated hierarchical mixtures of probabilistic principal component analyzers, Proceedings of the twenty-first international conference on Machine learning, p.98, July 04-08, 2004, Banff, Alberta, Canada
Ian T. Nabney , Yi Sun , Peter Tino , Ata Kaban, Semisupervised Learning of Hierarchical Latent Trait Models for Data Visualization, IEEE Transactions on Knowledge and Data Engineering, v.17 n.3, p.384-400, March 2005
Michael E. Tipping , Christopher M. Bishop, Mixtures of probabilistic principal component analyzers, Neural Computation, v.11 n.2, p.443-482, Feb. 15, 1999
Unsolved Information Visualization Problems, IEEE Computer Graphics and Applications, v.25 n.4, p.12-16, July 2005
Kui-Yu Chang , J. Ghosh, A Unified Model for Probabilistic Principal Surfaces, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.1, p.22-41, January 2001
Wang , Yue Wang , Jianping Lu , Sun-Yuan Kung , Junying Zhang , Richard Lee , Jianhua Xuan , Javed Khan , Robert Clarke, Discriminatory mining of gene expression microarray data, Journal of VLSI Signal Processing Systems, v.35 n.3, p.255-272, November
Pradeep Kumar Shetty , R. Srikanth , T. S. Ramu, Modeling and on-line recognition of PD signal buried in excessive noise, Signal Processing, v.84 n.12, p.2389-2401, December 2004
maximum likelihood;latent variables;principal component analysis;factor analysis;statistics;density estimation;hierarchical mixture model;EM algorithm;data visualization;clustering