Combining Belief Networks and Neural Networks for Scene Segmentation.
We are concerned with the problem of image segmentation, in which each pixel is assigned to one of a predefined finite number of labels. In Bayesian image analysis, this requires fusing together local predictions for the class labels with a prior model of label images. Following the work of, we consider the use of tree-structured belief networks (TSBNs) as prior models. The parameters in the TSBN are trained using a maximum-likelihood objective function with the EM algorithm and the resulting model is evaluated by calculating how efficiently it codes label images. A number of authors have used Gaussian mixture models to connect the label field to the image data. In this paper, we compare this approach to the scaled-likelihood method of where local predictions of pixel classification from neural networks are fused with the TSBN prior. Our results show a higher performance is obtained with the neural networks. We evaluate the classification results obtained and emphasize not only the maximum a posteriori segmentation, but also the uncertainty, as evidenced e.g., by the pixelwise posterior marginal entropies. We also investigate the use of conditional maximum-likelihood training for the TSBN and find that this gives rise to improved classification performance over the ML-trained TSBN.
Introduction
We are concerned with the problem of image segmentation in which each pixel will be assigned to one out of
a nite number of classes. Our work is applied to images of outdoor scenes, with class labels such as \sky",
\road", \vegetation", etc. Such scenes are typically complex, involving many dierent objects, and some of
these objects can be highly variable (e.g. trees). This means that model-based approaches are not readily
applicable.
Much work on scene segmentation has been based on approaches which rst segment the whole image
into regions, and then classify each region. To carry out this task successfully it is important to classify
each region using not only its own attributes, but also to take account of the context of the other regions.
Taking account of context can be handled in two ways; either by searching for a consistent interpretation of
the whole scene, or by taking account of the local context in which a region nds itself [16]. Examples of the
whole-scene method are [34, 29], while the local-context method is used in [50, 51]. A major problem for these
approaches is that the process of region creation can be unreliable, leading to under- or over-segmentations.
An alternative approach allows the segmentation to emerge along with the classication process. This
can be formulated in the Bayesian framework, using a prior model which represents our knowledge about
the likely patterns of labels in the image, and a likelihood function which describes the relationship of
the observations to the class labels 1 . Two main types of prior model have been investigated, called non-causal
Markov random elds (MRFs) and causal MRFs in the statistical image modelling literature. In
the graphical models community these two types of models are known as undirected and directed graphical
models respectively [21].
Early work on Bayesian image modelling concentrated on non-causal MRFs, see e.g. [3, 13, 27]. One
disadvantage of such models is that they suer from high computational complexity, for example the problem
of nding the maximum a posteriori (MAP) interpretation given an image is (in general) NP-hard.
The alternative causal MRF formulation uses a directed graph. The most commonly used form of these
models is a tree-structured belief network (TSBN) structure, as illustrated in Figure 1. For image modelling
the standard dependency structure is that of a quadtree. One attractive feature of the TSBN model is its
hierarchical multiscale nature, so that long-range correlations can readily be induced. By contrast, non-causal
MRFs typically have a \
at", non-hierarchical structure. Also, we shall see that inference in TSBNs
can be carried out in a time linear in the number of pixels, using a sweep up the tree from the leaves to the
root, and back down again. In the graphical models literature this inference procedure is known as Pearl's
message-passing scheme [35]. This algorithm is also known as the upward-downward algorithm [40, 24],
being a generalisation to trees of the standard Baum-Welch forward-backward algorithm for HMMs (see e.g.
[37]). One disadvantage of TSBNs is that the random eld is non-stationary. For example, in Figure 1, the
common parent of the fourth and fth pixels from the left in X L is the root node, whilst the third and
fourth pixels share a parent in the layer above. This can give rise to \blocky" segmentations.
TSBN models have been used by a number of authors for image analysis tasks. Bouman and Shapiro [5]
introduced such a model using discrete labels in the nodes for an image segmentation task. Perez et al [36]
have discussed MAP and MPM (maximum posterior marginal) inference on TSBNs for image processing
1 Some early work in this direction was carried out by Feldman and Yakimovsky [10].
tasks. Laferte et al [19, 20] have extended this model using a multiscale feature pyramid image decomposition
and by using the EM algorithm for parameter estimation. Cheng and Bouman [6] have investigated trainable
multiscale models, using decision trees to compactly represent conditional probability tables (CPTs) in their
model.
TSBN models can also be used for continuously-valued Gaussian processes in one and two dimensions.
These are the generalisation of Kalman lter models from chains to trees. They have been studied by a
number of groups, notably Prof. Willsky's group at MIT, who have developed the theory and derived fast
algorithms for problems such as optical
ow estimation, surface reconstruction and texture segmentation
[2, 26, 11, 25]; see also [17]. Also, Crouse et al [7] have used a multiscale TSBN to model wavelet coe-cients,
and DeBonet and Viola [9] have used an interesting tree-structured network for image synthesis using non-Gaussian
densities. However, we require a prior over class-label images, and so the work on TSBNs with
discrete-valued nodes is the most germane for our purposes.
As mentioned above, exact inference procedures for non-causal MRFs are, in general, NP-hard. However,
we note that there has been important recent work on approximate inference procedures for such graphs,
using local probability propagation schemes [46]. These local schemes (which are guaranteed to give the
correct answer on graphs without loops, see [35]) give only approximate answers for the grid-structured
graphs typically used in image analysis. Other work on message-passing schemes in \loopy" graphs includes
the decoding of error-correcting codes [12]. The work in [46] can be criticised on the basis that \
at" (i.e.
non-hierarchical) non-causal MRF models are used, although it should be possible to apply similar message
passing schemes on loopy hierarchical graphs (see section 7 for further discussion).
Our work is carried out on a database of images of outdoor scenes (see Section 4 for details) for which both
colour images and the corresponding label images are available. This paper makes a number of contributions:
We investigate the eects of the adaptation (or training) of the parameters of the tree in response to
data. Two methods are considered. In the rst the tree is trained to maximise the probability of the
labelled images; we call this maximum likelihood (ML) training. This is similar to the work of Laferte
et al in [20], but we allow a non-stationary parameterisation of the tree which re
ects regularities in
the image database. In the second method the tree is trained to maximise the probability of the correct
label image given the raw input image; we call this conditional maximum likelihood (CML) training.
The quality of the ML-trained TSBN model has been evaluated by comparing how well it codes a test
set label images. This performance is compared with a number of other coding schemes including the
JPEG-LS lossless codec. We also provide an analysis of TSBN coding which allows us to quantify the
benets of using higher levels of the tree (which correspond to longer-range correlations).
The TSBN comprises the prior aspect of the Bayesian model, and we also require a likelihood term
whereby the image data in
uences the segmentation. The direct approach is to produce a generative
model of the probability density of pixel features given the image class. For example in [5] the class
conditional densities were modelled using Gaussian mixture models. We compare this approach with
an alternative one, where neural networks are used to make local predictions given the pixel features.
These predictions can then be combined with the prior in a principled manner using the scaled-
likelihood method (see Section 2.2 for further details). The eects of combining these methods with
the ML and CML trained trees are also investigated. We evaluate the performance of the segmentation
algorithms not only using pixelwise classication rates, but also through the analysis of posterior
pixelwise entropies and through the conditional probabilities of the label images given the colour
images.
The remainder of this paper is organised as follows: In Section 2 we describe the TSBN model and the
scaled-likelihood method in more detail, and explain how the inference can be carried out. In Section 3 we
derive equations for training the TSBN using maximum likelihood (ML) and conditional maximum likelihood
(CML) methods, and image coding using TSBNs. In Section 4 we give details on the data and the training
of various models. In Section 5 results are presented concerning image coding and the estimation of the
CPTs and in Section 6 we analyse the classication results.
network and neural network models for image segmenta-
tion
2.1 Generative model
Our model for the data is illustrated in Figure 1. The observed data Y is assumed to have been generated
from an underlying process X, where X is a TSBN. The network is arranged in layers. At the highest level
(level there is one node X 0 , which has children in level 1. The lower levels are denoted
on, down to X L in level L. The fundamental property of belief networks is their encoding of conditional
independences [35, 15]. The layered structure means that the distribution of X n , for 1  n  L, given all
coarser scale nodes is only dependent on X n 1 . Indeed, the tree-structure of the network means that any
node in X n is only dependent on a single node in X n 1 . Typically in our experiments each parent node has
four children, giving rise to a quadtree-type architecture.
Each node is a multinomial variable, taking on one of C class labels. These labels are those used for the
segmentation, e.g. road, sky, vehicle etc 2 . The links between the nodes are dened by conditional probability
tables (CPTs). As it does not have any parent, the root node has an unconditional prior distribution instead
of a CPT.
The nodes in X L have a one-to-one correspondence with the observed data Y . (The observed data
in our case will be features derived from blocks of pixels, rather than individual pixels in the raw image.)
The model for the observations illustrated in Figure 1 is that the observation Y i is dependent only on the
corresponding variable X L
is
Y
2 Note that it is not necessary for the hidden nodes to be C-class multinomial variables. We have used this for convenience,
and because it gives rise to a simple initialisation of the TSBN-training, as described in Section 4.5.
Y

Figure

1: A 1-D graphical model illustrating a small tree-structured belief network. The layers in X are
denoted on, down to X L ; in this case denotes the raw image information.
2.2 Likelihood model
Above we have described a fully generative model for Y , dened by CPTs from the root downwards. Since
each pixel is composed of a number of components in feature space, P (Y
density function. One method is to use Gaussian mixture density, as used in [5] and described in section 4.4.
However, the database we are using provides both the raw data Y and labelled segmentations X L .
Given this information it is natural to train classiers (e.g. neural networks) to predict X L
is well known that neural networks trained with various error functions (including cross-entropy and mean-squared
error) approximate the posterior probabilities P (X L
Fusion of these predictions into the
belief network for X cannot then be achieved immediately, as this requires P (Y i jX L
terms. However, by using Bayes' theorem we obtain
For inference on X, the data Y is xed and so the factors P (Y i ) need not be considered further. Dening
the scaled likelihood L(X L
location i as
we obtain
Y
By replacing L(X L
principled method for the fusion of
the local predictions ^
with the global prior model P (X). (The notation ^
P denotes that
are estimates of the desired probabilities.) This method of combining neural networks
with belief networks has been suggested (for HMMs) in Smyth[42] and Morgan and Bourlard[31]. There is
an interesting connection between the scaled likelihood and the mutual information I(X between the
random variables X i and Y i , where I(X )]. The mutual information is the
expected value of the log scaled likelihood.
As described above, we would need to have and train a separate neural network to predict P (X
for each pixel. This is clearly undesirable, both in terms of the computational eort and in the amount of
data required. The solution we have adopted is to train just one neural network, but to make the position
coe-cients of the pixel be a part of the inputs to the neural network.
To use the scaled likelihood, we require not only ^
In our images it will turn
out that P (X i ) should depend on the position of pixel i, as we know that in the ensemble of images
there are regularities such as the sky appearing at the top. There are a number of ways to approach the
estimation of P (X i ). One is to train a network to predict the class labels given the position of the pixel. An
alternative would be to use the relationship P (X
and to approximate the integral
with an average over appropriate feature vectors. In our experiments (see Section 6) we compared a spatially
one derived from the pixelwise marginals of the ML-trained TSBN; the results obtained
were very similar.
A potential advantage of the scaled-likelihood method is that the generative model for P (Y i jX i ) may be
quite complex, although the predictive distribution P (X i jY i ) is actually quite simple. This means that the
generative approach may spend a lot of resources on modelling details of P (Y i jX i ) which are not particularly
relevant to the task of inferring X.
2.3 Inference
Given a new image we wish to carry out inference on X L , given the probabilistic model. Computing
the posterior P (X would be highly expensive, as it would require enumerating all possible
C K states in X L . There are two alternatives that are computationally feasible, (i) the computation of
the posterior marginals P (X L
giving rise to a segmentation based on maximum posterior
marginals (MPM) and (ii) the MAP interpretation of the data x . These can
be achieved by Pearl's - message passing schemes, as described in [35]. These schemes are non-iterative
and involve one upward and one downward pass through the tree. Details are given of the MPM computation
in

Appendix

A, along with a method for scaling the calculation to avoid under
ow, and the computation of
the marginal likelihood P (x L ).
3 Training the TSBN
Above it was assumed that the parameters used to dene P (X) are known. In fact we estimated these from
training data. Let  denote these parameters, which are the prior probabilities of the root node and the
CPTs in the tree. Let x il , denote the possible values of X i , and let pa ik ,
the set of possible values taken on by Pa i , the parent of X i . The parameter  ikl denotes the CPT entry
l  1. For simplicity the symbols X i and Pa i are dropped, and
the probability is written as P (x il jpa ik ).
For training the prior model it is assumed that a number of observation images y m and associated labelled
images x Lm are available, where is the index to the images in the training set. Let  denote
the parameters in the likelihood model P (yjx; ).
We discuss in turn maximum likelihood training (x3.1) and conditional maximum likelihood training
(x3.2).
3 Note that nding the conguration x  is not likely to be equivalent to nding the conguration that maximises P (X
x
3.1 Maximum likelihood
In maximum likelihood training the parameter vector, , is estimated so that
ML
Y
Y
We can see that the likelihood model parameters  and the TSBN model parameters  can be estimated
separately by choosing the likelihood model parameters to maximise
and the prior model
parameters to maximise
Assuming that the likelihood model is xed, we obtain ^
ML
j). The optimisation can be carried out using the EM algorithm. It uses bottom-up
and top-down message passing to infer the posterior probabilities of the hidden nodes in the E-step, and
then uses the expected counts of the transitions to re-estimate the CPTs [40, 24, 21].
The re-estimation formulas can be derived directly by maximising Baum's auxiliary function Q(
new estimated parameter vector), where x m
denotes the \hidden" variables x m nx Lm on pattern m. The update for each entry in a CPT is given by
The joint probability
can be obtained locally using the - message passing scheme (see


Appendix

A for details).
This update gives a separate update for each link in the tree. Given limited training data this is un-
desirable. If the set of variables sharing a CPT is denoted as X I , then the EM parameter update is given
by
~
If only the Y information is available, one can still carry out maximum likelihood training of the model;
both parameter sets  and  would be adapted in this case. This is known as unsupervised learning, and is
described for TSBNs in [20]. A disadvantage of the scaled-likelihood method is that it cannot be used for
unsupervised learning as P (Y ) is not available.
In [5] (Section III C), it appears that the parameters are re-estimated on the test image, which is unusual
from the standard pattern recognition methodology, where model parameters are estimated from training
data and then xed when applied to test images. We follow the standard methodology.
3.2 Conditional maximum likelihood
In the CML procedure, the objective is to predict correctly the labels x L associated with the evidence y.
The parameters are then estimated by maximising the probability of the correct labelling given the evidence
y,
CML
Y
Y
By analogy to the Boltzmann machine, we observe that computing the conditional probability requires
computation of (1) the probability P in the clamped phase (i.e. with x Lm and y m xed), and
(2) the probability p(y m j; ) in the free-running phase (with only y m xed) 4 .
Below we assume that the likelihood model P (yjx so that the objective function is viewed
as a function of  only. To carry out the optimisation in Equation 8 we take logarithms and dene
log
log P
Here we have used the subscripts c and f to mean \clamped" and \free". Using the decomposition of
can be further simplied as
log[
log
log
Then, to nd ^
CML
in Equation 8 we need to maximise
log P
4 The use of the terminology clamped and free-running here follows that in [18].
Unfortunately the EM algorithm is not applicable to the CML estimation, because the CML criterion is
expressed as a rational function[14] . However, maximisation of Equation 11 can be carried out in various
ways based on the gradient of L(). In speech analysis [18, 39], methods based on gradient ascent have been
used. The scaled conjugate gradient optimisation algorithm [30, 4] was used in our work. To use this search
method we need to calculate the gradient of L() w.r.t.
Letting
ik jx Lm ; ), it can be shown (see Appendix B
for details) that
@ ikl
ikl
where m ikl and n ikl can be obtained by propagating y m and x Lm respectively, see Equation 23.
When maximising L() it must be ensured that the probability parameters remain positive and properly
normalised. The softmax function is used to meet these constraints. We dene
l 0 e z ikl 0 where
the z ikl 's are the new unconstrained auxiliary variables and  ikl always sums to one over the l index by
construction. The gradients w.r.t. z ikl can be expressed entirely in terms of  ikl , m n
ikl and n n
ikl ,
@z ikl
ikl  ikl
l 0
3.3 Image coding
A TSBN provides a generative, probabilistic model of label images. We can evaluate the quality of a
model for the label process by evaluating the likelihood of a test set of label images under the model.
By calculating log 2 P (x L )=(#labelled pixels in the image) we obtain the coding cost in bits/pixel. The
minimum attainable coding cost is the entropy (in bits/pixel) of the generating process. As the computation
of P intractable in MRF models, we compare the TSBN results to those from the lossless JPEG-LS
codec [44, 45], available from http://www.hpl.hp.com/loco/.
3.3.1 Coding cost under a TSBN model
Using a TSBN to model the distribution of images, the marginal likelihood of a label image x L can be
calculated e-ciently at the root node X 0 of the tree (see Appendix A.2).
Below we will also consider the eect of truncating the tree at a level below the root of the tree. In this
case, instead of having one large tree, the image model consists of a number of smaller trees (and correlations
between the dierent trees are then ignored). This allows us to quantify the benets of using the higher
levels in the tree, which correspond to longer-range correlations. The priors for each of the smaller trees are
calculated by propagating the prior through the downward CPTs to obtain a prior for each root. The
likelihood of an image under the truncated model is simply the product of the likelihoods of the subimages
as computed in each of the smaller trees.
4 Experimental details
4.1 Data
Colour images of out-door scenes from the Sowerby Image Database 5 of British Aerospace are used in our
experiments. The database contains both urban and rural scenes. They feature a variety of every-day
objects from roads, cars and houses to lanes and elds in various places near Bristol, UK. All the scenes
were photographed using small-grain 35mm transparency lm, under carefully controlled conditions. Each
image of the database has been digitised with a calibrated scanner, generating a high quality 24-bit colour
representation.
Both colour images and their corresponding label images are provided in the database. The label images
were created by over-segmenting the images, and then hand labelling each region produced. There were 92
possible labels, organised in a hierarchical system. We combined labels to produce seven classes, namely
\sky", \vegetation", \road markings", \road surface", \building", \street furniture" and \mobile object".
For instance, the class \street furniture" is a combination of many types such as road sign, telegraph pole
5 This database can be made available to other researchers. Please contact Dr. Andy Wright or Dr. Gareth Rees, Advanced
Information Processing Department, Advanced Technology Centre - Sowerby, BAE SYSTEMS Ltd, PO Box 5 Filton, Bristol,
BS34 7QW, UK, email: gareth-s.rees@baesystems.com for details.
Undefined
Vegetation
Road Marking
Road Surface
Building
Furniture
Mobile Object

Figure

2: A rural and an urban scene and their hand-labelled classication. (a) Original images. (b) Hand-labelled
classication. On the right is a key describing the labels used.
and bounding object.
Figure 2a shows two scenes from the test image set of the database, one rural and one urban. Figure
2b shows their hand-labelled classications; dierent grey-levels in the label image correspond to the seven
dierent possible classes. The original 104 images were divided randomly into independent training and
test sets of size 61 and 43 respectively. The full-resolution colour images of size 512 by 768 pixels were
downsampled into 128 by 192 regions of size 4 by 4 pixels. The label of the reduced region was chosen by
majority vote within the region, with ties being resolved by an ordering on the label categories. From now
on we will refer to the reduced label images as label images because the original label images will no longer
be used.
4.2 Feature extraction
An important step in classication is that of feature selection. Initially forty features were extracted from
each region. Among them, six features were based on the R, G, B colour components, i.e., the mean and
variance of overall intensity in the region, the colour hue angle (sine and cosine) [1], (R-B) and (2G-R-B)/2
(as used by [34]), where R, G and B indicate the means of the red, green and blue components respectively.
The texture features are the grey-level dierence vectors (GLDV) textural features [48, 47] of contrast,
entropy, local homogeneity, angular second moment, mean, standard deviation, cluster shade and cluster
prominence. The GLDV features were extracted based on the absolute dierence between pairs of gray
levels at a distance apart at four angles . The (x; y) location of the region was
also included in the feature space, as described in Section 2.2. Each feature was normalised by using a linear
transformation to have zero mean and unit variance over the training set.
It is useful to limit the number of features used because increasing the number of features increases the
free parameters which need to be optimised in neural network training phase. A generalised linear model
(GLM) using the normalised features as inputs and softmax outputs [4] was used in feature selection. For
each input, the sum of the absolute values of the weights coming out of that input in the trained GLM
was calculated, and the twenty-one features for which this sum was larger than unity were retained. This
selection procedure is based on the idea that more important features will tend to give rise to larger weights
(cf the Automatic Relevance Determination idea of MacKay and Neal [32]).
4.3 MLP training
Multi-Layer Perceptrons (MLPs) were used for the task of predicting P (X L
As explained in Section 2.2,
these probabilities were estimated by a MLP that takes as input both the non-positional feature vector and
position of pixel i. The retained features produced a feature vector for each region and were fed to a MLP
with 21 input nodes, 7 output nodes and one hidden layer which was trained to classify each region into
one of the seven classes. The activation functions of the output nodes and hidden nodes were the softmax
function and tanh sigmoid functions respectively. The error function used in the training process was cross-entropy
for multiple classes (see [4]). A scaled conjugate gradient algorithm was used to minimise the error
function. The training is performed using over 51,000 regions extracted from the training image dataset,
and validating on an independent validation dataset of over 15,000 regions. The validation dataset was used
in order to choose the optimal number of hidden nodes in the MLP; eventually the best performance on the
validation set is obtained for a MLP with nodes.
The training dataset for MLP training was formed by choosing randomly up to 150 regions for each class
from each single image. By doing so we tried to use equal numbers of regions from each class in the training
set of the MLP. The aim of this rebalancing of the training set is to give the net a better chance to learn about
the infrequent classes (see [4], p 224). The probabilities for each class in the training set of the MLP, denoted
by ~
estimated simply by evaluating the fraction of the training set data points in
each class. The corresponding probabilities for all pixels in the whole of the training set images are denoted by
These turned out to be ~
and 0:0070). The ordering of the classes is \sky",
\vegetation", \road markings", \road surface", \building", \street furniture" and \mobile object". These
two sets of prior probabilities are very dierent; ~
almost uniformly distributed over all classes, but
biased towards classes two and four, corresponding to \vegetation" and \road surface" respectively.
Since the training set for the MLP reweights the classes according to ^
necessary to consider
what eect this will have on the scaled likelihood. In fact, following [4] (p. 223) we nd that
Z
~
~
where y i is the input to the network at pixel i, ~
is the network output for class k, P (c k jy i ) is the
compensated network output and Z is the normalising factor used to make
to one. Hence
we see that the scaled likelihood P is equal to ~
to an unimportant constant.
We call ~
prediction, and P (c k jy i ) (as given by Equation 14) the compensated MLP
prediction. A segmentation can be obtained by choosing the most probable class at each pixel independently.
We call these the raw MLP and compensated MLP segmentations, when using the uncompensated and
compensated predictions respectively.
4.4 Gaussian mixture model training
In section 4.3 we have described how MLPs were trained to relate the image features to labels. The alternative
approach is to build class-conditional density estimators for each class, and to use this along with Bayes'
rule to make predictions. Following [5, 20] we have used Gaussian mixture models (GMMs) for this task.
Specically the cluster program available from http://www.ece.purdue.edu/~bouman was used. The
same training set was used as for the MLP. We considered three dierent feature sets (i) the average R, G,
values in a region, (ii) the 21 features used to train the MLP and (iii) all 40 features. In addition two
dierent settings of the cluster program were used, allowing for either diagonal or full covariance matrices
for the Gaussians. The program selects the number of mixture components automatically using a MDL
criterion; the recommended initialisation of starting with three times as many components as features was
used.
The GMMs for each class were combined with the prior probabilities for each class (the P (c k )'s given in
section 4.3) to produce pixelwise classications. The overall classication accuracies were 68.72%, 49.76%,
75.95%, 71.38%, 77.45%, 71.92% for the 3-full, 3-diag, 21-full, 21-diag, 40-full, 40-diag models respectively
on the test set. The GMM model with highest pixelwise performance (namely 40-full) was then used in
further experiments (see section 6 for further details). The number of mixture components in the 40-full
model for each of the seven classes was 9; 9; 4; 9; 8; 7; 6 respectively.
We have found that the trained GMM sometimes makes very condent misclassications and that this
can cause under
ow problems when evaluating the conditional probability P TSBN (x L jy), the conditional
probability of the \ground truth" labelling x L given the image y (see section 6.4). For this reason we
replaced the likelihood term P (Y i jX L
, the minimum
value needed to avoid under
ow in the ML-TSBN.
4.5 TSBN training
The TSBN used was basically a quadtree, except that there were six children of the root node to take into
account the 2:3 aspect ratio of the images. For a down-sampled image with a total of 128 192 pixels, we
took each pixel in the down-sampled image as a leaf node in a belief network and built up an eight-level
TSBN with a total of 32756 links between nodes at adjacent levels. If each link had a separate CPT, a very
large training set would be needed to ensure the CPTs were well determined, and this in turn implies that
huge computational resources could be needed in order to nd a suitable minimum of the CML objective
function. In practice such an approach is clearly impractical. One technique for dimensionality reduction in
this case is to tie CPTs. In our experiments all of the CPTs on each level were constrained to be equal, except
for the transition from level 0 to level 1, where each table was separate. This
exibility allows knowledge
about the broad nature of scenes (e.g. sky occurs near to the top of images) to be learned by the network,
as is indeed re
ected in the learned CPTs (see section 5).
For training the ML-TSBN, the network parameters were initialised  in a number of dierent ways. It
was found that the highest marginal likelihood in the training data was obtained when the initial values of
were computed using probabilities derived from downsampled version of the images. The sparse-data problem
appeared in the initial values of the CPTs because some pairings do not occur in the training data 6 . We
dealt with the problem by adding a small quantity  to each conditional probability p(c k jc i ) for
then normalising the modied probabilities. We have used for the case that at least
one 1=C. The plot of likelihood against iteration number levelled
iterations. In the database some pixels are unlabelled; assuming these values are \missing at random", we
treated them as uninstantiated nodes, which can easily be handled in a belief network framework.
The CML training was initialised at the ML-TSBN solution. The plots of conditional likelihood against
iteration number had levelled 44 iterations of CML training by scaled conjugate gradient optimisation
for both the GMM and MLP predictors.
The upward () propagation in the tree takes around 10s, and the downward () propagation around
40s on a SGI R10000 processor; the tree has over 30,000 nodes.
We have made available the C++ code for TSBN training and inference, along with a MATLAB demonstration
which calls these functions at http://www.dai.ed.ac.uk/daidb/people/homes/ckiw/code/cbn.html.
4.6 Combining pixelwise predictions with trees
We now have GMM and MLP local predictors, and ML and CML trained TSBNs. This gives rise to a large
number of possible combinations of pixelwise predictors with trees. The ones that we have investigated are
6 The reason why it is important to consider this is that if a CPT entry is set to zero, the EM algorithm will not move it
away from zero during training.
1. raw GMM pixelwise predictions
2. compensated GMM pixelwise predictions (spatially-uniform compensation)
3. compensated GMM pixelwise predictions (using marginals of ML-TSBN)
4. GMM likelihood
5. GMM likelihood
6. raw MLP pixelwise predictions
7. compensated MLP pixelwise predictions (spatially-uniform compensation)
8. compensated MLP pixelwise predictions (using marginals of ML-TSBN)
9.
The TSBN methods calculated the scaled likelihoods as described in Section 4.3, and MAP inference was
used for the pixelwise predictions. For entries 3 and 8 (compensation using the marginals of the ML-TSBN)
note that dierent compensation probabilities are used in the six regions of the image dened by the six
CPTs from the root to level 1. The performance of these methods is investigated in sections 5 and 6.
5 Results from TSBN training
In this section we describe the results from training the TSBN using both ML and CML training. We rst
discuss label-image coding results using the ML-trained tree, and then inspect the learned CPTs from the
ML-trained TSBN and the CML-trained TSBN.
5.1 Image coding results
In this section we present results comparing the ML-trained TSBN and lossless JPEG coding. The relevant
theory has been described in Section 3.3 and the details of the TSBN training are given in Section 4.5.
0.611.4truncation level
bits/pixel

Figure

3: Bit rate (bits/pixel) as a function of the truncation level in the TSBN.
The average bit rate for the TSBN model was 0:2307 bits/pixel (bpp). For comparison purposes the
JPEG-LS codec gave an average bit rate of 0:2420 bpp. We also tried compressing the label images using
coding using the Unix utility gzip; this gave 0:3779 bpp. The fact that a similar level of
compression performance is obtained by JPEG-LS and the TSBN suggests that the TSBN is a reasonably
good model of the label images.
Using the \truncated tree" scheme discussed in Section 3.3, we can analyse the TSBN results further.

Figure

3 shows the bit rate (in bits/pixel) evaluated as a function of truncating the tree at levels 0 to 7. By
the time level 4 has been reached (corresponding to a 8  8 block size), almost all of the benet has been
attained.
5.2 The learned CPTs
The CPTs derived using ML training are shown in Figure 4. Note that six separate CPTs were used for the
transition from the root node to level 1, as explained in Section 4.5.
We can also calculate the prior marginals at each node in the tree, by simply taking the prior from the root
node and passing it through the relevant CPTs on the path from the root to the node under consideration 7 .
The fact that there are six CPTs for the root to level 1 transition means that there are, in eect, six dierent
prior marginals in levels 1 to 7, dened by the 2:3 aspect ratio of the image. These prior marginals are shown
in

Figure

5. It may not be easy to interpret the CPTs/marginals as a permutation of the state labels at
7 This can also be achieved using Pearl's propagation scheme outlined in Appendix A, with every leaf node uninstantiated.
a node and corresponding permutations of the incoming and outgoing CPTs would leave the overall model
unchanged; however, it appears that the downsampling initialisation means that this is not a large problem.
Analysing Figures 4 and 5, we see that (1) The prior marginals at level 7 re
ect the overall statistics of
the images. The sky, vegetation and road surface classes are the most frequently occurring, the sky class is
more likely to be found in the top half of the images and road surface in the bottom half. Similar patterns
are detectable at level 1 of Figure 5, although the vegetation label is less prevalent in the upper half at this
level. (2) The trained CPTs in levels 1 to 7 exhibit a strong diagonal structure, implying that the children
are most likely to inherit their parent's class. (3) The level 0 to level 1 CPTs need to be read in conjunction
with the root's prior distribution to provide a good explanation of the level 1 prior marginals.
Although Laferte et al [20] have carried out EM training of a TSBN, we note that they only estimated
CPTs that were tied on a layer-by-layer basis. For our data Figures 4 and 5 show that relaxing this constraint
can be useful.
The CPTs and prior marginals obtained with CML training were similar to those shown in Figures 4 and
5 respectively. This is probably due to the fact that CML training was initialised at the ML-TSBN solution
for both GMM and MLP predictors.
6 Segmentation results and performance evaluation
We now turn to the classication of the testing images. Often classication performance is evaluated on
pixelwise accuracies. However, for complex real-world classication task, such as ours, this does not tell
the whole story. There are a number of other factors that concern us, most notably the fact that we are
predicting the labels of pixels in an image, and that spatial coherence is important. We also note that the
fractions of pixels from dierent classes are tremendously dierent, and that the ground-truth labels used
in assessing performance are not 100 percent correct because of both the downsampling process, and also
because of inaccuracies in the hand-labelling process. Therefore, it is a di-cult task to assess the quality
of classication derived from the various methods, which may also depend on the uses the classication will
be put to. An early reference to assessing the quality of segmentations is [22]. More recently, there has
(a) Prior for root node:
(b) From root node to Level 1:
(c) From Level 1 to Level 2:
(d) From Level 2 to Level 3:
From Level 3 to Level 4:
(f) From Level 4 to Level 5:
(g) From Level 5 to Level
From Level 6 to Level 7:

Figure

4: Estimated prior for the root and CPTs (ML training) of an eight-level belief network after being trained on training
images. The area of each black square is proportional to the value of the relevant probability. (a) The prior probabilities at the
root node. (b) The six independent CPTs on the links from root node to its six children on the rst level. (c)-(h) CPTs for the
links between adjacent levels from Level 1 to Level 7 respectively. The seven labels are 1-Sky, 2-Vegetation, 3-Road marking,
4-Road surface, 5-Building, 6-Street furniture, 7-Mobile object. The CPTs have entry (1,1) in the top left-hand corner, and are
read with \from level l" indexing the rows and \to level l indexing the columns.
Level 0:
Level 1:
Level 2:
Level 3:
Level 4:
Level 5:
Level
Level 7:
Key:
Vegetation Road
Marking
Road
Surface
Building Street
Furniture
Mobile
Object
Vegetation Road
Marking
Road
Surface
Building Street
Furniture
Mobile
Object
Vegetation Road
Marking
Road
Surface
Building Street
Furniture
Mobile
Object
Vegetation Road
Marking
Road
Surface
Building Street
Furniture
Mobile
Object

Figure

5: The prior marginals after training with the ML algorithm. The area of each black square is
proportional to the value of the relevant probability. See text for further details.
been some realisation that the aim of segmentation may not be to return just a single segmentation, but
multiple solutions [33], or a probability distribution over segmentations P (x L jy). This posterior distribution
can be explored in many ways; below we describe two, namely (i) posterior marginal entropies and (ii) the
evaluation of the conditional probability P
jy), where x L
is the \ground truth" image for given input
data y.
In this section we compare the performance of classication based on the smoothness of the segmented
image in Section 6.1, the pixelwise prediction accuracies in Section 6.2, the marginal entropies in Section 6.3
and the conditional probability in Section 6.4.
6.1 Smoothness
For the rural scene in Figure 2, Figure 6 shows classications using most of the combinations outlined in
section 4.6. The classications obtained from the single-pixel methods typically have a lot of high-frequency
noise, due to locally ambiguous regions. Both the ML- and CML-trained trees tend to smooth out this noise.
A similar smoothing can be obtained using a majority lter [23], where one simply chooses the most
common class within a window centered on the pixel of interest. However, one drawback with this is that
majority-lter smoothing with a reasonably-sized window tends to remove ne detail, such as road markings;
in contrast it seems that the TSBN methods yield something like an adaptive smoothing, depending on the
strength of the local evidence. Also, note that majority-ltering does not return a probability distribution
over segmentations.
6.2 Pixelwise classication accuracy

Table

1 shows the pixelwise classication accuracy for each class, and the overall accuracy for each of the ten
methods listed in Section 4.6. For the TSBN methods the MAP segmentation result is reported; the MPM
results were similar, although they were generally worse by a few tenths of one percent. The most noticeable
feature is that the performance obtained with the MLP methods is superior to that from the GMM methods.
Looking at the results in detail, we notice that the raw results for both the GMM and MLP (columns 1
and are improved by compensation (columns 2 and 7 resp). The compensated methods simply give more

Figure

Classication of a rural scene. (i) raw GMM pixelwise predictions, (ii) raw MLP pixelwise predictions,
(iii) compensated GMM pixelwise predictions, (iv) compensated MLP pixelwise predictions, (v) MAP segmentation
segmentation for MLP segmentation for GMM
TSBN, (viii) MAP segmentation for MLP
weight to the more frequently occurring classes, as can be seen by comparing columns 1 and 2, and 6 and
7. There are only small dierences between the spatially uniform compensation of columns 2 and 7, and the
ML-TSBN marginal compensation scheme of columns 3 and 8.
Columns 4 and 8 combine pixelwise evidence with the ML-TSBN. For both GMM and MLP local models,
it is perhaps surprising that the performance decreases as compared to columns 3 and 7 respectively. (This
is a fair comparison as the methods of columns 3 and 7 use the marginals of the ML-TSBN, but not the
correlation structure).
Columns show the performance of the GMM and MLP local models combined with trees trained
using the CML method on the relevant data. In both cases the performance is better than fusion with the
ML-TSBN and for the MLP this method obtains the best overall performance.
For comparison, we note that McCauley and Engel [28] compared the performance of Bouman and
Shapiro's SMAP algorithm against a pixelwise Gaussian classier on a remote sensing task, and found that
the overall classication accuracy of SMAP was a 3.6% higher (93.4% vs 89.8%).
The reasons for the superior performance of the MLP in our experiments are not entirely clear. However,
we note that the test images are a relatively diverse set of images (although drawn from the same distribution
as the training images). It may be that features that are important to the MLP classier are similar in the
training and test images, while other features (whose distribution has to be modelled by the GMMs) do vary
between training and test sets. In contrast, some other evaluations in the literature (e.g. [28]) use only a
single test image with training data drawn from a subset of the pixels; in this case the issue of inter-image
variability does not arise. We also note that the comparison of the GMM and MLP classiers was carried
out using a training set of a particular size and composition; dierent results might be obtained if these
factors were varied.
6.3 Pixel-wise entropy
We are interested in understanding the uncertainty described by P (x L jy). It appears that the computation
of the joint entropy of this conditional distribution is intractable, however posterior marginal entropies are
readily computable from the posterior marginals P (x L

Table

1: Performance of the 10 methods, showing the percentage correct for each class and overall. The
second column in the table gives the overall percentage of each class in the test images.
Class
percentage
vegetation 40.12 61.04 78.92 77.46 67.92 75.72 79.32 92.41 90.40 81.67 90.45
road markings 0.17 55.14 42.26 44.33 43.09 27.36 78.61 68.97 67.91 70.04 67.91
road surface 39.5 62.14 78.18 80.45 70.10 73.45 94.52 97.16 96.26 94.99 96.85
building 6.11 44.19 46.98 49.67 63.70 74.92 67.69 44.43 52.73 79.40 64.60
street furniture 1.35 28.58 14.05 13.77 20.97 8.58 24.89 3.96 4.62 10.07 6.63
mobile object 0.57 58.85 43.05 43.10 72.76 76.23 49.13 28.83 32.15 78.89 44.74
overall 63.71 77.45 77.94 71.00 75.85 85.72 90.16 89.38 87.38 90.68
kjy) 8 . Images displaying these posterior marginal entropies are shown in Figure 7, pertaining to the original
image shown on the right in Figure 2. As expected, the pixelwise entropy is reduced by the use of the TSBN;
this is particularly eective for the CML trees. Notice that the pixels which have signicant posterior
marginal entropy are good indicators of the pixels that are misclassied; this is especially true of the CML-
combination in Figure 7. This property could well be useful information for a later stage of
processing.
6.4 Conditional probability
If the model we have developed is a good one, then P (x L jy) should ascribe high probability to the \ground
truth" labelling x L
. Dierent image models can be compared in terms of the relative values of P
jy).
In particular we compare the TSBN image models against independent pixel models.
Ignoring spatial correlations we obtain PMLP
for the MLP local prediction, and
similarly for GMM local prediction. For a TSBN P (x L
jy) can be calculated as follows:
8 During the revision of this paper we became aware that the calculation of posterior marginal entropies had been proposed
independently by Perez et al [36] to determine \condence maps".
(a)
(c) 0.0487 (d)

Figure

7: Posterior marginal entropies for the MLP predictor. (a) compensated pixel-wise predictions, (b)
ML-TSBN, (c) CML-TSBN. The greyscale is such that black denotes zero entropy, white denotes 2.45 bits.
The number underneath each plot is the average pixelwise posterior entropy. (d) Binary image showing
misclassied labels (bright) and correctly-classied labels (dark).
Equation 16 follows from Equation 15 as P (y i jx L
) and the denominator can be
evaluated by the methods outlined in Appendix A.2.
A complexity arises in this calculation when there are pixels which do not have a label. For PMLP (x  jy),
these pixels were simply ignored. For P TSBN (x L
jy), the unlabelled pixels were ignored in both the numerator
and denominator of Equation 17. This is achieved by setting the -vector at the appropriate nodes to
be the vector of ones (see Appendix A for details).
In

Figure

8 we plot 1
log P (x L
jy) under various models. Panel (b) shows that in all 43 test images,
the posterior probability under the MLP+CML-TSBN method is larger than that under the compensated
MLP using an independent-pixel model. Panel (a) shows that for a similar comparison using the GMM
predictor, the CML-TSBN method is better in 41 out of 43 cases. Notice also the relative scales of the plots,
and especially that the GMM model makes condent mistakes on some pixels, thereby dragging down the
average posterior probability.
log(P) for GMM
log(P)
for
GMM
CML-TSBN
-0.5log(P) for MLP
log(P)
for
MLP
CML-TSBN
(a) (b)

Figure

8: Comparison of log P (x L
jy)=N for (a) compensated GMM vs GMM compensated MLP
There are a large number of similar plots that can be made. A comparison of the posterior probabilities
under the ML-trained TSBN and independent models comes up with roughly equal numbers being better
coded under the two models, for both GMM and MLP predictors. The CML-trained TSBN with MLP
prediction is better than all other methods on 39 out of the 43 test images; on the remaining four the
ML-TSBN with MLP wins.
This paper has made a number of contributions:
We have used the EM algorithm to train the ML-TSBN and have observed that the learned parameters
do re
ect the underlying statistics of the training images. The quality of the probabilistic model
has been evaluated in coding terms and found to be comparable to state-of-the-art methods. The
\truncated tree" analysis shows over what scales correlations are important.
We have compared the performance of GMM and MLP pixelwise classiers on a sizable real-world
image segmentation task. The performance of the discriminatively-trained MLP was found to be
superior to the class-conditional GMM model. We have also shown that the scaled-likelihood method
can be used to fuse the pixelwise MLP predictions with a TSBN prior.
We have compared conditional maximum likelihood (CML) training for the tree against maximum
likelihood (ML) training on a number of dimensions including classication accuracy, pixel-wise entropy
and the conditional probability measure P (x L jy).
The problem of evaluating segmentations is an old one, and a full answer may well depend on a decision-theoretic
analysis which takes into account the end-use of the segmentation (e.g. for an automated driving
system). However, one attractive feature of the TSBN framework is that some aspects of the posterior
uncertainty can be computed e-ciently, e.g. the posterior marginal entropies discussed in Section 6.3.
architectures are not the ultimate image model, as we know that run generatively
they give rise to \blocky" label images. There are a number of interesting research directions which try to
overcome this problem. Bouman and Shapiro [5] suggest making a more complex, cross-linked model. The
problem with this is that inference now becomes much more complex (one needs to use the junction tree
algorithm, see e.g. [21]). One interesting idea (suggested in [5]) is to retain Pearl-style message passing even
though this is now not exact; this idea is analysed in [12] and [46]. Another approach to inference is to use
alternative approximation schemes, such as the \recognition network" used in Helmholtz machines [8], or
\mean-eld" theory [41].
An alternative to creating a cross-linked architecture is to retain a TSBN, but to move away from a rigid
quadtree architecture and allow the tree-structure to adapt to the presented image. This can be formulated
in a Bayesian fashion by setting up a prior probability distribution over tree-structures. Initial results of this
approach are reported in [49, 43]. We believe that the general area of creating generative models of (image)
data and nding eective inference schemes for them will be a fruitful area for research.


Appendix

A: Pearl's probability propagation procedures
Below we describe Pearl's scheme for probability propagation in trees, the computation of the marginal
likelihood P (x L j) and a scaling procedure for this algorithm to avoid under
ow.
A.1 Pearl's scheme
We rst consider the calculation of the probability distribution P (xje) at node X in a TSBN, given some
instantiated nodes (\evidence") e.
Consider the tree fragment depicted in Figure 9 (based on Figure 4.14 in [35]). P (xje) depends on two
distinct sets of evidence; evidence from the sub-tree rooted at X , denoted as e x , and evidence from the rest
of the tree, denoted as e
x . We shall assume that each node has a nite number of states C (each node can
have a dierent number of states but this adds some extra notation and is not necessary in our application).
Bayes' rule, together with the independence property in the TSBNs, yields the product rule,
x ) and
Partitioning e x as e
is dened recursively by
Y
where we have assumed that node X has n children and y ik is the kth value of node Y i .  y i (x) is known as
the -message sent to node X from its child node Y i .
(x) is given recursively by
z
Y
z 0 2s(X)
Y
z 0 2s(X)
where z k is the kth state of node Z, and s(X) denotes the siblings of X (i.e. the children of Z excluding
X itself) and  is the normalising factor so that the values of  x (z) sum to 1. In fact
z
where e s(X) denotes the evidence below the siblings of X .  x (z) is known as the -message sent to node X
e
e
x
x
Z
Figure

9: Fragment of causal network showing incoming message (named -message, shown as solid arrows) and outgoing
message (-message, broken arrows) at node X.
from its parent Z.
The propagation procedure is completed by dening the boundary conditions at the root and leaves of
the tree. The -vector at the root of the tree is equal to the prior probabilities for each of the classes. At
the leaves of the tree, the -vector is the vector of ones if the node is uninstantiated, and equal to the vector
with a single entry of 1 (and all other entries 0) corresponding to the instantiated state. The computation
of P (xje) at each node in the tree can now be performed using an upward phase of -message passing, and
a downward phase of -message passing.
To nd the maximum a posteriori conguration of the hidden variables X given evidence e, we can use
a similar message passing scheme, as described in Section 5.3 of [35].
The posterior marginal required for the EM updates and CML derivatives is given by
Y
where is the set of nodes that are siblings of node X i and  y (:) denotes the -message sent to node
Pa i by node y. To show this, we partition e pa i
as e s(X
and rst calculate P
Using the conditional independences described by the tree we obtain
Y
y (pa ik
computed by dividing through both sides of the equation by P (e
which can be calculated as
A.2: Marginal likelihood
Now we consider the procedure for computing P (ej). Assuming X 0 is the root node, we have,
where we have used
is the root node and e
x0 is empty.
A.3 Scaling Pearl's probability propagation
In order to understand where and why scaling is required for implementing the message propagation, we
consider the two distinct message passing schemes separately.
Firstly, consider the denition of (x) in Equation 21. (x) is the probability of given the evidence
x , this gives
long as the normalising factor  is applied in each time of the calculation of
-message. In this case, no scaling is needed for (x).
We now consider the denition of the (x) in Equation 19 and  y i (x) of Equation 20. -values of node
X are the product of all -messages sent to it by its children. Each child node forms a weighted sum of
its -values to form the -message sent to its parent. The element-wise multiplication of the -messages in
Equation 19 and the weighted sum calculation in Equation 20 cause the numerical values of the -vectors to
decrease exponentially with distance from the leaves of the tree.
We scale (x) with three goals: (1) keeping the scaled (x) within the dynamic range of the computer for
all nodes in the tree, (2) maintaining the local propagation mechanisms of Pearl's probability propagation,
(3) recovering the true values at the end of the computation.
This is achieved by the recursive formulae
Y
are the children of X . These equations are initialised with ^
at the leaves. d x
can be any value that gives a reasonable scaling; d
x
was used in our work. The unscaled value
(x) is computed using
x (x), where D x is the product of the scaling coe-cients in the subtree
rooted at X (and including d x ).
In fact if we are only interested in the calculation of P (xje), then it is not necessary to worry unduly
about scaling factors; the -vector can simply be rescaled at each node as required, and P (xje) can be
calculated from the scaled -vector and the -vector by requiring that P (xje) sums to 1. However, scaling is
important if we wish to calculate the marginal likelihood P (ej) 9 . Referring back to Equation 26, we nd
that
where DX0 is the product of all of the scaling factors used in the propagation procedure. Since DX0 could
be out of the machine dynamic range we compute log P
log DX0 .


Appendix

B: Calculation of derivatives for CML optimisation
In this appendix we calculate the gradient of w.r.t. As in section 3.2 we suppress
the dependence of P (yjx) on  for notational convenience.
First note that P (y m j) can be written as
the sum is over all
possible values of x in the TSBN. Using the conditional independence relations, P (xj) is easily decomposed
into a product of the transition probabilities on all links.
Following the ideas in Krogh[18] for HMMs, the derivative of L f () w.r.t  ikl is
@ ikl
@ ikl
@ ikl
9 Scaling issues are discussed in Perez et al [36]. It appears that they addressed the issue of scaling for the computation of
posterior marginals in their paper but not explicitly scaling for the computation of P (ej).
ikl
ikl
ikl
ikl
ikl
The step from equation 30 to equation 31 is derived from the fact that  ikl will only appear in the product
i is in state l and Pa m
i is in state k.
The derivative of the other term, L c (), can be calculated in a similar manner, except that the summation
over variables in the tree is now only taken over the hidden variables x

Acknowledgements

This work is funded by EPSRC grant GR/L03088, Combining Spatially Distributed Predictions From Neural
Networks and EPSRC grant GR/L78181 Probabilistic Models for Sequences. The authors gratefully acknowledge
the assistance of British Aerospace in the project and in making the Sowerby Image Database available
to us. We also thank Dr. Andy Wright of BAe for helpful discussions, Dr. Ian Nabney for help with NETLAB
routines for neural networks, Dr. Gareth Rees (BAe) for discussions on segmentation metrics, Dr. John Elgy
for introducing us to the work of [5] and Prof. Kevin Bowyer for pointing out the work of [33]. We also
thank the three anonymous referees and the Associate Editor Prof. Charles Bouman for helpful comments
and advice which have considerably improved the manuscript.



--R

Computer Vision.
Modelling and estimation of multiresolution stochastic processes.
On the statistical analysis of dirty pirtures.
Neural Networks for Pattern Recognition.
A Multiscale Random Field Model for Bayesian Image Segmentation.
Trainable Context Model for Multiscale Segmentation.

The Helmholtz Machine.

Decision Theory and Arti

A Revolution: Belief Propagation in Graphs with Cycles.
Stochastic Relaxation
An inequality for rational functions with applications to some statistical estimation problems.
An Introduction to Bayesian Networks.
Statistical pattern recognition in image analysis.
Multiresolution Gauss-Markov random eld models for texture segmentation
Hidden markov models for labeled sequences.


Graphical Models.
Dynamic measurement of computer generated image segmentations.
Remote Sensing and Image Interpretation.
Bayesian Belief Networks as a tool for stochastic parsing.

Likelihood Calculation for a Class of Multiscale Stochastic Models
Statistical methods for automatic interpretation of digitally scanned
Comparison of Scene Segmentations: SMAP


Neural Networks for Statistical Recognition of Continuous Speech.
Bayesian Learning for Neural Networks.
Textured image segmentation: returning multiple solutions.

Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference.

A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition.
Neural network classi
Hidden neural networks: a framework for HMM/NN hybrids.
Parameter Estimation of Dependence Tree Models Using the EM ALgorithm.

Hidden Markov models for fault detection in dynamic systems.
Dynamic positional trees for structural image analysis.

The LOCO-I Lossless Image Compression Algorithm: Principles and Standardization into JPEG-LS
Correctness of belief propagation in Gaussian graphical models of arbitrary topology.

A comparative study of texture measures for terrain classi
Dynamic Trees.
Image labelling with a neural network.
The use of neural networks for region labelling and scene un derstanding.
--TR

--CTR
Neil D. Lawrence , Andrew J. Moore, Hierarchical Gaussian process latent variable models, Proceedings of the 24th international conference on Machine learning, p.481-488, June 20-24, 2007, Corvalis, Oregon
Todorovic , Michael C. Nechyba, Dynamic Trees for Unsupervised Segmentation and Matching of Image Regions, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.11, p.1762-1777, November 2005
Amos J. Storkey , Christopher K. I. Williams, Image Modeling with Position-Encoding Dynamic Trees, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.7, p.859-871, July
Sanjiv Kumar , Martial Hebert, Discriminative Random Fields, International Journal of Computer Vision, v.68 n.2, p.179-201, June 2006
Todorovic , Michael C. Nechyba, Interpretation of complex scenes using dynamic tree-structure Bayesian networks, Computer Vision and Image Understanding, v.106 n.1, p.71-84, April, 2007
Richard J. Howarth, Spatial Models for Wide-Area Visual Surveillance: Computational Approaches and Spatial Building-Blocks, Artificial Intelligence Review, v.23 n.2, p.97-155, April     2005
Simone Marinai , Marco Gori , Giovanni Soda, Artificial Neural Networks for Document Analysis and Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.1, p.23-35, January 2005
neural network;scaled-likelihood method;conditional maximum-likelihood training;gaussian mixture model;expectation-maximization EM;Markov random field MRF;tree-structured belief network TSBN;hierarchical modeling