Probabilistic graphs using coupled random variables
Kenric P. Nelson*a, Madalina Barbub, Brian J. Scannellc
a
Raytheon Corporation, 5 Apple Hill Rd., Tewksbury, MA, USA 01876-0345; bRaytheon
Corporation, 125 President’s Way, Woburn, MA 01801; Nanigans, 60 State St, 12th Floor
Boston, MA 02109
ABSTRACT
Neural network design has utilized flexible nonlinear processes which can mimic biological systems, but has suffered
from a lack of traceability in the resulting network. Graphical probabilistic models ground network design in
probabilistic reasoning, but the restrictions reduce the expressive capability of each node making network designs
complex. The ability to model coupled random variables using the calculus of nonextensive statistical mechanics
provides a neural node design incorporating nonlinear coupling between input states while maintaining the rigor of
probabilistic reasoning. A generalization of Bayes rule using the coupled product enables a single node to model
correlation between hundreds of random variables. A coupled Markov random field is designed for the inferencing and
classification of UCI’s MLR ‘Multiple Features Data Set’ such that thousands of linear correlation parameters can be
replaced with a single coupling parameter with just a (3%, 4%) reduction in (classification, inference) performance.
Keywords: Probabilistic graphs, Markov Random Fields, Bayesian Networks, Nonlinear Statistical Coupling,
Nonextensive Statistical Mechanics

1. INTRODUCTION
What if the nonlinear functions of neural networks, which provide much of their computational power, could be
expressed within the constraints of a probabilistic graph? In this paper, we seek to show that the deformed algebra of
nonextensive statistical mechanics offers an approach to this objective.
Artificial neural network (ANN) design has sought to increase the ability of computers to learn fundamental tasks in
pattern recognition and control. While inspired by the functioning of biological neural networks, accurate models and
comparable performance are still being sought. Beginning with the perceptron 1, which models a weighted sum of inputs
and a bias followed by a nonlinear switch, the three fundamental components of an artificial neuron are conditioning of
the input, fusion of a large collection of inputs, and a nonlinear function at the output. The single-layer perceptron is
capable of linear discrimination. Multiple layers and backpropagation enabled the modeling of fundamental functions
such as the XOR2. The introduction of nonlinear kernels, such as those used in support vector machines, greatly
expanded the computational power and pattern recognition capabilities of ANNs.
Despite the computational power of neural network, trust in the application to real-world systems has continued to be
limited by uncertainty about tracing the learned functions of complex nonlinear network to computational building
blocks. The application of probabilistic graphs to the design of neural networks has sought to constrain the network
design such that each node models a specific random variable, thereby adding control and traceability to the design.
Furthermore, by restricting the nodes to well-defined random variables, modules can be systematically combined into
higher-levels of reasoning. Deep belief networks 3, such as those introduced by G. Hinton, segment learning into layers
which produce generative models for training of higher layers.
The challenge with probabilistic methods is that constraining each node to fusion using Bayes’ rule, requires the design
to either make assumptions about independence between random variables or significantly increase the complexity of
network connectivity. The deformed algebra4 of nonextensive statistical mechanics (NSM)5, developed by
*kenric_p_nelson@raytheon.com; phone 1 603 508 9827;

Borges and others, extends probabilistic reasoning to a family of distributions associated with nonlinear processes. In
particular a generalized product function can be used as a model of long-range dependence between random variables 6.
Factoring random variables, which for a Bayesian network requires identification of conditionally independence, can
now be done by tuning a degree of nonlinear coupling between the variables. Quite general kernel functions at the input
and sigmoid functions at the output can also be defined within this pseudo algebra; however, the focus of this paper will
be on demonstrating the utility of fusion with these nonlinear functions.
In Section 2 the concept of a coupled random variable is introduced using an interpretation of NSM called nonlinear
statistical coupling. This interpretation defines the generalized algebra in terms of the degree of nonlinear deformation.
In Section 3 the Risk Profile is introduced, which uses the methods of nonlinear statistical coupling to measure the
accuracy of inference as a function of risk. In Section 4 the design of a coupled Markov random field is demonstrated
where the dependence structure is now global rather pair-wise. In Section 5 the results of a computational experiment
using recognition of handwritten numerals are discussed.

2. DEFINITION OF A COUPLED RANDOM VARIABLE
The definition of a coupled random variable originates from the generalized product of nonlinear statistical coupling7,
which is an interpretation of nonextensive statistical mechanics. With this interpretation the statistical states of a
statistical distribution are modeled as being coupled by nonlinear dynamics, rather than being mutually exclusive.
Rather than measuring the probability of a state pi , the coupled probability of a coupled state is measured by
N

Pi

 

pi


N



j 1, j i
N

p 
i 1

i

pj

j 1, j  i

pi1



N

p



pj

i 1

.

(1)

1
i

The coupling  is a real-valued parameter and the non-coupled probability is recovered for   0 . The middle
expression makes explicit that the probability being measured is a coupling of the ith state with all the other states. The
x  1
entropy measure is deformed using a coupled logarithm ln x 
and averaged using the coupled probability



N

S   Pi   ln pi . ,


(2)

i 1

Equation (2) is the normalized Tsallis entropy8 expressed in terms of the degree of nonlinear coupling. Constraints on
the coupled-mean and coupled-variance
N

N

i 1

i 1

   xi pi1 ;  2   xi2 pi1

(3)

lead to a maximum entropy distribution, referred to as a coupled Gaussian

   xi   2 
pi  xi   A exp 

  2     2 


exp  x   ex  1   x 

.

1



The coupled exponential is the inverse of the coupled logarithm.
Combinations of coupled exponentials and coupled logarithms lead to generalization of the product and sum functions

(4)





ex  ey  ex  y   ex    ey   1




1



(5)

ln x  ln y  ln xy  ln x  ln y    ln x  ln y .

While the coupled product and coupled sum both satisfy associative and commutative properties, a distributive property
cannot be defined, so these functions are part of a pseudo-algebra, which facilitates reasoning about the coupled states.
The coupled sum makes explicit the non-additive property of the coupled entropy. The coupled division and coupled
subtraction can also be defined. This discussion is limited to reviewing the properties of the coupled product, but full
utilization of a probabilistic graph would include inferences that require division for independent variables and coupleddivision for coupled variables. The coupled product defines a method for combining distributions which will be utilized
to define a fusion of information referred to as a coupled random variable. For the purposes of this investigation
defining a coupled random variable as one in which the distributions of the random variables are combined by the
coupled product is sufficient to show an efficient model of correlation. Stronger definitions, using the definition of
generalized independence9 by Umarov, et. al. which uses the generalized product in a generalized Fourier domain and
proved to form a generalized central limit theorem is desirable.

3. PROBABILITY ACCURACY AS A FUNCTION OF RISK
The methods of nonlinear statistical coupling provide a metric for measuring the accuracy of probability inferences,
which is called a Risk Profile10. The Risk Profile measures the generalized mean of the true class probabilities. The
adjustment in the mean is associated with risk due to the connection with the negative coupled logarithm, which modifies
the cost of information. The generalized mean can be derived from the coupled entropy (1.2) by applying the coupled
exponential, which transforms the entropy measure back into probability space
 
N
Pi

Pavg  exp   S   exp  log   pi

i 1


N



   Pi


i 1

  



pi  Pi

 



1

1/ 


 
P
 N
i

p

i


 i 1 

1/ 

 N 

   Pi   pi 
 i 1


.

(6)

The derivation uses the properties that sums of coupled logarithms are equal to the coupled logarithm of the coupled
product of the arguments; that multiplying a coupled logarithm is equal to raising the argument to the coupled power,

x    ax   a  1 
a

1/ 

 P   1  1  N .
N

; and that

i 1

i

For the Risk Profile, each test sample has equal weight


Pi    1 / N and reports a probability for the true class pi ,true resulting in the generalized mean of the reported
probabilities for the true class
1/ 

 N

Paccuracy   N1  pi,true 
 i 1


(7)
1/ N

 N

For   0 the expression reduces to the geometric mean Pacc , 0    pi ,true  . Importantly, optimizing the accuracy
 i 1

of an inference, according to Shannon information theory, requires optimizing the geometric mean of the reported
probabilities. Positive risk bias or nonlinear coupling, provides metrics which lower the cost of information and are
similar to the finite costs on making a decision. Negative risk bias increases the cost of information, which is useful in
evaluating the robustness of an inference algorithm. Only   0 is a proper score11; however, the biasing with risk
provides useful insights and if necessary the full profile can be made proper, following the methods of the Tsallis proper
score12.

4. DESIGN OF A COUPLED MARKOV RANDOM FIELDS
Two prototypical probabilistic graphs are Bayesian networks and Markov networks13. Each is a model of a joint
distribution showing the dependency relationships between the individual random variables. Bayesian networks are

directed graphs which show the direction of causality. The children of a parent node represent the conditionally
independent random variables dependent on the parent. The strength of the dependence is represented in conditional
probability tables for each node. A Markov random field is an undirected graph with a weighted dependency between
pairs of random variables.
The generalization utilizing nonlinear statistical coupling is relevant for both forms of probabilistic graphs. For a
Bayesian network with child variable A and parent’s B and C, Bayes rule for conditional independence

P  A | B, C   P  B | A P C | A P  A / P( B, C )

(8)

is replaced by a Coupled Bayes rule modeling a nonlinear dependence between the conditional probabilities
1/ 

 P  B | Ai   P  C | Ai   1 P  Ai 
P  Ai | B  C  
.
1/ 
  P  B | Ai   P C | Ai   1 P  Ai 

(9)

i

The symbol P  Ai | B  C  is used to represent that the posterior probability is based on the conditional probabilities
given B and C are coupled. The function is written with respect to the ith class of A to show the normalizing sum of the
coupled evidence. A complete design of this type of network requires consideration of how the conditional probability
tables are modified given the coupled Bayes rule.
A coupled Markov random field provides a simpler illustration of how the use of coupled random variables provides an
efficient model of dependence for complex systems. Taking as an example a multivariate normal distribution, the
Markov random field shown in Figure 1 with dependencies shown as bold solid lines, is


 wAA

w

 AB
1
1
P  A, B, C , D, E   exp    xi  wAC
Z
 2 i , j 

 0
 0




wAB
wBB
0
0
0

wAC
0
wCC
wCD
wCE

The matrix is the inverse of the covariance matrix and shows the weights of
non-zero symmetric dependence. The means are set to zero and Z is the
normalization of the distribution. The power and utility of the Markov
random field is that additive exponent terms are also factorable into products
of exponentials which facilitates inferencing and other probabilistic analysis.
Unfortunately, as the size of a system grows inferencing becomes
computationally intractable. While approximations such as Markov chain
Monte Carlo are available, simpler models of the dependency are desirable.
We next show that nonlinear statistical coupling can be used to approximate
the dependency in a Markov random field with hundreds of random
variables14. For the illustration in Figure 1 the coupled Markov random field
forms a coupled multivariate Gaussian



 1
1
P  A  B  C  D  E   exp 
Z
 2 






i , j

 wAA
 0

xi  0

 0
 0


0
wBB
0
0
0

0
0
wCD
wDD
0

0  

0  
wCE x j  .
 
0  
wEE  

(10)

Figure 1. A Markov network defines
pair-wise dependencies between random
variables (bold). Statistical coupling
defines a nonlinear dependency between
a group of random variables (grey).

0
0
wCC
0
0

0
0
0
wDD
0

0  

0  
0 x j  .
 
0  
wEE  

(11)

In this model the pair-wise dependencies are set to zero and the additive arguments of the coupled exponential can be
factored by the coupled product due to the properties of Equation (5). The coupling term can be chosen to approximate
the joint distribution of Equation (10).

5. COUPLED MARKOV MODEL OF THE MULTIPLE FEATURES DATA SET
To illustrate the utility of a probabilistic graph using a coupled Markov random field the Machine Learning Repository’s
Multiple Features Data Set15 is used. This dataset consists of 2000 handwritten numerals in which six feature sets have
been extracted. Three models for fusion of the feature set will be compared; a multivariate normal distribution, single
variate normal distributions fused using naïve Bayes, and the coupled Markov random field; i.e. single variate normal
distributions fused using the coupled product. Table 1 shows the number of parameters, percent correct classification,
and the probability accuracy for each feature set and each model. The multivariate model which requires a matrix of
parameters grows as n-squared; whereas the coupled Markov model adds only one coupling parameter beyond the mean
and variance for each feature.
For each feature set 1000 samples (100 per numeral) are used to calculate the mean and variance for each feature and a
separate set of 1000 samples are used for testing. In the case of the multivariate normal the covariance parameters are
also calculated. For the coupled fusion model, the accuracy measured by the geometric mean is maximized, examining
coupling values between 0 and -2 in increments of -0.05. This is shown in Figure 3a.
The classification performance is similar for each fusion model. Assuming independence (naïve Bayes) or using the
coupling method reduces the classification performance by 2-10%. The ability to maintain reasonable classification
performance is why the assumption of independence is often acceptable. However, for resource management, fusion
with other sources of information, or opportunities to defer decisions in order to collect additional information,16 accurate
probabilities are important for managing uncertainty. The accuracy of the probabilities should be evaluated using the
geometric mean of the true class probabilities, given its connection to Shannon surprisal as explained in Section 3. This
metric shows that the overconfidence of naïve Bayes reduces the accuracy of the probabilities. However, fusion using
the coupled product restores the accuracy of the probabilities while maintaining a significantly less complex graphical
model. Figure 2 shows graphical the ability of the coupled Markov random field to maintain the accuracy of the
Table 1. Table of the number of parameters, percent correct classification, and the probability accuracy for
each feature set and each fusion model. The percent correct classification is reduce by a few percentage
points using naïve Bayes or coupling. The probability accuracy is measured using the geometric mean of the
true class probabilities. The accuracy of the probabilities suffers for naïve Bayes when the features are
correlated. The coupling model has comparable accuracy with the multivariate.

Feature Type &
Number
Fourier
Coefficients
76
Profile
Correlations
216
Principal Comp.
Analysis
64
Pixel Averages
240
Zernike Moments
47
Morphology
6

# of Parameters, Percent Correct Classification, Probability Accuracy
Naïve Bayes
152

76% 21%

432 89%

Coupled

Multivariate

153

76%

41%

5,852

81%

50%

3%

433

89%

73%

46,872

98%

88%

128

90%

56%

129

90% 56%

4,160

480

91% 9%

481

91% 69%

57,840

94%

65%

2,256

83%

60%

94

73%

17%

95

73% 44%

12

70%

25%

13

70%

40%

42

95% 80%

72%

48%

probabilities while significantly reducing the model complexity. The Principal Component Analysis, which are by
design independent, is the only feature set with reasonable accuracy for the naïve Bayes model and no improvement
using the coupled fusion. Accuracy similar to the full covariance matrix may be achievable for smaller than -0.05
coupling.
Figure 3b shows the Risk Profile for the pixel averages feature set. The metric plots the generalized mean of the true
class probabilities (7) as the nonlinear coupling or risk bias is adjusted. The naïve Bayesian model has overconfident
probabilities but good classification performance, which is reflected in the high accuracy given positive risk bias with a
steep reduction in accuracy as the risk bias is reduced. Use of the full covariance matrix shifts the Risk Profile curve to
the left increasing the zero-bias accuracy of the probabilities. Nevertheless, the training of 46,872 parameters for this
model runs the risk of being over fit to the data, which is reflected in the reduction in accuracy with negative risk bias.
The coupled fusion is optimized for accuracy with no risk bias. Since only 433 parameters are used for this model there

Figure 2. Plot of the number of parameters versus the probability accuracy for each feature fused using naïve
Bayes (star), coupling (diamond), and full correlation matrix (square). The accuracy is measured using the
geometric mean of the true class probabilities. Coupling uses only one more parameter than naïve Bayes, but has
accuracy comparable to the full covariance matrix. The Principal Component Analysis is the only set with
independent features, resulting in reasonable accuracy for naïve Bayes and no improvement with the coupling.
Coupling smaller than the -0.05 increments tested may provide a modest.

is less risk of over fitting reflected in the significant improvement in accuracy with negative risk bias. The gain in
robustness is at the cost of less decisive probabilities. In principal the coupling could be tuned to optimize other risk bias
points depending on the requirements of the inference algorithm.

Figure 3. a) The coupling parameter for each feature set was selected by maximizing the geometric mean (or surprisal
probability). b) The Risk Profile for the fusion of the pixel average features shows the characteristics of the three
fusion methods. Naïve Bayes is decisive (positive risk bias), but inaccurate (zero risk bias). The correlation matrix is
accurate but still not robust (negative risk bias). The coupling was optimized for accuracy and trades-off decisiveness
for increased robustness.

6. CONCLUSION
This paper provides an approach to applying coupled random variables to the design of Bayesian and Markov networks.
A numerical experiment using a coupled Markov random field to model the dependence between the features used for
image processing of handwritten numerals is reviewed. The use of the coupled random variable adds one additional
parameter to the naïve Bayesian model, but improves the probability accuracy to within a few percentage points of the
full Markov correlation model. For the case of pixel value and pixel correlation features, which have 100s of feature
values, there is 2 orders of magnitude reduction in the parameters between the full correlation and the coupling model.
Following information theoretic methods, the accuracy of probabilities is measured using the geometric mean of the true
class probabilities, which is a translation into probability space of the measured cross-entropy between the model and the
test samples. Further insight into the characteristics of the inference models is gained utilizing the cross-entropy of the
normalized Tsallis entropy. Translated into probability space this metric is the generalized mean of the true class
probabilities. Positive coupling or risk bias of this metric provides a measure of the decisive quality of the reported
probabilities, while negative coupling or risk bias provides a measure of the robustness of the reported probabilities.
The three models tested show the following characteristics; the full Markov correlation model is decisive and accurate,
but lacks robustness, which may be due to the large number of parameters necessary for the model; naïve Bayes is
decisive, but inaccurate and non-robust, with the benefit of orders of magnitude fewer parameters; the coupled Markov
model, which trains one coupling parameter to maximize the unbiased accuracy, does approximate the accuracy of the
full Markov model, is more robust than either of the other models, but sacrifices decisiveness. Despite the lack of
decisiveness in the coupled Markov model, the classification performance is equal to the naïve Bayes model.
Furthermore, the improved accuracy provides system-level options to applying additional resources to those objects
which report uncertainty.
Applying these methods to deep learning methodologies would be an important advance. The ability to factor the heavytail coupled-Gaussian using the coupled product, opens the possibility of developing networks which could generate
heavy-tail samples for training of higher layers of the network.

ACKNOWLEDGEMENTS
The authors benefited from discussion with Sabir Umarov and Fred Daum regarding modeling long range dependence
with nonextensive statistical mechanics. The research was funded by a Raytheon IDEA project on Coupled Bayesian
Networks.

REFERENCES

[1]

Cheng, B. and Titterington, D.M., "Neural networks: A review from a statistical perspective," Stat. Sci. 9(1), 2-30
(1994).

[2]

Grossberg, S., "Nonlinear neural networks: Principles, mechanisms, and architectures," Neur. Net. 1(1), 17-61
(1988).

[3]

Bengio, Y., Courville, A. and Vincent, P., "Representation learning: A review and new perspectives," Patt. Anal.
Mach. Intell., IEEE Trans. 35(8), 1798-1828 (2013).

[4]

Borges, E.P., "A possible deformed algebra and calculus inspired in nonextensive thermostatistics," Phys. A
340(1-3), 95-101 (2004).

[5]

Tsallis, C., "Nonadditive entropy and nonextensive statistical mechanics-an overview after 20 years," Braz. J.
Phys. 39(2a), 337-356 (2009).

[6]

Marsh, J.A., Fuentes, M.A., Moyano, L.G. and Tsallis, C., "Influence of global correlations on central limit
theorems and entropic extensivity," Phys. A 372(2), 183-202 (2006).

[7]

Nelson, K.P. and Umarov, S., "Nonlinear statistical coupling," Phys. A 389(11), 2157-2163 (2010).

[8]

Lenzi, E.K., Mendes, R.S. and da Silva, L.R., "Normalized Tsallis entropy and its implications for the
nonextensive thermostatistics," Phys. A 295(1), 230-233 (2001).

[9]

Umarov, S., Tsallis, C. and Steinberg, S., "On a q-central limit theorem consistent with nonextensive statistical
mechanics," Milan J. Math. 76(1), 307-328 (2008).

[10]

Nelson, K.P., Scannell, B.J. and Landau, H., "A Risk Profile for Information Fusion Algorithms," Entropy, 13(8),
1518-1532 (2011).

[11]

Gneiting, T. and Raftery, A.E., "Strictly proper scoring rules, prediction, and estimation," J. Am. Stat. Assoc.
102(477), 359-378 (2007).

[12]

Dawid, A.P. and Musio, M., "Theory and Applications of Proper Scoring Rules," arXiv:1401.0398 (2014).

[13]

Pearl, J., "Bayes and Markov Networks: a Comparison of Two Graphical Representations of Probabilistic
Knowledge," UCLA Comp. Sci. Tech. Rep. 860024, (1986).

[14]

Toscano, F., Vallejos, R.O. and Tsallis, C., "Random matrix ensembles from nonextensive entropy," Phys. Rev. E
69(6), 066131 (2004).

[15]

Duin, R., “Multiple Features Data Set,” UCI Machine Learning Repository,
https://archive.ics.uci.edu/ml/datasets/Multiple+Features, Irvine, CA: Univ. Cal.

[16]

Scannell, B.J., McCann, C., Nelson, K. P., Tgavalekos, N. T., "Fusion algorithm for the quantification of
uncertainty in multi-look discrimination," 8th Ann. U.S. Missile Def. Conf., 2010.

