A copula-based visualization technique for a neural network
Yusuke Kubo Yuto Komori Toyonobu Okuyama Hiroshi Tokieda
cs.LG stat.ML

Estimates for distribution of suprema of solutions to higher-order partial differential equations with random initial conditions
    [
    December 30, 2023. .../latex/6he-sdm-2/6he-sdm-2-3.tex
================================================================================================================================




[1]Both authors contributed equally to this study.
[2]Corresponding author for this study.



Interpretability of machine learning is defined as the extent to which
humans can comprehend the reason of a decision. However, a neural network
is not considered interpretable due to the ambiguity in its decision-making process.
Therefore, in this study, we propose a new algorithm that reveals which feature values 
the trained neural network considers important and which paths are mainly
traced in the process of decision-making.
In the proposed algorithm, the score estimated by the correlation
coefficients between the neural network layers
that can be calculated by applying the concept of a pair copula was defined.
We compared the estimated score with the feature importance values of
Random Forest,
which is sometimes regarded as a highly interpretable algorithm, in the experiment and
confirmed that the results were consistent with each other. 
This algorithm suggests an approach for compressing a neural network and its parameter tuning
because the algorithm identifies the paths that contribute
to the classification or prediction results.












§ INTRODUCTION

Interpretability of machine learning is being vigorously
discussed currently. Deep learning has an overwhelming
performance compared to other machine learning techniques
in the field of image recognition such as generic
object recognition. It is applied in a wide range
of fields, but is hesitated to be introduced under
the normal social conventions due to the difficulty of
interpretability in some cases. As for autonomous
driving technology, deep learning and
deep Q-network<cit.> 
bring the possibility of producing amazing
results<cit.>. 
However, the cause of accidents is unclear if traffic
accidents occur due to incorrect recognition. 
This problem also happens in the clinical practice.
When a doctor makes a diagnosis based on the machine
learning predictions, patients are not able to accept
the diagnosis if the predictions show the disease without 
appropriate reasons.
On top of that, security problems in machine
learning models against Adversarial
Attack<cit.> 
break gradually to the surface.
It is recommended at the political level that the process
of machine learning decision-making is clear and that humans
should employ machine learning predictions
with a deep understanding of how they function<cit.>.


What is the interpretability or explainability of machine
learning after all?
A good reference<cit.> defined the interpretability of machine learning as 
“Interpretability is the degree to which
a human can understand the cause of a decision."
Based on this definition, the Linear Model and Decision
Tree can be considered as interpretable 
because in the Linear Model, it becomes clear which parameters are more important while in the Decision Tree, there is no ambiguity in the process of if-then-else statement.
A rule-based algorithm can also be termed interpretable.


However, a neural network is not interpretable based on the 
definition given above because the process of decision-making
is ambiguous although several novel approaches to clarify 
the process have been attempted.
Among them are the sensitivity analysis method that calculates the influence rate of the output by changing the input feature values
slightly and expresses it in a human-readable format<cit.>; <cit.>,
the method that traces the network path from an output
to an input in reverse and identifies the effective input features<cit.>, 
the method that replaces the trained model with the Linear
Model as an interpretable model and evaluates the
important input features<cit.>,
the method that weighs the input feature values and predicts
the output using of them<cit.>; <cit.> 
and the method that mimics the uninterpretable models by using
the interpretable models<cit.>; <cit.>; <cit.>; <cit.>.


In this study, we propose a new algorithm that enables us 
to clearly understand and visualize the decision-making 
process of a neural network.
The algorithm is realized by considering a neural
network as a graphical model and calculating the correlation
coefficients between its layers.


A copula is a general technique that is extremely useful 
for calculating the correlation coefficients, and therefore,
we first introduce its theoretical background in
the following section.
In Section <ref>, the new algorithm is proposed 
using the concepts introduced in Section <ref>.
In Section <ref>, an experiment using the
algorithm to determine the paths important in a neural network
for classification or prediction is evaluated using 
a well-known dataset.
Notably, the visualization result
is presented in Section <ref>, which presents
our main findings.
A comparison with Random Forest is described in
Section <ref>, which discusses the validity
of interpretability. Finally, the conclusion and future works
are summarized in Section <ref>.






§ PRELIMINARIES


In this section, the concepts of a copula, correlation, and
the relationship between them are introduced to define
the proposed algorithm.



 §.§ Copula


Let X_1, … , X_n be random variables
and x_1, … , x_n be their values.
F_i is the distribution function of the random
variable
X_i for all x_i in ℝ, therefore, 
F_i = P(X_i≤ x_i).
We focus on the multivariate cumulative joint distribution 
F (x_1, … ,x_n) to consider the behavior
of multiple random variables.
There exists the following relationship between 
the joint distribution and the marginal distribution.

<cit.> 
Let F_1(x_1),...,F_n(x_n) be the marginal distributions
of F(x_1, … ,x_n).
Then, there exists a functional C such that

    F(x_1,...,x_n) = C(F_1(x_1), … ,F_n(x_n)).

If F_i(x_i) are continuous, then C is unique.

Here, the functional C is known as a copula and it gives
an alternative expression of the multivariate cumulative distribution
functions, i.e., it can be used for scale-free measures of 
dependencies on distributions such that 
C(F_1, … , F_n) is the joint distribution
with the marginal distributions as the variables.
This means that the continuous multivariate marginal distributions
are able to be decomposed into F_1, …, F_n
that expresses the behavior of random variables and
a copula C that expresses the dependency structure of 
the random variables.

A copula with joint probability distributions,
in the case of two variables of Theorem <ref>,
is sometimes called a `pair' copula.
A pair copula can be written as a graphical model.
Bedford and Cooke et al. introduced a novel
technique that homologizes them<cit.>.





 §.§ Correlation

A well-known indicator of variable dependencies is
the correlation coefficient. 
The correlation coefficient between the random variables
X and Y is defined as
ρ = E((X-μ_1)(Y-μ_2))/σ_1σ_2,
where μ_1 and μ_2 are the means of X and Y,
respectively, while σ_1 and σ_2 are the standard deviations of X and Y, respectively.
If a joint probability distribution between two variables has
a correlation coefficient, a range of ρ satisfies
-1 ≤ρ≤ 1.
The joint probability distribution for X and Y has a linear
correlation with an equation y=a + bx, b > 0, if ρ = 1.
This means that P(Y=a+bX)=1. Similarly, if ρ = -1,
we get the same result except b < 0.
If ρ≠± 1, is there a line in the xy-plane
where the joint probability for X and Y is concentrated
with a width? 
Under certain constraints, this line actually exists. 
At this time, the ρ can be regarded as a
measure of strength which the joint probability for
X and Y are concentrated on
the line<cit.>.

Clearly, linear correlation cannot represent dependencies 
that are non-linear. Rank correlation can be used to evade
these kinds of problems that represent a dependency between
the variables. Rank correlation is not a value of each variable
itself but is a correlation based on a rank of each variable
according to some criteria.

In this study, we used the Kendall rank correlation
coefficient (Kendall's τ) as the rank correlations.
The population version of Kendall's τ is defined
as the probability of the concordance minus the
probability of the distance:

    τ =      
    P(sgn{(X_1 - X_2) (Y_1 - Y_2) } = 1) 
        -P(sgn{(X_1 - X_2) (Y_1 - Y_2) } = -1)


If a pair of sgn{(X_1 - X_2) (Y_1 - Y_2) } = 1,
then we call this relationship concordance and there is an
increasing relationship between the random variables X and Y.
Otherwise, we call this relationship
disconcordance and there is a decreasing relationship
between the variables X and Y.
Furthermore, a range of Kendall's τ is -1 ≤τ≤ 1.
Kendall's τ can be expressed using a copula.

<cit.> Let τ denote the difference between the probabilities
of concordance and disconcordance of (X_1, Y_1)
and (X_2, Y_2), then

    τ = 4 ∫∫_I^2 C_2 (u, v) d C_1(u, v) - 1

where C_1 and C_2 are the copulas of (X_1, Y_1) and (X_2, Y_2), respectively. The unit square I^2 is the product I × I where I = [0, 1].


In the next section, we explain how to visualize a neural network by using Kendall's τ.







§ ALGORITHM



A neural network having nodes and edges can be drawn
as shown in Figure <ref>.
Therefore, the correlation coefficients between the neural network
layers can be calculated by applying the concept of a pair
copula described in the previous section.
The paths are obtained by connecting the input layer
to the output layer in a forward direction.
An example of the paths is denoted as red nodes and edges.
The calculation result reveals which paths in a
neural network are important for the classification or prediction
calculation of the correlation coefficients. In this section,
we propose a new algorithm that makes the decision-making
of a neural network clear, known as the Copula-based
Visualization Technique (CVT) for a neural network, 
as shown in Algorithm <ref>.




In the initial setting, we are required to train a neural network, the cumulative marginal distribution function, correlation matrix, and paths.
In the procedure step, we need to compute the variance of the convolution correlation coefficient: 

    VaR(CCC).

This is the importance degree of each path for visualizing and understanding the neural network model. The variance of CCC expresses the path sensitivity.

For example, we can arrange the ranking in a descending order of VaR(CCC).




In general, it is not possible to understand how a model makes
a judgment based only on weights in a neural network<cit.>; <cit.>.
For this reason, it is impossible to know how the feature value
affects the classification or prediction result and which node is
highly sensitive to the classification or prediction result
only by looking at the weights.







§ VISUALIZATION OF A NEURAL NETWORK


The experiment with CVT to a neural network is described
in this section. Subsection <ref>
presents the experimental set up such as the dataset and 
the calculation of the probability density functions (PDFs) 
and cumulative distribution functions (CDFs).
Subsection <ref> details the simultaneous 
visualization of the important feature values
and the paths that are traced in the process of decision-making.
This result presents our main contribution.
The feature values obtained by CVT were ranked
according to VaR(CCC) and were compared with the
Random Forest result in Subsection <ref>.
All the source codes to demonstrate this experiment are
available at Github[<https://github.com/covit2019/analysis_codes>].




The Fisher's Iris data set[<http://archive.ics.uci.edu/ml/datasets/Iris>] included in the scikit-learn package[<https://scikit-learn.org/stable/auto_examples/datasets/
plot_iris_dataset.html>]
was used to train the neural network.
The values of sepal length, sepal width,
petal length, and  petal width were used
as the input feature values which defined
x_0, x_1, x_2 and x_3, respectively.
The labels of Setosa, Versicolor, Virginica were used as
one-hot encoding vector.
Here, Setosa is pred_0, Versicolor pred_1, Virginica pred_2.
The structure of the trained neural
network is illustrated in Figure <ref>, comprised of 
four nodes (red) in the input layer, six (blue) in the first hidden layer, six (blue) in the second hidden layers, and three (green) in the output layer.
All the nodes in the hidden layers and the output layer express the values 
through the activation functions.
Rectified Linear Unit (ReLU) was used as the activation function 
in the hidden layers and Softmax was used in the output function.
This model is a standard neural network model.
The data used for training was 120 samples out of 150 and the
accuracy rate was approximately 97.5%.








CDFs are required to calculate the correlation coefficients as
described in Section <ref>.
Therefore, we first created histograms of the outputs of
the nodes at the time of prediction.
These histograms are equivalent to PDFs as shown 
in the left-side of Figure <ref>.
Here, the PDFs at the input nodes were, in fact, the feature values. 
Next, we integrated them to obtain the CDFs
as shown in the right-side of Figure <ref>.
The Y-axis values of the CDFs histograms were used for
this calculation.




 §.§ Calculation of Correlation Coefficients
            Between Neural Network Layers



As shown in Figure <ref>, the number of paths from the input
to the second hidden layer (Hidden1)
in the neural network is 4 × 6× 6 = 144 and
each path is branched off to the three outputs.
An example path from the 144 paths is drawn as a red line 
and the branch lines are drawn as blue dotted lines.


Next, VaR(CCC) for each path was calculated using the CVT algorithm described in Section <ref>. The calculation of the correlation coefficients was executed by the noble C++ library: vinecopulib<cit.>.







Table <ref> presents the top 10 important path
ranking in a descending order of VaR(CCC). This table presents which
paths are important for decision-making of the neural network
from which the feature values x_2 and x_3
are especially important. Besides that, Figure <ref> shows
the visualization of all the paths. The importance defined as
the sum of VaR(CCC) of paths through the edge is portrayed as a contrasting density, and the density corresponding
to blue (less important) is set transparent to emphasize the important paths.

The values listed in Table <ref> are sometimes
the same because the CDFs have discrete histograms.
This problem can be avoided by making the bin size of
the histograms finer or by using CDFs, which are estimated 
through function fitting.



 §.§ Comparison with Random Forest



We compared the CVT results with those obtained by 
Random Forest for the validity of interpretability. 
Random Forest implemented in the scikit-learn package[<https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html>]
with the default parameters were used.
The data used for training had 120 samples out of 150 and
the accuracy rate was approximately 97%. This accuracy rate
was extremely close to that of the neural network experimented 
in the previous subsection.
Random Forest, which is sometimes regarded as one of the 
highly interpretable algorithms, can calculate the importance 
of the feature values. 
Therefore, we compared the Random Forest feature
importance values with the following:

    E^x_i(VaR(CCC))

where E^x_i( · ) is an operator that takes the expected
value with respect to the feature values x_i and normalized
to be compared.
For example, E^x_2(VaR(CCC)) means that the average of all paths begin with x_2, i.e., that of the first row, the second, the fifth, ..., and so on as shown in Table <ref>.
Figure <ref> shows the result.
It was not compared directly because the importance was calculated 
using different methods, but the results were consistent with each other. This means that CVT can derive the importance
of feature values in a neural network.







§ RELATED WORKS

As summarized in the introduction, sensitivity analysis method<cit.>; <cit.> 
and reverse tracing method<cit.> have been studied intensively in the recent years.
CVT is complementary to the aforementioned methods in terms of explaining the models as these methods do not address the problem of explaining
important feature values and paths that contribute to the results of classification or prediction through all training data.
In other words, CVT is not appropriate for measuring the effect and contribution of feature values
of individuals
when only a portion of the data is focused on.
It is difficult to say which of these methods is better, and therefore, an appropriate method should be chosen based on the situation.






§ CONCLUSION AND FUTURE WORKS


In this study, we proposed a Copula-based Visualization Technique (CVT)
for a neural network.
CVT can easily visualize the decision-making process of a neural network
using correlation coefficients.
Information regarding the feature values that are considered to be important by the trained neural
network and paths that are mainly
traced in the process of decision-making are obtained by the proposed
algorithm. The experimental result of the proposed algorithm was consistent with
the importance values estimated using Random Forest.
This suggests that CVT can be used for the
interpretability of a neural network.

CVT was experimented using a simple neural network but can also be extended to other
algorithms, including a deep neural network that can be considered as a graphical model.
CVT can serve as a general method for machine
learning interpretability. Kendall's τ was used as the
correlation coefficients, but other correlation
coefficients, by all means, can also be used.
Future works include the theoretical analysis and the use of other correlation coefficients for CVT.
Additionally, CVT can suggest
an approach for compressing a neural network as doing so helps in identifying the nodes and edges that contribute to the classification or
prediction results. CVT can also use parameter
tuning for the abovementioned reason. Additionally, CVT may be able
to reveal the reasons for Adversarial
Attack cheating a neural network,
which can also be considered for future work.








unsrt  




1






[(Mnih, 2015)]Mnih2015 V. Mnih, et al. (2015).
    Human-level control through deep reinforcement learning.
    Nature 518: 529–533.
[(Sallab, 2017)]Sallab2017 A.El. Sallab, M. Abdou, P. Etienne and Y. Senthil (2017).
    Deep reinforcement learning framework for autonomous driving.
    Electronic Imaging, Autonomous Vehicles and Machines : 70–76.



    

[(Madry, 2018)]Madry2018 A. Madry, A. Makelov, L. Schmidt, D. Tsipras and A. Vladu (2018).
    Towards Deep Learning Models Resistant to Adversarial Attacks.
    International Conference on Learning Representations (ICLR).





[(Goodman, 2017)]Goodman2017 B. Goodman and S. Flaxman (2017).
    European union regulations on algorithmic decision making and a "right to explanation”.
    AI Magazine 38: 50–57.


[(Molnar, 2019)]Molnar2019 C. Molnar (2019).
    Interpretable Machine Learning. A Guide for Making Black Box Models Explainable.
    <https://christophm.github.io/interpretable-ml-book/> (accessed 2019-09-25).


[(Zeiler, 2014]Zeiler2014 M.D. Zeiler and R. Fergus (2014).
    Visualizing and Understanding Convolutional Networks.
    European Conference on Computer Vision (ECCV): 818–833.
[Smilkov, 2017)]Smilkov2017 D. Smilkov, N. Thorat, B. Kim, F. Viégas and M. Wattenberg (2017).
    SmoothGrad: removing noise by adding noise.
    arXiv:1706.03825v1 [cs.LG]. 


[(Bach, 2015)]Bach2015 S. Bach, A. Binder, G. Montavon, F. Klauschen, K-R. Muller and W. Samek (2015).
    On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation.
    PLOS ONE 10(7): e0130140.


[(Ribeiro, 2016)]Ribeiro2016 M.T. Ribeiro, S. Singh and C. Guestrin (2016).
    "Why Should I Trust You?": Explaining the Predictions of Any Classifier.
    Knowledge discovery and data mining (KDD): 1135–1144.


[(Vaswani, 2017]Vaswani2017 A. Vaswani, et al. (2017).
    Attention is all you need.
    Advances in Neural Information Processing Systems (NeurIPS): 5999–6009.

[(Bahdanau, 2014)]Bahdanau2014 D. Bahdanau, K. Cho and Y. Bengio (2014).
    Neural Machine Translation by Jointly Learning to Align and Translate.
    arXiv:1409.0473v7 [cs.CL].


[(Bucila, 2006)]Bucila2006 C. Bucila, R. Caruana and A.N. Mizil (2006).
    Model Compression.
    Knowledge discovery and data mining (KDD): 535–541.
    
[Ba, 2014]Ba2014 J. Ba and R. Caruana (2014).
    Do deep nets really need to be deep ?
    Neural Information Processing Systems (NeurIPS): 2654–2662.
    
[(Hinton, 2015)]Hinton2015 G. Hinton, Oriol Vinyals and Jeff Dean (2015).
    Distilling the Knowledge in a Neural Network.
    arXiv:1503.02531v1 [stat.ML].
    
[(Hendricks, 2016)]Hendricks2016 L.A Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele and T. Darrell (2016).
    Generating Visual Explanations.
    European Conference on Computer Vision (ECCV): 3–19.
 

[(Sklar, 1959)]Sklar1959 A. Sklar (1959).
    Fonctions de répartition à n dimensions et leurs marges.
    Publ. Inst. Statist. Univ. Paris (in French), 8: 229–231.

[(Bedford, 2001)]Bedford2001 T. Bedford and R.M. Cooke (2001).
    Probability density decomposition for conditionally dependent random variables modeled by vines.
    Annals of Mathematics and Artificial Intelligence, 32: 245–268.

[(Hogg, 2012)]Hogg2012 R.V. Hogg, J.W. Mckean, A.T. Craig (2012).
    Introduction to Mathematical Statistics 7th edition.
    Prentice Hall, p. 73–110.

[(Nelsen, 2010)]Nelsen2010 R.B. Nelsen (2010).
    An Introduction to Copulas Second edition.
    Springer, p. 157–225.










[(Haufe, 2014]Haufe2014 S. Haufe, et al. (2014).
    On the interpretation of weight vectors of linear models in multivariate neuroimaging.
    NeuroImage 87: 96–110.
[Kindermans, 2018)]Kindermans2018 P.J. Kindermans, et al. (2018).
    Learning how to explain neural networks: PatternNet and PatternAttribution.
    International Conference on Learning Representations (ICLR).


[(Nagler, 2019)]Nagler2019 T. Nagler et al. (2019).
    A C++ library for vine copula models.
    <https://vinecopulib.github.io/vinecopulib/> (accessed 2019-09-01).





