Interpreting Undesirable Pixels for Image Classification on Black-Box Models
Sin-Han Kang Hong-Gyu Jung Seong-Whan Lee





]Sin-Han Kang^1, Hong-Gyu Jung^1 and Seong-Whan Lee^2
[1]Department of Brain and Cognitive Engineering, Korea University, Seoul, Korea
[2]Department of Artificial Intelligence, Korea University, Seoul, Korea
[]{kangsinhan, hkjung00, sw.lee}@korea.ac.kr
 Interpreting Undesirable Pixels for Image Classification on Black-Box Models
    [
    Received: date / Accepted: date
=============================================================================

empty





   In an effort to interpret black-box models, researches for developing explanation methods have proceeded in recent years. Most studies have tried to identify input pixels that are crucial to the prediction of a classifier. While this approach is meaningful to analyse the characteristic of black-box models, it is also important to investigate pixels that interfere with the prediction. To tackle this issue, in this paper, we propose an explanation method that visualizes undesirable regions to classify an image as a target class. To be specific, we divide the concept of undesirable regions into two terms: (1) factors for a target class, which hinder that black-box models identify intrinsic characteristics of a target class and (2) factors for non-target classes that are important regions for an image to be classified as other classes. We visualize such undesirable regions on heatmaps to qualitatively validate the proposed method. Furthermore, we present an evaluation metric to provide quantitative results on ImageNet.

This work was supported by Institute for Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No.2017-0-01779, A machine learning and statistical inference framework for explainable artificial intelligence)
 



§ INTRODUCTION


The tremendous growth of deep networks has brought about the solvability of key problems in computer vision such as object classification <cit.>,<cit.> and object detection <cit.>,<cit.>. At the same time, the complexity of models has also increased, making it difficult for humans to understand the decisions of the model. To improve interpretability of black-box models, explanation methods have been proposed in terms of model inspection <cit.>,<cit.>, <cit.> and outcome explanation <cit.>, <cit.>. These studies focus on visualizing crucial pixels for a model prediction. In other words, if we remove those pixels, the prediction accuracy is significantly decreased. 



However, to obtain diverse interpretation on black-box models, it is also important to investigate pixels that interfere with the prediction. Thus, in this paper, we aim to find undesirable pixels that can improve the accuracy of a target class by perturbing the pixels. For example, Fig. <ref> shows the difference between <cit.> and our proposed method. While Fig. <ref>(a) explains that a hoodie and eyes play a major role to classify the image as an abaya that is a full-length outer garment worn by Muslim women, our method finds regions that help to improve the accuracy. Specifically, Fig. <ref>(b) interprets a mustache as undesirable pixels, which is generally not seen by women. Thus, perturbing the mustache leads to the accuracy improved for the abaya class. 

Figure  <ref> clearly shows that finding undesirable pixels of a target class can ease off the uncertainty about the decision of black-box models. Thus, we further define undesirable pixels with two different concepts. The first is factors for a target class (F-TC), which hinder that black-box models identify intrinsic characteristics of a target class. The second is factors for non-target classes (F-NTC) that are important regions for an image to be classified as other classes. In the following sections, we will mathematically elaborate on how these two different concepts interpret undesirable pixels for a model prediction. Then, we visually validate our idea on heatmaps and qualitatively evaluate the proposed method on ImageNet.



§ RELATED WORKS

Class activation map (CAM)<cit.> and Grad-CAM <cit.> analyze the decision of neural networks on heatmaps by utilizing activation maps of the last convolution layer in CNNs. Layer-wise relevance propagation (LRP) <cit.> computes gradients of the prediction score by exploiting a backward operation in neural networks. Model agnostic methods <cit.>, <cit.> approximate the perimeter decision boundary of a black-box model to a simple model such as logistic regression and decision tree. Local rule-based explanations (LORE) <cit.> applys a genetic algorithm to build rule-based explanations, offering a set of counterfactual rules. Contrastive explanations methods (CEM)<cit.> visualize a pertinent positive (PP) and a pertinent negative (PN) by using perturbation. But, PN is useful only when the meaning of classes for different inputs are similar to each other. Lastly, the most similar work to ours is local explanation<cit.> that learns a mask by perturbing important regions to a prediction. However, these methods do not clearly consider undesirable pixels of an image for a target class.



§ METHODS
 
Given an image X∈ℝ^H × W × 3, we generate a blurred image by applying Gaussian blur h(X)=g_σ,s(X) where σ and s are standard deviation and kernel size, respectively. In order to replace specific pixels in X with blurred pixels, we define a mask M∈[ 0,1 ]^d, where d is smaller than H × W. Thus, a perturbed image is generated by the masking operator <cit.> as follows.

    Q(X;M',h)=X ∘ M'+h ( X ) ∘( 1-M' ),

where M' = Inp ( M ) is an interpolated mask and Inp is a bilinear interpolation function. ∘ denotes the element-wise multiplication.
Given a black-box model f and an accuracy f_k(X) for a target class k, we expect that the perturbed image makes f_k(X) ≪ f_k( Q(X;M',h) ). In other words, the goal is to find an optimal mask M^* that improves the accuracy for a target class and an objective function can be defined as follows.

    M^*=_M f_k( Q(X;M',h) ).

Since <cit.> shows that total-variation (TV) norm and l1 norm can produce reasonable and precise interpretability for the masking operation, we also apply such regularizers to our objective.

    M^*=_M f_k( Q(X;M',h) ) -ℛ_M,

where R_M=λ_1∑_i,j  ((M_i+1,j-M_i,j)^β + 
(M_i,j+1-M_i,j )^β )^1 β+λ_2‖ 1-M ‖_1. λ_1, λ_2 and β
are hyper-parameters. 



However, this objective function generates different masks for each trial as shown in Fig. <ref> and do not provide consistency of an explanation. We conjecture that this is due to the softmax operation. Given an output before softmax y, f_k( Q (X;M',h ))=exp(y_k) ∑_i exp(y_i) can be higher when increasing exp(y_k) or decreasing ∑_i exp(y_i). That is, improving f_k( Q (X;M',h ) ) is affected by not only the output for a target class but also those for other classes. 

In order to solve this problem, we propose two types of regularizers for obtaining undesirable pixels. We first define factors for a target class (F-TC).

    ℛ_F-TC = γ‖1  N-1∑_i,i k{    f_i' ( Q (X;M',h )) 
        - f_i' ( X )}‖_2,

where f_i' ( ·) denotes the output before softmax for the i-th class, k is the index of the target class, γ is a hyper-parameter and N is the total number of classes. This regularizer forces the objective function into focusing on the target class itself. In other words, ℛ_F-TC finds the pixels that hinder intrinsic characteristic to be classified as the target class. The final objective function can be expressed as 

    M^* = _M f_k ( Q(X;M',h) ) -ℛ_M 
    - ℛ_F-TC.


Secondly, we define factors for non-target classes (F-NTC).

    ℛ_F-NTC = γ‖{    f_k' ( Q (X;M',h ) ) 
        - f_k' (X) }‖_2,

which encourages to find undesirable pixels by focusing on other classes except for the target class. When applying ℛ_F-NTC, we modify Eq. <ref> as follows.

    M^* = _M∑_i, i k   { f_i (Q(X;M',h)) }
        +ℛ_M + ℛ_F-NTC.


In the following section, we show several case studies to understand how these regularizations behave according to their definitions.




§ EXPERIMENTS



 §.§ Experimental Settings

We use VGG-19 <cit.> and ResNet-18 <cit.> pretrained on the ImageNet <cit.> and solve optimization problems using Adam <cit.>. We set the learning rate to 0.1 and iterations to 200. We use the hyper-parameters λ_1=1.7, λ_2 =3.0, β=2 and γ=0.3. A mask 28×28 is interpolated by 224 × 224 size by upsampling. The standard deviation σ and kernel size s for the Gaussian kernel are set to 5 and  11, respectively.



 §.§ Interpretability

In Sect. <ref>, we explained a main objective function with (1) TV norm and l1norms. Further, additional regularizers such as (2) F-TC and (3) F-NTC were proposed. We now compare interpretability among the three 
cases. First, as shown in Fig. <ref>, when merely using TV and l1 norms, the learned masks are generated irregularly for each trial. This makes us difficult to understand the decision of black-box models. On the other hands, Fig. <ref> shows that F-TC and F-NTC provide consistent visual interpretation. Moreover, each regularizer highlights the regions corresponding to their definitions such as Eq. <ref> and Eq. <ref>. Specifically, F-TC explains that the windows are undesirable pixels to identify the intrinsic characteristic of the mobile home. F-NTC explains that the parts of the truck are undesirable pixels since those are more important to classify other classes such as a truck. In this way, our algorithm can be exploited to understand the decision of black-box models.







 §.§ Qualitative Results

We provide more examples to qualitatively evaluate the proposed method. We used VGG-19 for all experiments of this section.

As illustrated in Fig. <ref>, the original image is classified as the tusker with the accuracy of 72.1%. When we set the African elephant as a target class, F-TC perturbs the end part of the horn, which results in improving the accuracy for the African elephant class. These results imply that the model generally regards the length of the horn as crucial features to distinguish between the tusker and the African elephant. More importantly, this is consistent with the fact that the horn of a tusker is longer than an African elephant. Thus, we argue that our method can provide reasonable interpretation about how trained networks distinguish similar classes.

Another example can be shown in Fig. <ref>. For ski and chocolate souce classes, F-TC highlights the human body connected to ski equipments and  the chocolate sauce that is falling from the spoon. These results suggest that portions connected to a target class have negative effect on a classification for a target class.




 §.§ Quantitative Results

We present the following evaluation metric to measure how effectively our method finds undesirable pixels.

    ϕ = 𝔼_X[(f_h( Q(X;M',h ) )-f_h(X)   1-f_h(X) )*100],

where h is a class that has the highest accuracy for an image. 1-f_h(X) is the residual accuracy that can be improved from f_h(X). Thus, Eq. <ref> measures the relative accuracy improvement. We randomly select 1,000 images from the ImageNet and compare results between F-TC and F-NTC with VGG-19 and ResNet-18. In Table <ref>, we observe that the accuracy can be effectively improved by perturbing undesirable pixels. We also measure the ratio of the number of undesirable pixels to the image size 224 × 224. In this case, we use the pixels that have magnitude above a threshold 0.6. Table <ref> shows that both F-TC and F-NTC yield a small number of undesirable pixels that are below 4%. 






§ CONCLUSION

We proposed an explanation method that visualizes undesirable regions for classification. We defined the undesirable regions by two terms. The first is factors for a target class, which hinder that black-box models identify intrinsic characteristics of a target class and the second is factors for non-target classes that are important regions for an image to be classified as other classes. We showed the proposed method successfully found reasonable regions according to their definitions and by perturbing the undesirable pixels, we could improve the accuracy for the target class.
ieee



