arXiv:1711.11443v1 [cs.LG] 30 Nov 2017

ConvNets and ImageNet Beyond Accuracy:
Explanations, Bias Detection, Adversarial Examples and Model Criticism
Pierre Stock
Facebook AI Research

Moustapha Cisse
Facebook AI Research

pstock@fb.com

moustaphacisse@fb.com

Abstract
ConvNets and Imagenet have driven the recent success
of deep learning for image classification. However, the
marked slowdown in performance improvement, the recent
studies on the lack of robustness of neural networks to
adversarial examples and their tendency to exhibit undesirable biases questioned the reliability and sustained development of these methods. This work investigates these
questions from the perspective of the end-user by using human subject studies and explanations. We experimentally
demonstrate that the accuracy and robustness of ConvNets
measured on Imagenet are underestimated. We show that
explanations can mitigate the impact of misclassified adversarial examples from the perspective of the end-user and
we introduce a novel tool for uncovering the undesirable biases learned by a model. These contributions also show that
explanations are a promising tool for improving our understanding of ConvNets’ predictions and for designing more
reliable models1 .

Figure 1: Top: CNNs performance evolution on Imagenet.
Bottom: some images sampled from the Internet misclassified by a ResNet-101.

genet features. Besides, convolutional architectures initially
developed for image classification such as Residual Networks are now routinely used for machine translation [13]
and speech recognition [45].

1. Introduction
Convolutional neural networks [29, 27] and Imagenet [9,
38] (the dataset and the challenge) have been instrumental to the recent breakthroughs in image classification. Imagenet has provided ConvNets with the data they needed
to demonstrate their superiority compared to the previously
used handcrafted features such as Fisher Vectors [35]. This
success has triggered a renewed interest in convolutional
approaches. Novel architectures such as ResNets [21] and
DenseNets [22] have been introduced to improve the state
of the art performance on Imagenet. The impact of this virtuous circle has permeated all aspects of computer vision
and deep learning at large. Indeed, the use of feature extractors pre-trained on Imagenet is now ubiquitous. For example, the state of the art image segmentation [15, 20] or pose
estimation models [23, 6] heavily rely on pre-trained Ima1 corresponding

Since 2012, the top-1 error of state of the art (SOTA)
models on Imagenet has been reduced from 43.45% to
22.35%. Recently, the evolution of the best performance
seems to plateau (see Figure 1) despite the efforts in designing novel architectures [21, 22], introducing new data augmentation schemes [46] and optimization algorithms [26].
Concomitantly, several studies have demonstrated the lack
of robustness of deep neural networks to adversarial examples [42, 16, 7] and raised questions about their tendency
to exhibit (undesirable) biases [37]. Adversarial examples [42] are synthetic images designed to be indistinguishable from natural ones by a human, yet they are capable
of fooling the best image classification systems. Undesirable biases are patterns or behaviors learned from the data,
they are often highly influential in the decision of the model

author: moustaphacisse@fb.com

4321

but are not aligned with the values of the society in which
the model operates. Examples of such biases include racial
and gender biases [5]. While accuracy has been the leading
factor in the broad adoption of deep learning across the industries, its sustained improvement together with other desirable properties such as robustness to adversarial examples and immunity to biases will be critical in maintaining
the trust in the technology. It is therefore essential to improve our understanding of these questions in the context
of Imagenet (the primary image classification benchmark),
especially from the perspective of the end-user.
In this work, we take a step in this direction by assessing the predictions of SOTA models on the validation set of
Imagenet.
We show that human studies and explanations can be
valuable tools to perform this task. Human studies yield
a new judgment of the quality of the model’s predictions
from the perspective of the end-user, in addition to the traditional ground truth used to evaluate the model. We also
use feature-based and example-based explanations. On the
one hand, explaining the prediction of a black box classifier
by a subset of the features of its input can yield valuable insights into the workings of the models and underline the essential features in its decision. On the other hand, examplebased explanations provide an increased interpretability of
the model by yielding examples representative of the distribution of a given category as captured by the model.
The particular form of example-based explanation we use
is called model criticism [24]. It combines both prototypes
and criticisms and is proven to capture better the complex
distributions. Therefore, it facilitates human understanding.
Our main findings are summarized below:

solved. In the next section, we summarize the related work
before presenting our experiments and results in details.

2. Related Work
Adversarial examples. Deep neural networks can
achieve high accuracy on previously unseen examples
while being vulnerable to small adversarial perturbations of
their inputs [42]. Such perturbed inputs, called adversarial
examples, have recently aroused keen interest in the community [17, 42, 43, 7]. Several studies have subsequently
analyzed the phenomenon [11, 40, 12, 12] and various
approaches have been proposed to improve the robustness
of neural networks [34, 8, 44, 46, 19]. More closely
related to our work are the different proposals aiming at
generating better adversarial examples [17, 32]. Given an
input (train or test) example (x, y), an adversarial example
is a perturbed version of the original pattern x̃ = x + δx
where δx is small enough for x̃ to be undistinguishable
from x by a human, but causes the network to predict an
incorrect target. Given the network gθ (where θ is the set
of parameters) and a p-norm, the adversarial example is
formally defined as:

x̃ = argmax ` gθ (x̃), y
(1)
x̃:kx̃−xkp ≤

where  represents the strength of the adversary. Assuming the loss function `(·) is differentiable, [40] propose to
take the first order taylor expansion of x 7→ `(gθ (x), y) to
compute δx by solving the following simpler problem:
T
(2)
x̃ = argmax ∇x `(gθ (x), y) (x̃ − x)
x̃:kx̃−xkp ≤

When p = ∞, then x̃ = x +  · sign(∇x `(gθ (x), y)) which
corresponds to the fast gradient sign method [17]. If instead p = 2, we obtain x̃ = x +  · ∇x `(gθ (x), y) where
∇x `(gθ (x), y) is often normalized. Optionally, one can perform more iterations of these steps using a smaller step-size.
This strategy has several variants [32, 28]. In the rest of
the paper, we refer to this method by iterative fast gradient
method (IFGM) and will use it both to measure the robustness of a given model and to perform model criticism.

• The accuracy of convolutional networks evaluated on
Imagenet is vastly underestimated. For example, the
top-1 error of a ResNet-101 trained on Imagenet and
evaluated on the standard validation set is 9.47% instead of 22.69% when the errors are validated by human subjects. Similarly, the top-5 error is 1.94% instead of 6.44%. This observation holds across models.
• The robustness of ConvNets to adversarial examples is
also underestimated. In addition, providing explanations helps to mitigate the misclassification of adversarial examples from the perspective of the end-user.

Model Criticism. Example-based explanations are a
well-known tool in the realm of cognitive science for facilitating human understanding [41]. They have extensively
been used in case-based reasoning (CBR) to improve the interpretability of the models [1, 4, 25]. In most cases, it consists in helping a human to understand a complex distribution (or the statistics captured by a model) by presenting her
with a set of prototypical examples. However, when the distribution or the model for which one is seeking explanation
is complex (as is often the case with real-world data), prototypes may not be enough. Recently, Kim et al. [24] have

• Model Criticism is a good tool for detecting the biases
in the predictions of the models. Further, adversarial
examples can be effectively used for model criticism.
Similar observations to our first point existed in prior
work [2]. However, the scale, the conclusions and the implications of our study are different. Indeed we consider that if
top-5 error is the measure of interest, Imagenet is (almost)
4322

where the best explanation minimizes a local weighted loss
L between g and h in the vicinity πx of x, regularized by
the complexity Ω(h) of such an explanation. The authors
restrict h ∈ H to be a linear model such that h(z 0 ) = wh z 0 .
They further define the vicinity of two samples using the
exponential kernel

proposed to use, in addition to the prototypes, data points
sampled from regions of the input space not well captured
by the model or the prototypes. Such examples, called criticism, are known to improve the human’s mental model of
the distribution.
Kim et al [24] introduced MMD-critic, an approach inspired by bayesian model criticism [14] to select the prototypes and the critics among a given set of examples.
MMD-critic uses the maximum mean discrepancy [18] and
large-scale submodular optimization [3]. Given a set of examples D = {(x, y)}ni=1 , let S ⊂ {1, . . . , n} such that
DS = {xi , i ∈ S}. Given a RKHS with kernel function
k(·, ·), the prototypes of MMD-critic are selected by minimizing the maximum mean discrepancy between D and DS .
This is formally written as:
max
S∈2[n] ,|S|≤m

Jb (S)

πx (z) = e

and define the local weighted loss L as:
X
2
L(g, h, πx ) =
πx (z) (g(z) − h(z 0 ))

(6)

where Z if the dataset of n perturbed samples obtained from
x by randomly activating or deactivating some super-pixels
in x. Note that z 0 denotes the one-hot encoding of the superpixels whereas z is the actual image formed by those superpixels. Finally, the interpretability of the representation is
controlled by
(
0
if kwh k0 < K
(7)
Ω(h) =
+∞ otherwise

(3)

i,j∈S

Given a set of prototypes, the criticisms are similarly selected using MMD to maximize the deviation with the prototypes. The objective function in this case is regularized to
promote diversity among the criticisms. A greedy algorithm
can be used to select both prototypes and criticisms since
the corresponding optimization problems are submodular
under certain conditions [3]. In this work, we use MMDcritic as a baseline for example-based explanations.

The authors solve this optimization problem by first selecting K features with Lasso and then learning the weights wg
via least squares. In the case of DNNs, an explanation generated by this algorithm called LIME allows the user to just
highlight the super-pixels with positive weights towards a
specific class (see Fig 5). In what follows we set σ = 0.25,
n = 1000 and and keep K constant. In the following we
refer to an image prompted solely with its top super-pixels
as an image with attention.

Feature-based Explanation. Machine learning and especially Deep Neural Networks (DNNs) lie at the core
of more and more technological advances across various
fields. However, those models are still widely considered
as black boxes, leading end users to mistrust the predictions or even the underlying models. In order to promote
the adoption of such algorithms and to foster their positive
technological impact, recent studies have been focusing on
understanding a model’s behavior from a human point of
view [30, 39, 31, 10].
In particular, [36] propose to explain the predictions of
any classifier gθ (denoted as g) by approximating it locally
with an interpretable model h. The role of h is to provide
qualitative understanding between the input x and the classifier’s output g(x) for a given class. In the case where the
input x is an image, h will act on vector x0 ∈ {0, 1}d denoting the presence or absence of the d super-pixels that
partition the image x to explain the classifier’s decision.
Finding the best explanation ξ(x) among the candidates
h can be formulated as:
ξ(x) = argmin L(g, h, πx ) + Ω(h)

(5)

z,z 0 ∈Z

where Jb (S) can be expressed as follows:
X
1 X
2
k(xi , xj ) −
k(xi , xj )
Jb (S) =
n|S|
|S|2
i∈[n],j∈S

kx−zk2
σ2

3. Experiments
3.1. Human Subject Study of Classification Errors
We conduct a study of the misclassifications of various pre-trained architectures (e.g. ResNet-18, ResNet101, DenseNet-121, DenseNet-161) with human subjects
on Amazon Mechanical Turk (AMT). To this end, we use
the center-cropped images of size 224 × 224 from the standard validation set of Imagenet. The same setting holds in
all our experiments. For every architecture, we consider
all the examples misclassified by the model. Each of these
examples is presented to five (5) different turkers together
with the class (C) predicted by the model. Each turker is
then asked the following question ”Is class C relevant for
this image.” The possible answers are yes and no. The former means that the turker agrees with the prediction of the
network, while the latter means that the example is misclassified. Figure 2 shows a breakdown of the misclassified images by a ResNet-101 (resp. a ResNet-18) according to the
number of positive answers they receive from the turkers.

(4)

h∈H

4323

Model
Resnet-18
Resnet-101
Densenet-121
Densenet-161

Top-1 error
31.39
22.69
25.53
22.85

Rectified Top-1 error
17.93
9.47
14.37
10.87

Table 1: Standard and rectified Top-1 errors for various
models. Rectified Top-1 errors are significantly lower.
Our first results show that for a ResNet-101 (resp. a
ResNet-18), for 39.76% (resp. 27.14%) of the misclassified
examples, all the turkers (5/5) agree with the prediction
of the model. If we further consider the prediction of the
model for a given image to be correct if at least four turkers
out of five (4/5) agree with it, the rectified Top-1 error of
the models are drastically reduced.
Table 1 shows the original Top-1 errors together with the
rectified versions. For the ResNet-101 and the DenseNet161, the rectified Top-1 error is respectively 9.47% and
10.87%. Similarly, the top-5 error of the ResNet-101 on
a single crop is 6.44. When we present the misclassified
images to the turkers together with the top 5 predictions of
the model, the rectified top-5 error drops to 1.94%. When
instead of submitting to the turker the misclassified images,
we present them the pictures for which the ground truth is
not in the top-5 predictions, the rectified top-5 error drops
further to 1.37%. Therefore, the observation on top-5 is in
line with the conclusions regarding top-1. If top-5 is the important measure, this experiment suggests that Imagenet is
(almost) solved as far as accuracy is concerned. Therefore
explaining the marked slowdown in performance improvement observed recently on Imagenet.
The difference between the ground truth labeling of the
images and the predictions of the models validated by the
turkers can be traced back to the collection protocol of Imagenet. Indeed to create the dataset, Deng et al. [9] first
queried several image search engines (using the synsets extracted from WordNet) to obtain good candidate images.
Turkers subsequently cleaned the collected images by validating that each one contains objects of a given synset. This
labeling procedure ignores the intrinsic multilabel nature of
the images, neither does it take into account important factors such as composionality. Indeed, it is natural that the
label wing is also relevant for a picture displaying a plane
and labelled as airliner. Figure 3 shows examples of
misclassifications by a RestNet-101 and a DenseNet-161.
In most cases, the predicted class is (also) present in the
image.

Figure 2: Positive answers for misclassified samples. An image
is prompted to 5 different subjects and a positive answer means
the subject agrees with the predicted class.

used the center-cropped images of the Imagenet validation
set. Next, we consider a subset of 20 classes and generate
an adversarial example from each legitimate image using
the IFGSM attack on a pre-trained ResNet-101. We used a
step size of  = 0.001 and a maximum number of iterations
of M = 10. This attack deteriorated the accuracy of the
network on the validation set from 77.31% down to 7.96%.
Note that by definition, we only need to generate adversarial
samples for the correctly predicted test images.
We consider two settings in this experiment. In the first
configuration, we present the turkers with the whole adversarial image together with the prediction of the network. We
then ask each turker if the predicted label is relevant to the
given picture. Again, the possible answers are yes and no.
Each image is shown to five (5) different turkers. In the second configuration, we show each turker the interpretation
image generated by LIME instead of the whole adversarial
image using the top 8 most important features. The rest of
the experimental setup is identical to the previous configuration where we showed the whole images to the turkers. If
a turker participated in the second experiment with the interpretation images after participating in the first experiment
where the whole images are displayed, his answers could be
biased. To avoid this issue, we perform the two studies with
three days intervals. Similarly to our previous experiments,
we report the rectified Top-1 error on adversarial examples
by considering the prediction of the model as correct if at
least 4/5 turkers agree with it.
Table 2 shows the standard and the rectified Top-1 errors for the adversarial examples. Two observations can be
made. First, the robustness of the models as measured by
the Top-1 error on adversarial samples generated from the
validation set of Imagenet is also underestimated. Indeed,
when the whole images are displayed to them, the turkers
agree with 22.01% of the predictions of the networks on adversarial examples. This suggests that often, the predicted

3.2. Study of Robustness to Adversarial Examples
We conducted a human subject study to investigate the
appreciation of adversarial perturbation by end-users. Similarly to the human subject study of misclassifications, we
4324

Figure 3: Some test samples misclassified by a ResNet-101 (first row) and a Densenet-161 (second row). The predicted class
is indicated in red, the ground truth in black and in parenthesis. All those examples gathered more than four (4 or 5) positive
answers over 5 on AMT. Note that no adversarial noise has been added to the images.
Setting
Whole image
Explanation

Top-1 error
92.04
92.04

Rectified Top-1 error
76.78
70.68

Table 2: Standard and rectified Top-1 errors on adversarial
examples for the whole images and the explanations.

We further inspected the adversarial images on which the
turkers agree with the model’s prediction when the explanation is shown but mostly disagree when the whole image is
shown. Figure 6 displays examples of such images. In most
cases, even though the predicted label does not seem correct
when looking at the whole image, the explanation has one
of the two following effects. It either reveals the relevant
object supporting the prediction (e.g., Tripod image) or
creates an ambiguous context that renders the predicted label plausible (e.g., Torch or Honeycomb images). In all
cases, providing explanations to the user mitigates the impact of misclassifications due to adversarial perturbations.

Figure 4: Positive answers for adversarial samples. The images are either displayed as a whole (entire image) or with
attention (explanation). Every image is prompted to 5 different subjects. A positive answer means the subject agrees
with the predicted adversarial class.

label is easily identifiable in the picture by a human. Figure 5 shows an example (labeled jeep) with the explanations of the predictions for the legitimate and adversarial
versions respectively. One can see that the adversarial perturbation exploits the ambiguity of the image by shifting
the attention of the model towards regions supporting the
adversarial prediction (e.g. the red cross).
The second substantial observation is that the percentage
of agreement between the predictions of the model and the
turkers increases from 22.01% to 30.80% when the explanation is shown instead of the whole image (see also Figure 4).

3.3. Adversarial Examples for Model Criticism
Example-based explanation methods such as model criticism aim at summarizing the statistics learned by a model
by using a carefully selected subset of examples. MMDcritic proposes an effective selection procedure combining prototypes and criticisms for improving interpretability.
Though applicable to a pre-trained neural network by using
the hidden representation of the examples, it only indirectly
exploits the discriminative nature of the classifier. In this
4325

Condition
Adv - Prototypes and Criticisms
MMD - Prototypes and Criticisms
Probs - Prototypes and Criticisms
Adv - Prototypes only
MMD - Prototypes only
Probs - Prototypes only
Random

Figure 5: Left: an adversarial image of the true class Jeep
predicted as Ambulance by the network. Center: the
explanation of the clean image for its prediction (Jeep).
Right: the explanation of the adversarial image for its prediction (Ambulance).

Mean (%)
57.06
50.64
49.36
55.02
53.18
52.44
49.80

Std (%)
3.22
3.25
3.26
3.23
3.24
3.25
3.25

Table 3: Results of the Prototype and Criticism study. We
report the average success rate of the task (assign a target
image to one of the six groups) for the seven conditions.
The adversarial selection procedures outperforms MMDcritic as well as the other baselines used in the study.

work, we argue that adversarial examples can be an accurate alternative to MMD-critic for model criticism. In particular, they offer a natural way of selecting prototypes and
criticism based on the number of steps of FGSM necessary
to change the decision of a classifier for a given example.
Indeed, for a given class and a fixed number of maximum
steps M of IFGSM, the examples that are still correctly
classified after M steps can be considered as useful prototypes since they are very representative of what the classifier has learned from the data for this class. In contrast,
the examples whose decision change after 1 or few steps of
FGSM are more likely to be valid criticisms because they
do not quite fit the model.
We conducted a human study to evaluate our hypothesis. At each round, we present a turker with six classes
represented by six images each, as well a target sample randomly drawn from one of those six classes. We measure
how well the subject can assign the target sample to the
class it belongs. The assignment task requires a class to be
well-explained by its six images. The challenge is, therefore, to select well those candidate images. The adversarial
method selects as prototypes the examples not misclassified
IFGSM after M = 10 steps, and as prototypes, the examples misclassified after one step. In addition to MMD-critic
(using λ = 10−5 ), we compare the adversarial approach to
a simple baseline using the probabilities for selecting the
prototypes (considerable confidence, e.g., > 0.9) and the
criticisms (little confidence, e.g., < 0.1). We also compare
with the baseline randomly sampling examples from a given
class. For each method (except the random baseline), we
experiment with showing six prototypes only vs. showing
three prototypes and three criticisms instead.
To properly exploit the results, we discarded answers
based on the time spent by the turkers on the question to
only consider answers given in the range from 20 seconds to
3 minutes. This resulted in 1, 600 valid answers for which
we report the results in Table 3. Two observations arise
from those results. First, the prototypes and criticisms sampled using the adversarial method represent better the class

distribution and achieve higher scores for the assignment
task both when only prototypes are used (57.06%) and when
prototypes are combined with criticisms (55.02%). Second,
the use of criticisms additionally to the prototypes always
helps to better grasp the class distribution.
More qualitatively, we display in Figures 7 and 8 some
prototypes and criticisms for the class Banana generated
using respectively the MMD and the adversarial method for
the test samples, that demonstrate the superiority of the adversarial method.

3.4. Uncovering Biases with Model Criticism
To uncover the undesirable biases learned by the model,
we use our adversarial example approach to model criticism
since it worked better than MMD-critic in our previous experiments. We consider the class basketball (for which
humans often appear in the images).We select and inspect a
reduced subset of prototypes and criticisms from the category basketball. The percentage of basketball
training images on which at least one white person appears
is about 55%. Similarly, the percentage of images on which
at least one black person appears is 53%. This relative balance contrasts with the statistics captured by the model. Indeed, on the one hand, we found that about 78% of the prototypes contain at least one black person and only 44% for
prototypes contain one white person or more. On the other
hand, for criticisms, 90% of the images contain at least one
white person and only about 20% include one black person or more. This suggests that the model has learned a
biased representation of the class basketball where images containing black persons are prototypical. To further
validate this hypothesis, we sample pairs of similar pictures
from the Internet. The pairs are sampled such that the primary apparent difference between the two images is the
skin color of the persons. We then fed images to the model
and gathered the predictions. Figure 9 shows the results of
the experiments. All images containing a black person are
classified as basketball while similar photos with persons of
4326

Figure 6: Adversarial samples, displayed as a whole (entire image, first row) or with explanation (only k top super-pixels,
second row). The adversarial class is indicated in red, the true class in black and between parenthesis. In red is also displayed
the percentage of positive answers for the displayed image (over 5 answers). In both cases, a positive answer means the
subject agrees with the predicted adversarial class.

Figure 8: Prototypes (1st row) and Criticisms (2nd row), for
the Banana class when using the adversarial selection.

Figure 7: Prototypes (first row) and Criticisms (second row)
for the Banana class when using the MMD-critic method.

further investigations on this to future studies.

different skin color are labeled differently. Figure 12 provides additional supporting evidence for this hypothesis. It
shows the progressive feature based explanation path uncovering the super-pixels of an example (correctly classified
as basketball) by their order of importance. The most
importance super-pixels depict the jersey and the skin color
of the player. The reasons why the model learns these biases are unclear. One hypothesis is that despite the balanced
distribution of races in pictures labeled basketball, blacks
persons are more represented in this class in comparison to
the other classes. A Similar phenomenon have previously
been noted in the context of textual data [33, 5]. We defer

Remark have focused on racial biases because a human
can easily spot them. In fact, we have found similar biases
where pictures displaying Asians dressed in red are very
often classified as ping-pong ball. However, we hypothesize the biases of the model are numerous and diverse.
For example, we also have found that the model often predicts the class traffic-light for images of a blue sky
with street lamps as depicted in Figure 11. In any case,
model criticism has proven effective in uncovering the undesirable hidden biases learned by the model.
4327

Figure 9: Pairs of pictures (columns) sampled over the Internet along with their prediction by a ResNet-101.

Figure 10: Prototypes (first row) and Criticisms (second row) for the test images of the Basketball class, adv method.

Figure 11: Images sampled from the Internet classified as
Traffic Light for a ResNet-101.

Figure 12: Explanation path of a ResNet-101 for a test sample of the Basketball class.

4328

4. Conclusion

[12] A. Fawzi, S.-M. Moosavi-Dezfooli, and P. Frossard. Robustness of classifiers: from adversarial to random noise. In
Advances in Neural Information Processing Systems, pages
1624–1632, 2016.
[13] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N.
Dauphin. Convolutional sequence to sequence learning.
arXiv preprint arXiv:1705.03122, 2017.
[14] A. Gelman. Bayesian data analysis using r. 2006.
[15] R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448,
2015.
[16] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
arXiv:1412.6572, 2014.
[17] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and
harnessing adversarial examples. In Proc. ICLR, 2015.
[18] A. Gretton, K. M. Borgwardt, M. J. Rasch, and y. Bernhard Schölkopf and Alexander J. Smola, booktitle=NIPS. A
kernel method for the two-sample-problem.
[19] C. Guo, M. Rana, M. Cisse, and L. van der Maaten. Countering adversarial images using input transformations. arXiv
preprint arXiv:1711.00117, 2017.
[20] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn.
arXiv preprint arXiv:1703.06870, 2017.
[21] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
770–778, 2016.
[22] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten.
Densely connected convolutional networks. arXiv preprint
arXiv:1608.06993, 2016.
[23] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, and
B. Schiele. Deepercut: A deeper, stronger, and faster multiperson pose estimation model. In European Conference on
Computer Vision, pages 34–50. Springer, 2016.
[24] B. Kim, R. Khanna, and O. O. Koyejo. Examples are not
enough, learn to criticize! criticism for interpretability. In
Advances in Neural Information Processing Systems, pages
2280–2288, 2016.
[25] B. Kim, C. Rudin, and J. A. Shah. The bayesian case model:
A generative approach for case-based reasoning and prototype classification. In NIPS, 2014.
[26] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[27] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
clas sification with deep convolutional neural networks. In
Advances in neural information processing systems, pages
1097–1105, 2012.
[28] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533,
2016.
[29] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
[30] Z. C. Lipton. The mythos of model interpretability. arXiv
preprint arXiv:1606.03490, 2016.

Through human studies and explanations, we have
proven that the performance of SOTA models on Imagenet
is underestimated. This leaves little room for improvement
and calls for new large-scale benchmarks involving for example multi-label annotations. We have also improved our
understanding of adversarial examples from the perspective
of the end-user and positioned model criticism as a valuable tool for uncovering undesirable biases. These result
open an exciting perspective on designing explanations and
automating bias detection in vision models. Our study suggests that more research in these topics will be necessary
to sustain the use of machine learning as a general purpose
technology and to achieve new breakthroughs in image classification.

References
[1] A. Aamodt and E. Plaza. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI Commun., 7:39–59, 1994.
[2] K. Andrej. What i learned from competing against a convnet
on imagenet. http://karpathy.github.io/2014/
09/02/what-i-learned-from-competingagainst-a-convnet-on-imagenet/, 2014.
[3] A. Badanidiyuru, B. Mirzasoleiman, A. Karbasi, and
A. Krause. Streaming submodular maximization: massive
data summarization on the fly. In KDD, 2014.
[4] I. Bichindaritz and C. R. Marling. Case-based reasoning in
the health sciences: What’s next? Artificial intelligence in
medicine, 36 2:127–35, 2006.
[5] T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and
A. T. Kalai. Man is to computer programmer as woman is
to homemaker? debiasing word embeddings. In Advances in
Neural Information Processing Systems, pages 4349–4357,
2016.
[6] A. Bulat and G. Tzimiropoulos. Human pose estimation via
convolutional part heatmap regression. In European Conference on Computer Vision, pages 717–732. Springer, 2016.
[7] M. Cisse, Y. Adi, N. Neverova, and J. Keshet. Houdini:
Fooling deep structured prediction models. arXiv preprint
arXiv:1707.05373, 2017.
[8] M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and
N. Usunier. Parseval networks: Improving robustness to adversarial examples. arXiv preprint arXiv:1704.08847, 2017.
[9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei. Imagenet: A large-scale hierarchical image database.
In Computer Vision and Pattern Recognition, 2009. CVPR
2009. IEEE Conference on, pages 248–255. IEEE, 2009.
[10] Y. Dong, H. Su, J. Zhu, and F. Bao. Towards interpretable
deep neural networks by leveraging adversarial examples.
arXiv preprint arXiv:1708.05493, 2017.
[11] A. Fawzi, O. Fawzi, and P. Frossard. Analysis of classifiers’ robustness to adversarial perturbations. arXiv preprint
arXiv:1502.02590, 2015.

4329

[31] G. Montavon, W. Samek, and K.-R. Müller. Methods for
interpreting and understanding deep neural networks. Digital
Signal Processing, 2017.
[32] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. Deepfool: a simple and accurate method to fool deep neural networks. arXiv preprint arXiv:1511.04599, 2015.
[33] D. Paperno, M. Marelli, K. Tentori, and M. Baroni. Corpusbased estimates of word association predict biases in judgment of word co-occurrence likelihood. Cognitive psychology, 74:66–83, 2014.
[34] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami.
Distillation as a defense to adversarial perturbations against
deep neural networks. In Security and Privacy (SP), 2016
IEEE Symposium on, pages 582–597. IEEE, 2016.
[35] F. Perronnin, J. Sánchez, and T. Mensink. Improving the
fisher kernel for large-scale image classification. Computer
Vision–ECCV 2010, pages 143–156, 2010.
[36] M. T. Ribeiro, S. Singh, and C. Guestrin. ”why should I trust
you?”: Explaining the predictions of any classifier. CoRR,
abs/1602.04938, 2016.
[37] S. Ritter, D. G. Barrett, A. Santoro, and M. M. Botvinick.
Cognitive psychology for deep neural networks: A shape
bias case study. arXiv preprint arXiv:1706.08606, 2017.
[38] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge.

[39]

[40]

[41]
[42]

[43]
[44]

[45]

[46]

4330

International Journal of Computer Vision, 115(3):211–252,
2015.
W. Samek, T. Wiegand, and K.-R. Müller. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296,
2017.
U. Shaham, Y. Yamada, and S. Negahban. Understanding adversarial training: Increasing local stability of neural nets through robust optimization.
arXiv preprint
arXiv:1511.05432, 2015.
H. A. Simon and A. Newell. Human problem solving : the
state of the theory in 1970. 1972.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan,
I. Goodfellow, and R. Fergus. Intriguing properties of neural
networks. arXiv preprint arXiv:1312.6199, 2013.
P. Tabacof and E. Valle. Exploring the space of adversarial
images. arXiv preprint arXiv:1510.05328, 2015.
F. Tramèr, A. Kurakin, N. Papernot, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses.
arXiv preprint arXiv:1705.07204, 2017.
Y. Wang, X. Deng, S. Pu, and Z. Huang. Residual convolutional ctc networks for automatic speech recognition. arXiv
preprint arXiv:1702.07793, 2017.
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz.
mixup: Beyond empirical risk minimization. arXiv preprint
arXiv:1710.09412, 2017.

