Can Deep Neural Networks Match the Related Objects?:
A Survey on ImageNet-trained Classification Models
Han S. Lee, Heechul Jung, Alex A. Agarwal, Junmo Kim

arXiv:1709.03806v1 [] 12 Sep 2017

School of Electrical Engineering
Korea Advanced Institute of Science and Technology
hansanglee@kaist.ac.kr

Abstract
Deep neural networks (DNNs) have shown the state-of-theart level of performances in wide range of complicated tasks.
In recent years, the studies have been actively conducted to
analyze the black box characteristics of DNNs and to grasp
the learning behaviours, tendency, and limitations of DNNs.
In this paper, we investigate the limitation of DNNs in image
classification task and verify it with the method inspired by
cognitive psychology. Through analyzing the failure cases of
ImageNet classification task, we hypothesize that the DNNs
do not sufficiently learn to associate related classes of objects.
To verify how DNNs understand the relatedness between object classes, we conducted experiments on the image database
provided in cognitive psychology. We applied the ImageNettrained DNNs to the database consisting of pairs of related
and unrelated object images to compare the feature similarities and determine whether the pairs match each other. In the
experiments, we observed that the DNNs show limited performance in determining relatedness between object classes. In
addition, the DNNs present somewhat improved performance
in discovering relatedness based on similarity, but they perform weaker in discovering relatedness based on association.
Through these experiments, a novel analysis of learning behaviour of DNNs is provided and the limitation which needs
to be overcome is suggested.

Introduction
In a recent decade, deep neural networks (DNNs) have
achieved remarkable performances in a wide range of complicated tasks in image recognition, speech processing,
health care, robotics, and gaming (LeCun, Bengio, and Hinton 2015). Especially, in image recognition, the DNNs have
shown outstanding performances for image classification
and object detection in ImageNet database (Russakovsky et
al. 2015), and have opened the era of deep learning in the
field of computer vision.
One of the interesting characteristics of DNNs is that it is
difficult to intuitively grasp the principle and internal structure of learning, which is so-called black box characteristic. This black box characteristic causes can cause a number
of problems in several tasks. For instance, when analyzing
medical images, it is very important not only to improve the
Copyright c 2018, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.

Figure 1: Examples of the concept of matching related objects. The task usually connects two objects, which have
similar meanings or interact closely with each other.

accuracy of the diagnosis, but also to provide a detailed explanation that the physician and the patient can understand
the basis of the diagnosis. In addition, when DNNs do not
achieve satisfactory performance for a given task, one cannot figure out which part of DNNs the limitation comes from
because the internal structure is not known. The fact that one
should heavily rely on fine-tuning parameters and models to
improve the performance of DNNs, can be also considered
as the limitation caused by the black box characteristic of
DNNs.
To address these limitations, various researches have been
currently suggested to discover the internal structure of the
DNN black boxes. Many of them have visualized the features learned in DNNs to provide intuition of learning behaviours of DNNs (Higgins et al. 2016; Karpathy, Johnson, and Li 2015; Li et al. 2015; Raposo et al. 2017;
Yosinski et al. 2015; Zeiler and Fergus 2013). In addition,
the studies on the weaknesses of DNN learning have been investigated through the degradation of its performance in certain images (Nguyen, Yosinski, and Clune 2014; Hosseini
and Poovendran 2017). By reversing the information about
the internal structure of DNNs obtained by the above analy-

Table 1: Examples of the ImageNet classification failure cases in which the predicted classes and the ground truth class are similar. It is difficult to conclude that the CNN performs incorrectly because these images are classified as the classes semantically
similar to the ground truths. However, the rankings of the ground truths in the predicted probabilities are relatively low despite
the semantic similarities between the classes. This can be understood as evidence that the current CNN training does not fully
take into account the semantic similarities between classes.

Images

Top-5 predicted classes
(Predicted probabilities)
Ground truth classes
(Predicted probabilities,
ranks in the predicted
classes)

desk (23.5%)
file (13.1%)
desktop computer (9.8%)
library (7.5%)
monitor (3.1%)

palace (65.9%)
triumphal arch (5.8%)
fountain (4.4%)
pedestal (3.4%)
monastery (3.3%)

tow truck (9.5%)
garbage truck (9.2%)
tank (7.9%)
jeep (7.5%)
trailer truck (3.6%)

notebook (1.5%, 11th )

castle (0.19%, 18th )

moving van (0.85%, 22nd )

ses, the recent studies have even generated the images which
deteriorate the performance of DNNs (Szegedy et al. 2013;
Goodfellow, Shlens, and Szegedy 2014; Kurakin, Goodfellow, and Bengio 2016; Papernot et al. 2016; Behzadan and
Munir 2017; Tramèr et al. 2017; Huang et al. 2017).
In this paper, we investigated the weakness of DNN learning through a cognitive psychology method. A recent study
visually evaluated the failure cases of the ImageNet classification results to analyze how the DNNs fail to recognize
certain images (Lee, Agarwal, and Kim 2017). In addition,
one recent research studied the shape bias in DNN learning through cognitive psychology-inspired approaches (Ritter et al. 2017). Inspired by both of these studies, we conducted experiments in which we constructed the hypothesis
about the weakness of DNN learning from observing the
ImageNet failure cases, and verified it with the cognitive
psychology approach. We hypothesized that the DNNs are
successful in learning by distinguishing objects of different
classes in highly sophisticated units, but fail in relating the
objects of similar classes with each other. To verify our hypothesis, we used DNNs on the database including the pairs
of related/unrelated object images provided from the cognitive psychology, to evaluate and analyze how well DNNs
determine the relatedness between these object images.
Our results are summarized as follows. First, the DNNs
are not successful in learning the relatedness of similar
classes in general. Second, the performance of relatedness
matching is generally proportional to the ImageNet classification performance. Third, there are two types of relatedness
between object images: One is relatedness based on similarity in which two objects share similar characteristics, and the
other is relatedness based on association in which two objects usually interact each other. Fourth, it is analyzed that

the DNNs can grasp the relatedness based on similarity but
not the relatedness based on association. Through these experiments, we not only investigated the limitations of DNN
learning, but also obtained the insight on the principle of relatedness between objects.

Motivation from ImageNet Failure Cases
We first hypothesized the weakness of DNN learning by
observing the failure cases of ImageNet classification. Lee
et al. applied the ensemble model of ResNet-based convolutional neural networks (CNNs) to train ImageNet and
performed visual inspection on 400 failure images sampled
from the validation set (Lee, Agarwal, and Kim 2017). They
then categorized the images into five categories, and one of
them is so-called similar class group, which accounts for
about 15% of the failure cases. In the images of this similar
class group, the top-1 predicted class and the ground truth
class have similar or almost the same meaning. In fact, these
cases can not be regarded as a 100% failure because the
DNNs predict semantically correctly for these cases. Thus,
Lee et al. suggested to improve the evaluation method of ImageNet and put these failure cases into the successful cases.
However, it is noteworthy that the ground truth class is not
shown in the top-5 predicted classes, although the ground
truth and the top-1 predicted class are similar. Of course, in
many cases in the similar class group, the ground truth class
is ranked in the top 6th or top 7th in the predicted probability
list. However, there are the cases in which ground truth class
in ranked below the top 10th in the predicted probability list,
as shown in Table 1. In the middle image, the probability
of castle is measured to be 0.19% while the probability of
palace is 65.9%. It inevitably raises the question of how extreme DNNs treat castle and palace differently. According to

Figure 2: A framework for matching images of POPORO database with semantic relatedness using ImageNet-trained CNN
models.

(a) Objects

(b) Related

(c) Unrelated

Figure 3: Example images of POPORO database. For each
object image, one ‘related’ image and one ‘unrelated’ image are provided. The labeling of related/unrelated was performed based on the user ratings of semantic relatedness.

the common knowledge, it is thought that the classes of similar objects should have similar probabilities, but it seems
that the learning behaviour of DNNs shown in these ImageNet failure cases is not been significantly affected by our
intuition. Therefore, we hypothesized that the DNNs do not
sufficiently learn the relatedness between the similar object
classes, and conducted the experiments to verify it.

Experiments
To evaluate how DNNs grasp the relatedness between the
similar classes, we suggest the experiment summarized
in Fig. 2. As experimental data, we used the Pool Of
Pairs Of Related Objects (POPORO) database (Kovalenko,
Chaumon, and Busch 2012) from cognitive psychology researches. The POPORO database consists of 400 pairs of images and each pair consists of 3 images, as shown in Fig. 3.
The pair consists of an object image, an image of an object
which is ‘related’ to the object, and an image of an object

which is ‘unrelated’ to the object. The basic experiments
were performed by showing three image sequentially and
letting users to rate the relatedness with the object on two
non-object images. In our experiments, the similarities between the object image and related/unrelated images were
determined using DNNs, and the image with high similarity
was matched to the object, to evaluate the matching performance of DNNs.
Our experimental framework consists of three steps.
First, the features of three images are extracted using the
ImageNet-trained DNN models. Two state-of-the-art CNN
models, ResNet (He et al. 2015) and DenseNet (Huang, Liu,
and Weinberger 2016), were used as DNN models. In order to observe the tendency in performance according to
the internal parameter changes, six ResNet models and eight
DenseNet models with different number of layers and channels were used, as summarized in Table (?). We used open
source implementations for ResNet 1 and DenseNet 2 to extract the features from the last layer of DNNs. Second, the
similarities between the object feature and related/unrelated
features were calculated using cosine similarity (Ritter et al.
2017). Using these cosine similarities, finally, the non-object
image with higher similarity with the object was selected as
matched image. In the experiments, we evaluated the matching performances as an error rate for whether the matched
image determined by DNNs is the ‘related’ image.

Results
Fig. 4 shows the distribution of cosine similarities of DNN
features and the user ratings provided by the database (Kovalenko, Chaumon, and Busch 2012) for all pairs of (object,
related) and (object, unrelated). In user ratings, a significant
difference was observed between the related (red) and unrelated classes (blue). On the other hand, the cosine similarities are found to have no meaningful difference between the
related and unrelated classes. As a result, the ratio of the correct matches (o) to the incorrect matches (x) is almost 5:5.
1
2

https://github.com/facebook/fb.resnet.torch
https://github.com/liuzhuang13/DenseNet

Figure 4: A distribution of cosine similarities of CNN features for each pair of POPORO images. The x-axis represents the
average of the user ratings used in the relatedness labeling for the POPORO database. The points with the related/unrelated
ground truth labels are shown in red/blue, respectively. The points for which the matching results by similarity comparison
are correct/incorrect, are indicated by o/x, respectively. In the graph, we can see that the related and unrelated pairs have a
large difference between user ratings, but their cosine similarities are distributed in almost same range without a tendency. The
cosine similarities are calculated from the features extracted from the ResNet-200, which shows the best matching results, and
the cosine similarities computed with other CNN models show almost the same distributions.
In (a) and (d), the bag and the other bag, and the backpack
and the backrest, were matched well because the class and
the shape of the bags were clearly identifiable. However, if
the class information is not enough clear for object or nonobject and the two have large shape similarity such as (c), or
if the two classes share no image similarity and the relatedness should be defined based on the association property between two, DNN matching showed incorrect results. through
the cosine similarity distribution, it can be observed that the
DNNs does not show a meaningful behaviour in determining the relatedness between the two classes. Especially, as
shown in Fig. 5, the cosine similarity distribution shows the
same tendency for all the DNN models applied to the experiments, so all DNN models have a similarity distribution
similar to Fig. 4.
With the cosine similarities computed from 400 related
pairs and 400 unrelated pairs, we first predicted the relatedness for each data by changing the threshold for the cosine similarity. The receiver-operator characteristic (ROC)
curves and their area-under-curve (AUC) values are shown
in Fig. 5 and Table 2. As mentioned above, because the
ratio of the correct matches to the incorrect matches was
nearly 5:5, the ROC currves appeared almost diagonal. Inside the DNN models, the ResNet-200 which has the most
layers outperformed other ResNet models, and the most finetuned DenseNet-cosine-264 with k=48 outperformed other

DenseNet models. From all models, the ResNet-200 showed
the best matching performance with an AUC of 0.611.
Table 2 shows the matching error rates of the matching
results obtained comparing the cosine similarities of the related pair and unrelated par for each data, according to the
suggested experiments. These error rates also vary slightly
within the DNN models at approximately 40%, and the tendency is similar to the tendency of ROC curves and AUC
values in the above threshold-based prediction. Interestingly,
the tendency of the matching performance is similar to that
of the top-1 error performance from ImageNet classification.
This is shown more clearly in Fig. 6, which shows the error
rates in Table 2 as a bar graph. In Fig. 6, it can be observed
that the classification error, matching error, and AUC error
increase and decrease with the almost same tendency according to the DNN models. However, there is a little difference that the ResNet tends to outperform the DenseNet models in matching performance for the same level of ImageNet
performance. As a result, the ResNet-200 model achieved
the best matching performance with matching error rate of
38.75%.
Figs. 7 & 8 illustrate the example images from the correct
matches and incorrect matches by ResNet-200 model, respectively, where the difference between the two cosine similarities is large. In the ‘best’ cases, the similarity with the
related is much larger than that with the unrelated, where it

(a) Cosine similarities

(b) ROC curve (ResNet models)

(c) ROC curve (DenseNet models)

Figure 5: (a) A distribution of cosine similarities between different CNN models. The related/unrelated data are shown in
red/blue, respectively. The cosine similarities between the models are shown to be almost in linear relationship, so that the
models can be assumed to share similar cosine similarity distributions. (b)(c) ROC curves for threshold-based matching results
with cosine similarities of each pair. It can be seen that every curve shapes are close to diagonal, but ResNet-200 from ResNet
models and DenseNet-cosine-264 with k=264 from DenseNet models show the best curves among them.
Table 2: Numerical evaluation of the matching results for different CNN models. Inside the ResNet models or inside the
DenseNet models, the matching performance usually increases as the ImageNet performance increases. Between the different
models, for the same ImageNet performance, the ResNet models showed the better matching performance than DenseNet
models. As a result, the ResNet-200 showed the best matching performance with the matching error rate.
Models
ResNet-18
ResNet-34
ResNet-50
ResNet-101
ResNet-152
ResNet-200
DenseNet-121 (k=32)
DenseNet-169 (k=32)
DenseNet-201 (k=32)
DenseNet-161 (k=48)
DenseNet-264 (k=32)
DenseNet-232 (k=48)
DenseNet-cosine-264 (k=32)
DenseNet-cosine-264 (k=48)

ImageNet top-1 error
rates (%)
30.43
26.73
24.01
22.44
22.16
21.66
25
23.6
22.5
22.2
22.1
21.2
21.6
20.4

POPORO matching
error rates (%)
46
42.75
41.5
40.75
40.25
38.75
43.75
44.25
44.25
41.5
44
41
44.5
39.75

POPORO cosine
similarity AUC
0.528
0.551
0.583
0.584
0.585
0.611
0.562
0.565
0.572
0.57
0.556
0.597
0.574
0.594

can be seen that the object and the related have similar class
as well as similar shape. On the other hand, in the ‘worst’
cases, the similarity with the unrelated is much larger than
that with the related, where it can be observed that the object
and the related have low class similarity while the unrelated
has a very similar shape with the object, so the DNNs match
them incorrectly.

incorrectly matched, the cases show similar trend with the
‘good luck’ cases, but the object class is ambiguous or probably not identified in the ImageNet database, so the DNNs
consider the shape similarity only and match them incorrectly.

Figs. 9 & 10 show the example images from the correct
matches and incorrect matches by ResNet-200 model, respectively, where the difference between the two cosine similarities is small. In the ‘good luck’ cases where the related
was fortunately correctly matched, the unrelated has more
similar shape with the object, but the related also has a similar class with the object so the DNNs match them correctly.
In the ‘bad luck’ cases where the related was unfortunately

Our proposed experiments and their analyses provide a number of unique insights about the weakness of DNN learning.
First, the DNNs are not fully aware of the relatedness between similar classes. This is due to the fact that the most
of the image recognition tasks such as ImageNet are set up
to classify and distinguish the given classes independently
without considering their semantic similarity, rather than the
limitation of the DNN model itself. Thus, it is not unnatural

Discussion

Figure 6: A bar graph for different CNN model performances shown in Table 2. To observe the tendency of errors, the AUC of
the matching results is converted to the AUC error as 1–AUC to be plotted in the graph. It can be observed that the matching
performances show a significant correlation with the ImageNet performances.

(a) Objects

(b) Related

(c) Unrelated

Figure 7: The ‘best’ cases of the correct matching results in
which the difference of cosine similarity between the related
and unrelated is large. The cases where the object and the related object are visually highly similar are mainly included.

to assume that the feature distance between the two semantically similar classes can be large in the feature space, even
though the classes are highly semantically similar, i.e. the
palace and the castle. We however provided relatively realistic information about to what level the relatedness can be
recognized by the current DNNs through cognitive psychology approaches.

(a) Objects

(b) Related

(c) Unrelated

Figure 8: The ‘worst’ cases of the incorrect matching results
in which the difference of cosine similarity between the related and unrelated is large in negative direction. The cases
where the object and the related object are semantically related rather than visually, and the unrelated object is more
visually similar to the object than the related one, are mainly
included.

Second, the performance of relatedness matching is generally proportional to the ImageNet classification performance. This interesting observation suggests that the DNNs,
which currently are trained on the semantically similar

(a) Objects

(b) Related

(c) Unrelated

Figure 9: The ‘good luck’ cases of the correct matching results where the difference of cosine similarity between the
related and unrelated is small. The cases where the object
and the unrelated object share large visual similarity, but the
semantic similarity between the object and the related one is
also large, are mainly included.

classes independently, is somewhat learning some degree of
relatedness between the classes. We can also conversely argue that the learning of relatedness between the classes can
be helpful for the learning of existing class-independent image recognition tasks such as ImageNet classification.
Third, there are two types of relatedness between the
classes: One is relatedness based on similarity in which two
classes share similar characteristics in shape and hierarchy,
and the other is relatedness based on association in which
two classes usually interact each other. Consider a monkey,
a panda bear, and a banana. If one is told to match the related two, some will match the monkey and the panda bear,
because the monkey and the panda bear are both animals
and like to hang on trees. However, some may match the
monkey and the banana, because the monkey loves the banana (although this is not a fact). The former matching is
determining the relatedness based on similarity with similar hierarchy or shape between two objects or classes. The
latter matching is then determining the relatedness based on
association with interaction between two different objects or
classes.
Finally, it was analyzed that the DNNs can grasp the relatedness based on similarity well, but hardly captures the
relatedness based on association. This can be confirmed by
observing the ‘best’ and ‘worst’ cases in Figs. 6 & 7. In the
‘best’ cases, the DNNs correctly matched all the related ones
which show relatedness based on similarity. In the ‘worst’
cases, the DNNs failed to match the related ones which
show relatedness based on association. This also can explain

(a) Objects

(b) Related

(c) Unrelated

Figure 10: The ‘bad luck’ cases of the incorrect matching
results where the difference of cosine similarity between the
related and unrelated is small. The cases are similar to the
‘good luck’ cases, but it is analyzed that if the class of the
object is not easily determined, CNN tends to match the object with the unrelated object according to their visual similarity.

the reasons why DNNs showing better performance in ImageNet also show better matching performance. If the DNN
does not capture the proper relatedness based on similarity
from the semantically similar classes, this causes the degradation in ImageNet classification performance as shown in
our ImageNet failure cases. Thus, it can be analyzed that
as the DNNs have been improved to achieve better ImageNet performance, they unintentionally learn the relatedness based on similarity between the semantically similar
classes. Therefore, it is expected that it will be a direction to
improve the performance of DNNs in existing image recognition tasks, as well as the next step of cognitive machine
learning, to enable learning of relatedness based on association.

Conclusion
In this paper, we investigated the weakness of DNN learning,
by hypothesizing the lack of learning of relatedness through
the ImageNet failure cases, and verifying it by using the cognitive psychology database. Our analysis provided not only
performance of DNNs in determining relatedness, but also
the insight of how the current DNNs recognize the concept
of relatedness and how to enhance it. To improve the DNN
learning that can determine relatedness based on association
remains as future works.

References
[Behzadan and Munir 2017] Behzadan, V., and Munir, A.
2017. Vulnerability of deep reinforcement learning to policy
induction attacks. CoRR abs/1701.04143.
[Goodfellow, Shlens, and Szegedy 2014] Goodfellow, I. J.;
Shlens, J.; and Szegedy, C. 2014. Explaining and Harnessing Adversarial Examples. ArXiv e-prints.
[He et al. 2015] He, K.; Zhang, X.; Ren, S.; and Sun, J.
2015. Deep residual learning for image recognition. CoRR
abs/1512.03385.
[Higgins et al. 2016] Higgins, I.; Matthey, L.; Glorot, X.;
Pal, A.; Uria, B.; Blundell, C.; Mohamed, S.; and Lerchner,
A. 2016. Early Visual Concept Learning with Unsupervised
Deep Learning. ArXiv e-prints.
[Hosseini and Poovendran 2017] Hosseini, H., and Poovendran, R. 2017. Deep neural networks do not recognize negative images. CoRR abs/1703.06857.
[Huang et al. 2017] Huang, S. H.; Papernot, N.; Goodfellow,
I. J.; Duan, Y.; and Abbeel, P. 2017. Adversarial attacks on
neural network policies. CoRR abs/1702.02284.
[Huang, Liu, and Weinberger 2016] Huang, G.; Liu, Z.; and
Weinberger, K. Q. 2016. Densely connected convolutional
networks. CoRR abs/1608.06993.
[Karpathy, Johnson, and Li 2015] Karpathy, A.; Johnson, J.;
and Li, F. 2015. Visualizing and understanding recurrent
networks. CoRR abs/1506.02078.
[Kovalenko, Chaumon, and Busch 2012] Kovalenko, L. Y.;
Chaumon, M.; and Busch, N. A. 2012. A pool of pairs
of related objects (poporo) for investigating visual semantic
integration: Behavioral and electrophysiological validation.
Brain Topography 25(3):272–284.
[Kurakin, Goodfellow, and Bengio 2016] Kurakin,
A.;
Goodfellow, I. J.; and Bengio, S. 2016. Adversarial
examples in the physical world. CoRR abs/1607.02533.
[LeCun, Bengio, and Hinton 2015] LeCun, Y.; Bengio, Y.;
and Hinton, G.
2015.
Deep learning.
Nature
521(7553):436–444. Insight.
[Lee, Agarwal, and Kim 2017] Lee, H. S.; Agarwal, A. A.;
and Kim, J. 2017. Why do deep neural networks still not
recognize these images?: A qualitative analysis on failure
cases of imagenet classification. CoRR abs/1709.03439.
[Li et al. 2015] Li, J.; Chen, X.; Hovy, E. H.; and Jurafsky,
D. 2015. Visualizing and understanding neural models in
NLP. CoRR abs/1506.01066.
[Nguyen, Yosinski, and Clune 2014] Nguyen, A. M.; Yosinski, J.; and Clune, J. 2014. Deep neural networks are easily
fooled: High confidence predictions for unrecognizable images. CoRR abs/1412.1897.
[Papernot et al. 2016] Papernot, N.; McDaniel, P. D.; Goodfellow, I. J.; Jha, S.; Celik, Z. B.; and Swami, A. 2016. Practical black-box attacks against deep learning systems using
adversarial examples. CoRR abs/1602.02697.
[Raposo et al. 2017] Raposo, D.; Santoro, A.; Barrett, D.
G. T.; Pascanu, R.; Lillicrap, T. P.; and Battaglia, P. W. 2017.

Discovering objects and their relations from entangled scene
representations. CoRR abs/1702.05068.
[Ritter et al. 2017] Ritter, S.; Barrett, D. G. T.; Santoro, A.;
and Botvinick, M. M. 2017. Cognitive Psychology for Deep
Neural Networks: A Shape Bias Case Study. ArXiv e-prints.
[Russakovsky et al. 2015] Russakovsky, O.; Deng, J.; Su, H.;
Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.;
Khosla, A.; Bernstein, M.; Berg, A. C.; and Fei-Fei, L. 2015.
ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115(3):211–
252.
[Szegedy et al. 2013] Szegedy, C.; Zaremba, W.; Sutskever,
I.; Bruna, J.; Erhan, D.; Goodfellow, I. J.; and Fergus, R.
2013. Intriguing properties of neural networks. CoRR
abs/1312.6199.
[Tramèr et al. 2017] Tramèr, F.; Kurakin, A.; Papernot, N.;
Boneh, D.; and McDaniel, P. 2017. Ensemble Adversarial
Training: Attacks and Defenses. ArXiv e-prints.
[Yosinski et al. 2015] Yosinski, J.; Clune, J.; Nguyen, A. M.;
Fuchs, T. J.; and Lipson, H. 2015. Understanding neural networks through deep visualization. CoRR abs/1506.06579.
[Zeiler and Fergus 2013] Zeiler, M. D., and Fergus, R. 2013.
Visualizing and understanding convolutional networks.
CoRR abs/1311.2901.

