arXiv:1711.08502v1 [] 22 Nov 2017

Train, Diagnose and Fix: Interpretable Approach for Fine-grained Action
Recognition
Jingxuan Hou∗
Tsinghua University
Beijing, China

Tae Soo Kim∗
Johns Hopkins University
Baltimore, MD

Austin Reiter
Johns Hopkins University
Baltimore, MD

houjx14@mails.tsinghua.edu.cn

tkim60@jhu.edu

areiter@cs.jhu.edu

Abstract
Despite the growing discriminative capabilities of modern deep learning methods for recognition tasks, the inner workings of the state-of-art models still remain mostly
black-boxes. In this paper, we propose a systematic interpretation of model parameters and hidden representations
of Residual Temporal Convolutional Networks (Res-TCN)
for action recognition in time-series data. We also propose a Feature Map Decoder as part of the interpretation
analysis, which outputs a representation of model’s hidden
variables in the same domain as the input. Such analysis
empowers us to expose model’s characteristic learning patterns in an interpretable way. For example, through the diagnosis analysis, we discovered that our model has learned
to achieve view-point invariance by implicitly learning to
perform rotational normalization of the input to a more discriminative view. Based on the findings from the model
interpretation analysis, we propose a targeted refinement
technique, which can generalize to various other recognition models. The proposed work introduces a three-stage
paradigm for model learning: training, interpretable diagnosis and targeted refinement. We validate our approach
on skeleton based 3D human action recognition benchmark of NTU RGB+D. We show that the proposed workflow is an effective model learning strategy and the resulting Multi-stream Residual Temporal Convolutional Network (MS-Res-TCN) achieves the state-of-the-art performance on NTU RGB+D.

Figure 1. The ’Train, Diagnose and Fix’ paradigm provides means
of systematic model interpretation which enables targeted model
refinement strategies for learning stronger recognition models. All
figures are best viewed in color.

ment cycle for model training mostly reduces down to ’trialand-error’ [30]. In this light, we propose a more systematic and closed-loop approach to learning models through
model diagnosis and targeted refinement. We wish to take a
step towards a ’train-diagnose-and-fix’ paradigm and move
away from an exhaustive ’train-and-error’ routine. As Figure 1 suggests, interpretable model diagnosis enables us to
form better understanding of the problem of interest, reach a
deeper understanding of how the model is learning, identify
model’s weaknesses and ultimately form better hypotheses
to train stronger models. In this work, we propose two key
components of the ’train-diagnose-fix’ framework: the diagnosis tool and an approach to fix a model via targeted
model refinement.
In human action recognition from skeleton data in particular, despite the significant discriminative power of LSTMbased Recurrent Neural Networks architectures [8, 25, 33,
20, 27, 9, 21], the growing complexity of recent models hinders our ability to intuitively interpret them. A recently proposed Res-TCN [15] is specifically designed to improve the
interpretability of model parameters while efficiently learning a discriminative spatio-temporal representation for human action recognition with a convolutional neural network

1. Introduction
Despite the remarkable recognition power of recent deep
learning methods, our ability to clearly explain the inner
workings of deep networks has been unsatisfying [19, 24,
5, 13]. Fundamentally, without the potential to fully understand and interpret deep learning models, the develop∗ Both

authors contributed equally to this work.

1

(CNN) based approach.
However, the work of [15] limits its narrative on the interpretation of the model parameters and expand only a cursory exploration to hidden representations. Moreover, the
adopted visualization technique is relatively primary, which
lends no sufficient support to the proposed interpretation
and leaves it mostly intuitive. With the presented visualization results, little extra information can be further extracted
for diagnosis and targeted refinement. In this light, we propose a more systematic approach for model interpretation
and the means for model refinement given the findings from
the diagnosis. The main contribution of this work is three
fold:
• A model diagnosis tool, the Feature Map Decoder, is proposed for systematic interpretation of model parameters
and hidden representations. Feature Map Decoder visualizes the hidden representations in the same domain as
the input. The proposed Feature Map Decoder effectively
exposes the learning patterns and weaknesses of a trained
model. In our work, the diagnosis uncovered that the
model learned to implicitly transform the input sequence
into a more discriminative view point over the layers of
the network and the characteristic spatio-temporal portions of the input is gradually accentuated throughout the
layers.
• A targeted refinement technique is proposed to encode
cognitive-driven prior knowledge into a data-driven deep
learning model. A Targeted Attention (TA) stream is introduced as part of the refinement technique to force the
model to learn additional representation over the specific
dimensions of the input. Such set of dimensions where
the model struggles the most is identified by the interpretable diagnosis. A Pipe architecture is introduced to
effectively fuse the information learned in the TA stream
to the original model.
• Following the ’train-diagnose-and-fix’ pipeline, the resulting Multi-Stream Residual Temporal Convolutional
Network (MS-Res-TCN) achieves the state-of-the-art
performance on NTU-RGBD dataset [25].

2. Related Work
In this section, we first provide a brief review on relevant
3D skeleton based human action recognition approaches.
We then extend our survey to current literature on interpretation of modern deep neural networks.

2.1. 3D Skeleton Based Human Action Recognition
Given the time-series nature of the data, models that employ LSTM-based Recurrent Neural Networks have become
an active architecture of the research. HBRNN [8] divides
the whole skeleton into five parts. The parts are first fed separately into five bidirectional recurrent neural subnets, and
then fused hierarchically as the layers go deeper. In Part-

Aware LSTM model [25], part-based cells are introduced to
keep the context of each of the five body parts independent.
The final classifier is learned over these independent sub
predictions. In the work of [33], rather than pre-defining the
joint groups as in [8, 25], a co-occurrence exploration process is applied to achieve flexible automatic co-occurrence
mining. The work of [20] extends the analysis to the spatial domain to substitute the concatenation of the skeleton
joint information as in previous methods [8, 25, 33]. In this
fashion, a spatio-temporal LSTM model with trust gates is
introduced to concurrently learn both the spatial structure
of the joints and the temporal dependencies among frames.
Similarly, in STA-LSTM [27], a spatial attention module
is designed to automatically select dominant joints within
each frame, and a temporal attention module is designed to
assign different degrees of importance among the frames.
In action segmentation and detection, a new class of
temporal models, Temporal Convolutional Network (TCN)
[17], is proposed to capture long-range temporal dependencies, which outperforms other LSTM-based Recurrent Networks in both accuracy and training time. The work of
[15] employs a re-designed TCN model (Res-TCN) with
residual connections [9, 10] to skeleton based human action
recognition task. The proposed Res-TCN also learns both
spatial and temporal attention for human action recognition.
The authors of [15] argue that Res-TCN is specifically designed to enhance the interpretability of model parameters
and features compared to other recurrent models. However,
the work of [15] remains mostly intuitive and unsystematic
as opposed to our work.

2.2. On Neural Network Interpretability
Recently, there has been an increase in efforts to interpret
deep neural networks [4, 22, 30, 19, 24, 23, 26, 16, 31, 32,
14, 7]. The behavior of a complex black-box deep neural
network can be understood by generating a locally faithful
interpretable model around the test point and observing the
output [24]. The authors of [18, 2] perturb the test point
and observe how the prediction of the model changes. Additionally, [30, 23, 26, 31] attempt to visualize the synthesized inputs to the network that maximize the activations of
hidden units.
Another important class of approaches attempts to understand a CNN-based deep learning model by looking directly into the hidden representations with well-designed visualization techniques, which is most relevant to our current
work. In the work of [4], the proposed Network Dissection
method performs a binary segmentation on every feature
map with respect to every visual concept in Broden Dataset
[4]. The intersection-over-union (IoU) score for every segmentation task is computed. A visual concept is matched to
every neuron in the network by ranking the IoU scores. The
work of [22] proposes an optimization method to compute

an approximate inverse of the hidden representations. Note
that the motivation of the work of [22] is to reconstruct the
input image from the hidden representations as accurate as
possible. We argue that the proposed optimization method
has no interpretable basis as opposed to our Feature Map
Decoder which can be viewed as an inverse computation of
a single forward pass in the neural network.

3. Overview of Residual Temporal Convolutional Networks
In this section, we provide a brief overview of the Residual Temporal Convolutional Network [15] (Res-TCN) architecture (Figure 2). Res-TCN is an extension of TCN
[17] with residual connections with identity mappings as
proposed in [9, 10]. Res-TCN can naturally model a wide
range of time-series data. Let X0 ∈ RT ×D be the input that
represents a time-series data of temporal length T with D
dimensional feature per time step. Res-TCN hierarchically
stacks building blocks called Residual Units. Each unit in
layer l ≥ 2 performs the following computation:

Figure 2. Res-TCN model architecture. Except the first convolution layer (in gray), the model consists of stacked residual units.

Xl = Xl−1 + F (Wl , Xl−1 )

(1)

4. Interpretation of Residual Temporal Convolutional Networks

F (Wl , Xl−1 ) = Wl ∗ σ(Xl−1 )

(2)

In this section, we first propose an interpretation of
model parameters and hidden representations on an intuitive level. The insight gained from such analysis leads to
the formulation of our Feature Map Decoder. The decoded
skeleton sequences from the Feature Map Decoder expose
the characteristic learning patterns of Res-TCN. Finally, we
introduce a targeted refinement technique, which effectively
encodes the newly obtained knowledge gained by interpretation analysis into a deep learning model.
If all dimensions of the input have interpretable semantic
meaning associated with them, then we can directly understand every parameter of the first convolution layer [15].
For example, given a sequence of 3D human joints as input,
the first-layer filters can be directly interpreted as filters that
learn to detect snippets of joint motion. Each dimension
(i)
of a first-layer filter, W1 ∈ Rf1 ×D , has a corresponding
semantic joint associated with it, where f1 denotes the temporal filter length and D is the dimension of the input per
frame. We will refer the snippets of motion learned in the
first layer as motion primitives in the following narrative.
Filters belonging to deeper layers can also be interpreted.
Given the model architecture of Res-TCN, the convolution
filters in deeper layers act as gating operations. For example, the hidden representation after the first residual unit is:

F denotes the residual unit. For the l-th layer, Xl−1 denotes the input, Wl is the set of learnable parameters and σ
is batch normalization [12] followed by a ReLU activation
function. Note that the first convolution layer in Res-TCN
operates on raw time-series input and the resulting activation map, X1 , is passed on to subsequent layers without
going through a normalization or an activation layer. Given
a Res-TCN with N residual units, the hidden representation
after N residual units is:
XN = X1 +

N
X

Wi ∗ σ(Xi−1 )

(3)

i=2

X1 = W1 X0

(4)

The set of filters in W1 and the resulting activation map,
X1 , are readily interpretable given that each dimension of
X0 has a semantic meaning associated with it. For example,
a sequence of robot kinematics or 3D skeletons are such
input domains. This property of Res-TCN is the driving
motivation of the formulation of the proposed Feature Map
Decoder in Section 4.1.
For prediction, we apply global average pooling after the
last merge layer across the entire temporal sequence and attach a softmax layer with number of neurons equal to number of classes.

X2 = X1 + W2 ∗ σ(0, X1 )

(5)

where X1 is the output of the first convolution layer, W2
represents the set of filters in the second convolution layer.

Figure 3. Left: The first two filters visualize the temporal filters in the first layer. Middle: The two filters in the middle are examples of
deeper-layer filters. Right: First layer conv filters plotted directly as moving skeletons. This shows that such filters can be thought of as
motion primitives.

X1 represents to what extent and how the motion primitives in W1 are distributed in the input X0 . Following this
logic and observing Equation 5, we view W2 as a gate that
learns to find a discriminative weighted combination of motion primitives encoded in X1 . Given Equation 3, the same
argument can be made for all subsequent layers. Figure
3 visualizes the conv filters in the first and deeper layers.
The filters of the first layer show a continuous and smooth
changing motion-like structure for each dimension (corresponding to a semantic joint) across time, while filters of
deeper layers show a discrete and sparse gate-like structure. Note that the channel dimensionality of conv1 filters
are equivalent to the dimensionality of the input skeleton.
Then, to be more explicit, we provide visualization of the
first-layer filters as a sequence of moving skeletons (Figure
3), which in fact validates that these filters can be thought of
as motion primitives. We simply frame-wise map each dimension of a filter back to its corresponding semantic joint,
and then add back the mean skeleton which is subtracted in
the training process.
Let us now extend our analysis to hidden representations.
A convolution operation with a filter produces a high positive response over the regions of the input where the shape
of the filter is highly correlated with the input region. If the
input is a sequence of 3D skeletons, a filter in W1 (motion
primitive) produces a high positive convolution response in
temporal locations of X0 where X0 shows a similar motion
pattern defined in that particular filter. Let the activation
map of layer l be Xl ∈ RT ×Nl , where Nl denotes the number of filters in layer l and T represents the input length.
(i)
xl (t) can be interpreted as the i-th motion primitive’s activation response for the input sequence X0 at frame t.

4.1. Feature Map Decoder
As mentioned above, the first-layer filters learn a set
of motion primitives; the deeper-layer filters can be interpreted as gates, which hierarchically fuse the motion primitives learned in the first layer; the activation map of each

layer represents a temporal distribution of the learned motion primitives’ responses. Therefore, we argue that using
the set of motion primitives, the hidden representations of
the model can be converted back to a sequence of 3D skeletons performing some combination of these motion primitives. This leads to our proposed Feature Map Decoder
formulation.
Let X̂l ∈ RT ×D be the skeleton sequence decoded from
the activation map of the l-th layer, Xl ∈ RT ×Nl . We first
consider a simplified case when the number of filters is constant across all layers, i.e. Nl = Nc , ∀l ≥ 1. For each filter
(i)
(i)
1
of the first layer, W1 = {w1 (t)}ft=1
, where f1 denotes
the filter length of the first-layer filters, the compressed version of the filter is formulated as:
(i)

(i)

(i)

Ŵ1 = (w1 (f1 /2) + w1 (f1 /2 + 1))/2

(6)

(i)

where Ŵ1 ∈ R1×D . We are computing the mean of the
inner most two time steps of the filter following the observation that the most characteristic motion patterns appear
near the middle time steps of the temporal filters. In this
fashion, motion primitives of length f1 learned by the firstlayer filters are compressed into a single time step. Then,
the filter responses in each time step are used to compute
(i)
c
the weighted sum of {Ŵ1 }N
i=1 :
x̂l (t) =

Nc
X

(i)

(i)

Ŵ1 × xl (t)

(7)

i=1
(i)

where xl (t) is a i-th channel response at time step t of
Xl . x̂l (t) ∈ R1×D is the decoded skeleton at time step
t computed as a weighted combination of all compressed
motion primitives. We repeat this process for all t ∈ (0, T )
and temporal concatenation of all {x̂l (t)}Tt=1 yields a full
decoded sequence X̂l ∈ RT ×D . Note that in the training
stage, a mean skeleton has been subtracted from input samples. Hence, we add the mean skeleton back to the decoded
skeleton sequence to maintain mathematical consistency.

Figure 4. Visualization of decoded skeleton sequences from layers 1,4,7,10. The visualization shows that discriminative motion patterns
are gradually emphasized through the layers. Left: An example sequence of action class jump up. Right: An example sequence of action
class cheer up.

Figure 5. Analysis of the model through Feature Map Decoder reveals that the model has achieved view-point invariance. Left: An example
sequence of action class throw viewed from the side. Right: An example sequence of action class throw viewed from the front.

Now, let us extend our discussion to the circumstance
when the number of filters changes between layers. In the
Res-TCN architecture, layers in the same block share equal
number of filters. Let N1,2,3,4 = Nc1 , N5,6,7 = Nc2 ,
N8,9,10 = Nc3 . Note that when the number of filters
changes between the residual units (l = 5, 8), the identity
mapping connection requires an additional convolution for
channel shape matching purposes:
Xl = W̃l Xl−1 + F (Wl , Xl−1 ),

l = 5, 8

F (Wl , Xl−1 ) = Wl ∗ σ(Xl−1 )

trieval procedure is equivalent to steps listed in Equation
6 and 7 but the activations are first decoded using a set
(i)
of Ŵ5 ∈ R1×Nc1 compressed filters instead of a set of
(i)
Ŵ1 ∈ R1×D compressed filters. The retrieval is computed as follows:
0

x̂l (t) =

Nc2
X

(i)

(i)

(i)

[Ŵ5 × (xl (t) − x5 (t))

i=1

(8)

(i)

(i)

+ W̃5 × x5 (t)],

l = 5, 6, 7

(10)

(9)

where W̃l denotes the convolution layer attached to the
identity mapping connection.
For the first layer and the units in Block-A, no additional
decoding step is needed because N1 = Nc1 and the feature map decoding procedure follows the steps described
in Equations 6 and 7. For the layers in Block-B, one-step
retrieval needs to be performed to reshape the decoded se0
quence from Xl ∈ RT ×Nc2 to Xl ∈ RT ×Nc1 . The re-

(i)

Ŵ5 ∈ R1×Nc1 is the i-th compressed filter of the residual
(i)
unit of layer l = 5. W̃5 ∈ R1×Nc1 is the i-th conv filter of
(i)
(i)
temporal length 1 in layer l = 5. xl (t) ∈ R and x5 (t) ∈
R are the i-th filter’s activation response in layer l and the
0
fifth layer respectively, at time step t. x̂l (t) ∈ R1×Nc1 is
the retrieved activation map at time step t and the temporal
concatenation across all time steps yields the intermediate
0
result Xl ∈ RT ×Nc1 . Then, the steps listed in Equation

Figure 6. Examples of decoded skeleton sequences output from the Feature Map Decoder, which exposes Res-TCN’s failure to capture and
model detailed hand joints motion. Left: A case study of action class reading. Right: A case study of action class writing.
0

6 and 7 are repeated with Xl to produce the final decoded
sequence of Xl ∈ RT ×ND .
Similar operations are performed for hidden represen(i)
tations in Block-C. For such activations, a set of Ŵ8 ∈
1×Nc2
R
compressed filters first retrieve the sequence to activation shapes of Block-B. Then, the decoding procedure
follows the same steps as the activations in Block-B.
The Res-TCN down samples the temporal dimension by
a factor of 2 after each block. To simplify the narrative, we
have omitted the interpolation step required to up-sample
the temporal dimension of the retrieved decoded sequences.

4.2. Targeted Model Refinement
Through intuitive visualization, the proposed Feature
Map Decoder in Section 4.1 exposes the learning patterns
and weaknesses of the model. Our diagnosis findings will
be discussed in more depth in Section 5.2 but as a running example, we identified that our model struggles to detect very fine-grained motion patterns of the hands. Given
such finding, we introduce Targeted Attention (TA) stream
as an approach to explicitly address the weaknesses identified in the diagnosis step. The TA stream receives input,
0
X0 ∈ RT ×D , from a masked version of the original input where all dimensions are masked except the targeted
0
dimensions. For the example discussed above, X contains only the time-series information from the input dimensions corresponding to the hands and all other dimensions
are masked. The explicitly targeted representation of the
TA stream is fused with the original stream at every merge
layer (the Pipe) so that the representation learning of the
two streams is mutually interactive. A convolution layer
followed by a ReLU activation is employed on each Pipe to
control how much information can be transferred between
two streams. Figure 7 depicts the resulting architecture,
Multi-Stream Res-TCN (MS-Res-TCN).
Each unit in layer l ≥ 2 performs the following compu-

tation:
0

Xl = Xl−1 + F (Wl , Xl−1 ) + H(WlP , Xl )
0

0

0

0

Xl = Xl−1 + F (Wl , Xl−1 ) + H(WlP , Xl )
F (Wl , Xl−1 ) = Wl ∗ σ(Xl−1 )
0

0

0

0

F (Wl , Xl−1 ) = Wl ∗ σ(Xl −1)

0
0
l = 2k
σ(WlP ∗ Xl )
H(WlP , Xl ) =
0
l = 2k + 1

0
l = 2k
H(WlP , Xl ) =
σ(WlP ∗ Xl ) l = 2k + 1

(11)
(12)
(13)
(14)
(15)
(16)

0

where H denotes the Pipe operation, Wl is a set of convolution filters in the l-th layer of the TA stream, WlP represents
the learnable convolution filter parameters in the l-th P ipe.
The outputs from the two streams are concatenated. The
concatenated data is then fed into the same prediction layer
(global average pool followed by a softmax layer).

5. Experiments
In this section, we analyze our findings from the model
diagnosis and validate the effectiveness of our targeted refinement technique on 3D skeleton based human activity
recognition benchmark of NTU RGB+D [25].

5.1. Dataset and Settings
NTU RGB+D Dataset (NTU): The NTU RGB+D
dataset [25] is currently the largest 3D skeleton human action recognition dataset, which contains 56880 video samples of 60 action classes collected from 40 distinct subjects.
For each setup, a subject’s action is recorded from three
cameras of same height but from different viewing angles:
−45◦ , 0◦ and +45◦ . The dataset provides two criteria of
validation: Cross-Subject (CS) and Cross-View (CV) evaluations. The provided skeletons have 25 joints.

Figure 7. Targeted Model Refinement technique yields MS-Res-TCN architecture. The TA stream receives a masked input sequence and
the main stream receives the original input.

Implementation Details: We do not perform view-point
or scale normalization [8, 25, 33] and use the raw (X, Y, Z)
coordinates. Per time-step feature is 120 dimensional (25
joints, 2 at most actors, 3 xyz positions per joint). We
perform all our experiments on a Nvidia K80 GPU with
Keras2.0 [6] using a TensorFlow [1] backend. The learning
rate is initially set to 0.01 and then decreases by a factor of
10 when the testing loss plateaus. We train with Stochastic
Gradient Descent with all standard parameters and with a
batch size of 128. L-1 regularizer with a weight of 1e − 4 is
applied to all convolution layers and Dropout [28] (p = 0.5)
is applied after every ReLU operation. The implementation
and converged model weights will be made publicly available. (The link will not shared for this submission for the
sake of double-blind review).

and 6. Through the proposed diagnosis, we identified three
characteristic learning patterns of Res-TCN:

• The model learns a set of discriminative joints for actions
and the movement of these joints are amplified in deeper
layers. For example, as visualized on the left of Figure
4 the motion of joints related to legs, head and neck are
greatly magnified in deeper layers for action class jump
up. This makes intuitive sense that the vertical movement of the head is a rare occurrence in the training data
compared to other human actions in the dataset. Similar learning pattern is observed for action class cheer up
where vertical displacement of the hand joints is significantly amplified as the layers get deeper.
• In the case of NTU, the actions are recorded from multiple view angles (−45◦ , 0◦ and +45◦ ). Compared to other
models
[8, 25, 27] that manually normalize and rotate the
5.2. Interpretable Model Diagnosis
skeleton to a frontal view, we do not perform any scale or
Feature Map Decoder takes an arbitrary hidden represenrotational normalization. The sequences in Figure 5 detation Xl from Res-TCN and outputs a sequence of moving
pict the same action recorded from different view angles.
skeletons. The visualized results are shown in Figures 4, 5
Interestingly, the decoded skeleton sequences in Figure
5 show that the model has learned to rotate the skeleton
sequence implicitly to a more discriminative viewpoint
Table 1. Comparison to other state-of-the-art models on NTU.
(frontal view). Moreover, the decoded sequences proModel
Cross-Subjects Cross-Views
vide strong evidence that the model has achieved viewDynamic Skeletons [11]
60.2
65.2
point invariance. Despite given inputs from different
HBRNN [8]
59.1
64.0
view points, the model has learned to produce a comDeep LSTM [25]
60.7
67.3
mon representation for both inputs as evidenced by the
P-LSTM [25]
62.9
70.3
similarity of the decoded skeletons.
Trust Gate [20]
69.2
77.7
• Res-TCN fails to learn a representation for a detailed
STA-LSTM [27]
73.4
81.2
fine-grained motion of the hands. The decoded sequences
JTM [29]
76.3
81.1
for action classes such as Reading and Writing clearly
Pose Conditioned STA [3]
77.1
84.5
highlight that the model failed to learn a discriminative
Res-TCN baseline [15]
74.3
83.1
hidden representation. As seen in Figure 6, the decoded
MS-ResTCN, our model
79.0
86.6
skeletons for Reading and Writing show significant re-

Table 2. Detailed performance comparison between the Res-TCN baseline and MS-Res-TCN with Cross-Views evaluation. Highlighted in
bold are action classes that show meaningful increase in performance.
Baseline
MS-Res-TCN

Baseline
MS-Res-TCN

Baseline
MS-Res-TCN

Baseline
MS-Res-TCN

Baseline
MS-Res-TCN

Baseline
MS-Res-TCN

Drink
Water
0.7839
0.8418

Eat Meal
/ Snack
0.7225
0.7690

0.9589
0.9873

Brushing
Teeth
0.8168
0.8354
Tear Up
Paper
0.8174
0.8987
Hand
Waving
0.7652
0.8892

Taking a
Selfie

Check Time
(from watch)

0.8111
0.8386

0.8677
0.8544

Brushing
Hair
0.7915
0.8861
Wear
Jacket
0.9356
0.9778
Kicking
Something
0.8877
0.9082
Rub Two
Hands
Together
0.5962
0.6741

Reading

Writing

0.3759
0.5429
Take Off a
Hat/Cap
0.8810
0.9494
Pointing to
Something with
Finger
0.8540
0.8476

0.4610
0.4603

Sneeze/Cough

Staggering

Falling

Touch Head

0.7158
0.8291

0.8899
0.9146

0.9867
0.9873

0.7316
0.7500

Kicking
Other
Person

Pushing
Other
Person

Pat on Back
of Other Person

Point Finger at
Other Person

Hugging
Other
Person

Giving
Something to
Other Person

Touch Other
Person’s
Pocket

Hand
Shaking

Walking
Towards
Each Other

0.8673
0.9010

0.9300
0.9340

0.9007
0.8651

0.9166
0.9040

0.9651
0.9732

0.8424
0.9112

0.7997
0.9148

0.9280
0.9276

0.9492
0.9547

Cheer Up

0.9835
0.9937

Sitting
Down
0.9587
0.9714
Wear On
Glasses
0.8256
0.8544
Make a
Phone Call
0.7877
0.8323

Standing
Up
0.9832
0.9842
Take off
Glasses
0.9196
0.9082
Playing with
Phone
0.6665
0.6677

Shake Head

Wipe Face

Salute

Put the
Palms Together

Cross Hand
in Front

0.9323
0.9367
Touch
Chest
0.8085
0.8006

0.8582
0.8797

0.6196
0.7500

0.9073
0.9367

Touch Back

Touch Neck

Nausea

0.6949
0.7184

0.6006
0.7595

0.8500
0.9082

0.8519
0.8829
Use a Fan /
Feeling Warm
0.7750
0.7975

0.9067
0.9391
Punching /
Slapping
0.7905
0.8590
Walking
Apart
from
Each Other
0.9455
0.9677

Drop

Pick Up

Throw

0.8892
0.9177
Take Off
Jacket
0.9244
0.9492
Put Something
Inside Pocket
0.7535
0.8481

0.9098
0.9335
Wear
Shoe
0.7280
0.8280

0.9263
0.9589
Take Off
Shoe
0.6416
0.7524

Hopping

Jump Up

0.9715
0.9778

Nod Head/Bow

semblance. Moreover, as depicted in Figure 5.2, unlike
well-discriminative action classes such as Jump Up, response magnitudes of filters show no characteristic signature or no response at all for cases such as Reading.

Clapping
0.6614
0.7373
Put On a
Hat/Cap
0.8143
0.9270
Typing on
Keyboard
0.6335
0.6804

5.3. Targeted Refinement: Multi-stream Residual
Temporal Convolutional Networks
The diagnosis analysis revealed Res-TCN’s weakness
that it can not recognize fine-grained hand movements. We
aimed to pin point this shortcoming of our model through
the targeted refinement technique discussed in Section 4.2.
Table 1 shows that the diagnosis based refinement approach
(MS-Res-TCN) improves the classification performance of
the baseline Res-TCN model and achieves the state-of-theart on NTU benchmark. We also provide a detailed performance comparison between the models on the NTU benchmark in Table 2. The highlighted cells in the table indicate where the MS-Res-TCN has improved upon the baseline Res-TCN by a healthy margin. The highlighted action
classes share a common trait that the actions involve nuanced hand motion which validates the effectiveness of the
proposed targeted refinement approach.

6. Conclusion

Figure 8. The figures display filter response magnitudes plotted
over time. Top: Response plots of 5 samples with action class
Jump Up. The filters show Bottom: Response plots of 5 samples
with action class Reading. Lack of filter responses for Reading
indicates that Res-TCN has failed to learn motion primitives to
discriminate this action as opposed to well-converged action class
of Jump Up.

Through the ’train-diagnose-and-fix’ paradigm, this
work proposes an interpretable, intuitive yet an effective approach for learning powerful models for fine-grained action
recognition. The proposed Feature Map Decoder provides
an useful model interpretation and diagnosis tool which exposes key learning traits of the model. Through the analysis, we revealed that the model has learned to implicitly
rotate the input sequence to a more discriminative view and
gradually emphasize the characteristic spatio-temporal patterns of the input. We also demonstrate that our targeted
refinement technique is an effective and a systematic pro-

cedure to encode newly gained intuition into our models.
MS-Res-TCN, the resulting model after one iteration of the
’train-diagnose-and-fix’ cycle, validates that the proposed
approach is not only intuitive but also potent in learning discriminative representations for fine-grained skeleton based
action recognition tasks.

References
[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen,
C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia,
R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané,
R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster,
J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker,
V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org. 7
[2] P. Adler, C. Falk, S. A. Friedler, G. Rybeck, C. Scheideggery,
B. Smith, and S. Venkatasubramanianz. Auditing black-box
models for indirect influence. In ICDM, 2016. 2
[3] F. Baradel, C. Wolf, and J. Mille. Pose-conditioned spatiotemporal attention for human action recognition. CoRR,
abs/1703.10106, 2017. 7
[4] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual
representations. In CVPR, 2017. 2
[5] R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and N. Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. Proceedings of
the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1721–1730, 2015.
1
[6] F. Chollet et al. Keras. https://github.com/
fchollet/keras, 2015. 7
[7] F. Doshi-Velez and B. Kim. Towards a rigorous science of
interpretable machine learning. arXiv, 2017. 2
[8] Y. Du, W. Wang, and L. Wang. Hierarchical recurrent neural network for skeleton based action recognition. In CVPR,
2015. 1, 2, 6, 7
[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR, 2016. 1, 2, 3
[10] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in
deep residual networks. In ECCV, 2016. 2, 3
[11] J. Hu, W.-S. Zheng, J.-H. Lai, and J. Zhang. Jointly learning heterogeneous features for rgb-d activity recognition. In
CVPR, pages 5344–5352. IEEE Computer Society, 2015. 7
[12] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift. In
ICML, 2015. 3
[13] B. Kim. Interactive and interpretable machine learning models for human machine collaboration. Ph.D. thesis, Massachusetts Institute of Technology, pages 135–143, 2015. 1
[14] B. Kim, R. Khanna, and O. Koyejo. Examples are not
enough, learn to criticize! criticism for interpretability. In
NIPS, 2016. 2

[15] T. Kim and A. Reiter. Interpretable 3d human action analysis with temporal convolutional networks. In CVPR BNMW
Workshop, 2017. 1, 2, 3, 7
[16] P. Koh and P. Liang. Understanding black-box predictions
via influence functions. In ICML, 2017. 2
[17] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager.
Temporal convolutional networks for action segmentation
and detection. In CVPR, 2017. 2, 3
[18] J. Li, W. Monroe, and D. Jurafsky. Understanding neural
networks through representation erasure. arXiv, 2016. 2
[19] Z. C. Lipton. The mythos of model interpretability. In ICML
WHI Workshop, 2016. 1, 2
[20] J. Liu, A. Shahroudy, D. Xu, and G. Wang. Spatio-temporal
lstm with trust gates for 3d human action recognition. In
ECCV, 2016. 1, 2, 7
[21] J. Liu, G. Wang, P. Hu, L. Duan, and A. C. Kot. Global
context-aware attention lstm networks for 3d action recognition. In CVPR, 2017. 1
[22] A. Mahendran and A. Vedaldi. Understanding deep image
representations by inverting them. In CVPR, 2015. 2, 3
[23] A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and
J. Clune. Synthesizing the preferred inputs for neurons in
neural networks via deep generator networks. In NIPS, 2016.
2
[24] M. T. Ribeiro, S. Singh, and C. Guestrin. why should i trust
you?: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, pages 1135–1144,
2016. 1, 2
[25] A. Shahroudy, J. Liu, T. Ng, and G. Wang. Ntu rgb+d: A
large scale dataset for 3d human activity analysis. In CVPR,
2016. 1, 2, 6, 7
[26] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proc. ICLR, 2014. 2
[27] S. Song, C. Lan, J. Xing, W. Zeng, and J. Liu. An end-to-end
spatio-temporal attention model for human action recognition from skeleton data. In AAAI, 2017. 1, 2, 7
[28] N. Srivastava, G. Hinton, A. Krizhevsky, I., and R. Salakhutdinov. Dropout: A simple way to prevent neural networks
from overfitting. Journal of Machine Learning Research,
15:1929–1958, 2014. 7
[29] P. Wang, Z. Li, Y. Hou, and W. Li. Action recognition based
on joint trajectory maps using convolutional neural networks.
In Proceedings of the 2016 ACM on Multimedia Conference,
MM ’16, pages 102–106, New York, NY, USA, 2016. ACM.
7
[30] M. D. Zeiler and R. Fergus. Visualizing and understanding
convolutional networks. In ECCV, 2014. 1, 2
[31] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba.
Object detectors emerge in deep scene cnns. In ICLR, 2015.
2
[32] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization.
In CVPR, 2016. 2
[33] W. Zhu, C. Lan, J. Xing, W. Zeng, Y. Li, L. Shen, and
X. Xie. Co-occurrence feature learning for skeleton based

action recognition using regularized deep lstm networks. In
AAAI, 2016. 1, 2, 6

