Understanding Hidden Memories of Recurrent Neural Networks
Yao Ming *

Shaozu Cao *
Ruixiang Zhang *
Zhen Li *
Yuanzhe Chen *
*
Yangqiu Song, Member, IEEE
Huamin Qu, Member, IEEE *
Hong Kong University of Science and Technology

Models

Main

A

Model
Cell: BasicLSTM

cell state

0.7

LayerSize: 1500

hidden state

Layer

D

C
investors banks

some

Understanding Hidden Memories of RNNs for
Text.

lower

interest

cash

E

market court company prices

markets week concernsale
operations bonds
unit
bill

board years

$
funds days time
months rates
firm month plan bid

-0.7

-0.5
-1.0
-1.5
0

200

400

600

1,000 1,200 1,400

dim

of(sentence)

Info: Hidden State
words

a
if
this
least
by
cents
of
between
-1.5-1.0-0.50.00.51.01.5

Dim: 189

our american he net

federal there the

its
another friday
japan mr. it any

first
those a state
california city

-0.7

words

third u.s. million

0.7

r

-2.0
-1.5
-1.0
-0.50.00.51.01.52.0

Dim: 888

-0.7

quarter

office rateday department
year

already then n't capital

fund just money growth
now tax also
only

least still

text

by
of
in
from
million
billion
according
part

issues

0.7

for

of

800

far

operating

used
sold
fell up
says say
added earlieroff

<unk>

1.5
1.0
0.5
0.0

early

during according
due said
reported end
ago rose over

0.7

response

been were

late less more was
even

have law
cents securities

-0.7

Info: Word

trade work

being financial get well
closed is has 're had

of

Link Filter

insurance

three recent two many
much several buy
few
both big good higher

0.7

Stroke Width

most

where to of who how
through at before
since <eos>
with in than after
between or so and but

-0.7

Cluster Num

that

out
world
president partone
all

0.7

Sentences

an

last

corp.

Algin

POS Tag

VERB
NOUN
PRON
ADJ
ADV
ADP
CONJ
DET
NUM
PRT
X
.

i

would 's
' do
should
might no did about are very

-0.7

memories

arXiv:1710.10777v1 [cs.CL] 30 Oct 2017

B

hidden

Hidden State

LayerNum: 2

-1.0

own income

1.0

0

Figure 1: The interface of RNNVis. The control panel (A) shows parameter settings of an RNN and allows users to adjust visualization
style. The main view (B-D) contains glyph-based sentence visualization (B), memory chips visualization for hidden state clusters (C),
and word clouds visualization for word clusters (D). The detail view (E) shows the distributions of models responses to selected word
“of” and interpretations of selected hidden units.

A BSTRACT
Recurrent neural networks (RNNs) have been successfully applied
to various natural language processing (NLP) tasks and achieved
better results than conventional methods. However, the lack of
understanding of the mechanisms behind their effectiveness limits further improvements on their architectures. In this paper, we
present a visual analytics method for understanding and comparing
RNN models for NLP tasks. We propose a technique to explain
the function of individual hidden state units based on their expected
response to input texts. We then co-cluster hidden state units and
words based on the expected response and visualize co-clustering
results as memory chips and word clouds to provide more structured
knowledge on RNNs’ hidden states. We also propose a glyph-based
* e-mail:
{ymingaa, scaoad, rzhangav, zhen, ychench, yqsong,
huamin}@ust.hk

sequence visualization based on aggregate information to analyze
the behavior of an RNN’s hidden state at the sentence-level. The
usability and effectiveness of our method are demonstrated through
case studies and reviews from domain experts.
Keywords: recurrent neural networks, visual analytics, understanding neural model, co-clustering
1

I NTRODUCTION

Recurrent neural networks (RNNs) are deep learning models that
are very effective in modeling sequence data such as text and speech.
RNNs and their variants, especially long short term memory (LSTM)
networks and gated recurrent units (GRU), have been successfully
applied in various natural language processing (NLP) applications,
which include language modeling [30], machine translation [44],
speech recognition [17] and sentiment analysis [46]. Recent work
extended RNNs to solve challenging tasks such as reading comprehension [19] and video analysis [9], and achieved better performance
than traditional approaches.
Despite their impressive performances, RNNs are still “black

boxes” that are difficult for humans to understand. As a basic component for various applications, RNNs have recently attracted researchers’ interests in designing better model architectures (e.g., the
recently proposed GRU [5]). However, the lack of understanding of
how RNN models work internally with their memories has limited
researchers’ ability to introduce further improvements. A recent
study [37] also emphasized the importance of interpretability of
machine learning models in building users’ trust: if users do not
trust the model, they will not use it.
The three major challenges of interpreting RNNs are as follows.
First, RNNs maintain memory-like arrays called hidden states which
store information extracted from a long input sequence (e.g., texts
or audios). As the input comes in, hundreds or thousands of hidden
state units are updated with nonlinear functions specified by millions
of parameters. The size of hidden states and parameters brings difficulties of analyzing an RNN’s behavior. The number of parameters
will further increase as the model becomes more complex. Second,
RNNs mainly deal with sequential data such as texts and audios. The
complex sequential rules (e.g., grammar, language models and social
language codes) embedded in texts are intrinsically difficult to be
interpreted and analyzed. Third, through experiments, we found that
semantic information in hidden states is highly distributed, i.e., each
input word will generally result in changes almost every hidden state
units. Similarly, each hidden state unit may be highly responsive to
a set of words. The many-to-many relationship between hidden state
units and words further impedes researchers from understanding the
information embedded in hidden states of RNNs.
To address these challenges in interpreting RNN models, a few
recent studies have compared the behavior of hidden states in different RNN architectures. Karpathy et al. [25] found that some hidden
state units can be interpreted as language context, such as quotes or
punctuations, whereas others do not seem to represent any semantics.
Strobelt et al. [42] used parallel coordinates to present values of a
few selected hidden state units with a pattern searching function
for experts to explore the dataset. However, most existing studies
focused on a limited aspect of the model only. There is still a lack of
thorough understanding of what information is recorded in hidden
states and how the information is distributed and updated.
In this work, we develop a general visual analytics system called
RNNVis (Fig. 1) for deep learning practitioners to understand and
diagnose RNNs and explore the behavior of their hidden memories.
Inspired by previous work in convolutional neural networks (CNNs),
we introduce a technique to interpret functions of each hidden state
unit using its highly correlated words from the input space. The
correlation between a hidden state unit and a word is measured using
the expected update of the hidden state unit given the word as input.
To understand how the functions of hidden state units in an RNN are
composed together, we model the relation between hidden states and
words as a bipartite graph, in which hidden state units and words
are treated as two types of nodes connected by weighted edges. To
enable scalable visual analysis for RNNs with a large size of hidden
states, we arrange the bipartite graph into co-clusters. Hidden state
clusters and word clusters are then visualized as memory chips and
word clouds, which are convenient for explorations. Based on the
co-cluster visualization, the sentence-level behavior of an RNN is
examined by a glyph-based visualization using aggregate statistics
of hidden states. Rich interactions are also provided for users to
explore and compare the hidden behavior of different models.
Our major contributions are:
• The design and implementation of RNNVis, a visual analytics
system for understanding, comparing, and diagnosing RNNs
for general text-based NLP tasks.
• A new technique for bridging hidden states and textual information with expected response.
• A glyph-based design for sequence visualization to analyze
the sentence-level behavior of RNNs.

• Case studies on the analysis of hidden mechanisms of RNNs
with different architectures using the proposed system.
2

R ELATED W ORK

In this section, we review some related work on understanding RNNs
and approaches to visualize neural networks.
2.1

Understanding RNNs

In the field of computer vision, significant efforts were exerted
to visualize and understand how the components of a CNN work
together to perform classifications. These studies (Zeiler & Fergus
[52], Dosovitskiy & Brox [10]) provided researchers with insights
of neurons’ learned features and inspired designs of better network
architectures (e.g., the state-of-the-art performance on the ImageNet
benchmark in 2013 proposed by Zeiler & Fergus [52]).
However, these methods are difficult to be applied or adapted to
RNNs. The lack of interpretability of the hidden behavior in RNNs
recently raised concerns in the NLP community. As a critical basis
of various neural models in NLP, RNN has attracted an increasing
number of studies that focus on analyzing its hidden activities and
improving its architectures. These studies can be divided into two
categories: performance-based analysis and interpretability-guided
extensions.
Performance-based methods analyze model architectures by altering critical network components and examining the relative performance changes. Greff et al. [18] conducted a comprehensive study
of LSTM components. Chung et al. [7] evaluated the performance
difference between GRUs and LSTMs. Jozefowicz et al. [22] conducted an automatic search among thousands of RNN architectures.
These approaches, however, only show overall performance differences regarding certain architectural components, and provide little
understanding of the contribution of inner mechanisms.
Another worth mentioned type of neural models extends RNN
with an attention mechanism to improve the performance on specific
tasks. Bahdansu et al. [2] applied the attention in machine translation
and showed the relationship between source and target sentences. Xu
et al. [50] designed two attention-based models in image captioning,
which revealed the reasons behind the effectiveness of their models.
Although the attention mechanism can benefit the interpretation
without extra effort, it requires jointly training different models or
modifying the original model, which limits its application in general
RNN models.
2.2

Visualization for Machine Learning

There is a trend for combining visualization and machine learning
in recent years.
On the one hand, visualization has been increasingly adopted by
the machine learning community to analyze [36], debug [38], and
present [1] machine learning models. On the other hand, a number
of human-in-the-loop methods have been proposed as competitive
replacements of full-automatic machine learning methods. These
methods include: visual classification [6, 32, 48], visual optimization
[24, 45], and visual feature engineering [4, 41, 49].
In the field of deep learning, some recent studies have utilized
visualization to help understand RNNs. Tang et al. [47] studied the
behavior of LSTM and GRU in speech recognition by projecting
sequence history. Karpathy et al. [25] showed that certain cell
states can track long-range dependencies by overlaying heat map on
texts. Li et al. [26] also used heat maps to examine sensitiveness of
different RNNs to words in a sentence. However, their visualizations
only provided an overall analysis of RNNs. These studies did not
explore RNN’s hidden states in detail.
In the field of visualization, recent work has exhibited the effectiveness of visual analytics in understanding, diagnosing and
presenting neural networks. Liu et al. [27] treated deep CNN as
a directed acyclic graph and built an interactive visual analytics

system to analyze CNN models. Rauber et al. [34] applied dimensionality reduction to visualize learned representations, as well as
the relationships among artificial neurons, and provided insightful
visual feedback of artificial neural networks. While visualization
has achieved considerable success on CNNs, little work has focused
on RNNs. Most related to our work, Strobelt et al. [42] has proposed
an interactive visualization system to explore hidden state patterns
similar to a given phrase on a dataset. This system also allows users
to flexibly explore given dimensions of hidden states. However, the
parallel coordinates design is not scalable for efficiently analyzing
hundreds or thousands of hidden state dimensions.
2.3

Co-Clustering and Comparative Visualization

Our work is closely related to two techniques: co-clustering and
comparative visualization.
We formulate the relation between hidden state units and discrete inputs of RNNs as bipartite graphs to investigate the structure
of information stored hidden states. Co-clustering is a widely used
method for analyzing bipartite graphs, which simultaneously clusters
two kinds of entities in a graph [28]. Some recent work combined
co-clustering with visualization to assist intelligence analysis, where
different types of entities are considered [13,43]. A most recent work
proposed by Xu et al. [51] presented an interactive co-clustering visualization where cluster nodes are visualized as adjacency matrices
or treemaps. Although both adjacency matrices and treemaps used
in this visualization are well established, none could be adjusted to
visualize abstract entities like hidden states.
Comparative visualization was adopted to fulfill the design requirements of RNNVis. Gleicher et al. [15] suggested three typical
strategies for comparative visualization, namely, juxtaposition (or
separation), superposition (or overlay), and explicit encoding. We
mainly employ juxtaposition and superposition for comparing RNNs
at three different levels, namely, detail, sentence, and overview levels.
The details of the design choices are discussed in Sect. 6.2.
3

BACKGROUNDS

The basic architecture and concepts of RNNs are introduced to serve
as a basis for the discussions of the next sections.

···

y(t − 1)

y(t)

y(t+1)

h(t − 1)

h(t )

h(t+1)

tanh

tanh

tanh

x(t− 1)

x(t)

···

x(t+1)

Figure 2: Illustration of the vanilla recurrent neural network.

Recurrent Neural Network. As shown in Fig. 2, a vanilla RNN
takes sequential inputs {xx(0) , · · · , x (T ) }, where x (t) ∈ Rm and maintains a time-variant hidden state vector h (t) ∈ Rn . At step t, the
model takes input x (t) , and updates the hidden state h (t−1) to h (t)
using:
W h (t−1) +V
V x (t) )
h (t) = f (W
(1)
where W and V are weight matrices and f is a nonlinear activation
function. In this paper, we use tanh as activation function, which
constraints value range of h (t) to (−1, 1).
Softmax Output. After the updates at each step, h (t) may be
further processed or directly used as output. For instance, to perform

classification, a probability distribution p over K classes can be
computed after processing the whole sequence at step T :
U h (T ) )i =
pi = softmax(U

exp(uuTi h (T ) )
K exp(u
u j h (T ) )
∑ j=1

(2)

where U = [uu1 , · · · , u n ]T is the output projection matrix.
We can associate RNNs with Turing Machines in the sense that
both of them maintain a piece of “memory”. As the input sequence
passed in step by step, an RNN updates its memory and outputs
some results according to its current memory.
In this paper, we will also use two RNN variants: long short-term
memory (LSTM) networks and gated recurrent units (GRUs). Their
definitions can be found in Appendix A. The multi-layer models are
also defined in the appendix.
y(t )

y(0)

y(··· )

y(t )

y(0)

y(··· )

y(t)

RNN

RNN

RNN

RNN

RNN

x(0)

x(··· )

x(t)

RNN

RNN

RNN

RNN

RNN

RNN

x(0)

x(··· )

x(t)

x(0)

x(··· )

x(t)

(a)

(b)

(c)

Figure 3: Input-output schemes of RNN.

In different application scenarInput-Output Schemes.
ios, an RNN typically has one of the following input-output
schemes: sequence-to-one, sequence-to-sequence, synced sequenceto-sequence, as shown in Fig. 3. RNNs can take a sequence of
inputs and only output at the end of the whole sequence (Fig. 3(a)),
which is typical in sentiment analysis or document classification.
The sequence-to-sequence (Fig. 3(b)) formulation is widely used in
machine translation (e.g., translating a sentence to another language).
More generally, certain tasks (e.g., language modeling and video
classifications) generate outputs at each step of the input sequence
(Fig. 3(c)). To evaluate our proposed system, we used sequenceto-one (e.g., sentiment analysis for texts) and synced sequence-tosequence (language modeling) for illustrations. Other RNN-based
models can also be analyzed given that the proposed visual analytic
method only requires recorded information of hidden states.
4

S YSTEM D ESIGN

In this section, the requirements of the current system for interpreting
and analyzing the hidden states of RNNs are discussed and formulated. The proposed techniques and visual designs are presented in
Sect. 5 and Sect. 6 respectively.
Throughout the design and implementation of RNNVis, we
closely worked with two experts in the deep learning and NLP
domains, who are also co-authors of this work. One expert (E1) is a
deep learning researcher with strong industrial experience, and the
other expert (E2) is a senior researcher who specializes in natural
language understanding and text mining. Decisions on algorithm
design (Sect. 5) and visual encoding choices (Sect. 6) for the system
are determined through iterative discussions with collaborators.
4.1

Requirement Analysis

The major focus of RNNVis is to provide intuitive interpretations of
RNN’s hidden states, and make use of the knowledge to diagnose
RNN models and inspire better architecture designs. Based on the
discussions with domain experts and literature review, the specific
requirements are formulated as follows:

R1 Clearly interpret the information captured by hidden
states. Current visualization techniques for hidden states either
select a few state units that may be easy to interpret [25], or
directly map hidden state values to visualizations like parallel
coordinates [42]. However, as most interpretable information
is distributively stored in hidden states, direct visualization
cannot provide explanations for each hidden unit. Thus, the
most basic requirement is to visually explain the semantic information captured by each hidden state unit. For example,
what kinds of words or grammars are captured and stored in a
hidden unit?
R2 Provide the overall information distribution in hidden
states. Besides the interpretation of individual hidden states,
an overview of how semantic information is structured within
hidden memories can provide experts with a full picture of the
model. For example, how is the stored information differentiated and correlated across hidden states?
R3 Explore hidden states mechanisms at the sequence-level.
During discussions, domain experts expressed the need to analyze RNN’s ability in modeling sequences, which is the most
advantageous feature of RNN. A widely used visualization
technique for RNN is to project the learned word embedding
to a 2-D space. However, word-level visualization cannot
reveal the reasons behind RNN’s effectiveness in modeling
sequence. That is, how does the internal memory updating
mechanism of an RNN result in its specific behavior when
dealing with sequences?
R4 Examine detailed statistics of individual states. Experts
also suggested that concise and detailed information, such
as the distribution of hidden state values or gate activations, is
required for quantitatively analysis.
R5 Compare learning outcome of models. A general but important requirement is the comparison of different RNN architectures. For example, what are the internal reasons that one
model is better than the other? What are the mechanisms that
arm LSTMs the ability to master long-term dependency?
Interaction

Model Manager

Evaluator

Load / Save

Correlation
Estimator

Train / Test

Log

Datasets
Model Records & Logs

DataBase

after new $ though
big not i free

why
full neverca
're
went if price
work n't away
eat

we

she

help

Config / Build

Feed

Visualization

Hidden state
Interpretor

every

chicken both where
experienceplaces
still 'mnext helpful
around love
bar should then
well like year
say
years awesome
put different
friendlycustomer

Spectral
Co-Cluster
Sentence
Evaluator

Figure 4: System architecture. Left: the model manager that manages
RNN models and datasets. Middle: the RNN evaluator that evaluates
the model provided by the model manager. Right: the visualization
module.

4.2

System Overview

As shown in Fig. 4, RNNVis consists of three major modules: model
manager, RNN evaluator, and interactive visualization.
The model manager utilizes TensorFlow [1], a numerical computation library for machine learning, to build, train and test RNN
models. The model manager is designed to loosely couple with
other modules to offer flexibility when adapted to other RNN based

models or different machine learning platforms. Users are allowed
to edit configuration files to easily change models’ architectures.
New datasets can also be added by extending the pre-defined data
pipeline provided by the model manager.
The RNN evaluator then analyzes trained models to extract
learned representations in hidden states, and further processes the
evaluation results for visualization. This module also offers functionalities to derive interpretations for each hidden state unit in word
space. The relation between hidden states and input words are then
stored as a bipartite graph. For scalable visualization, we applied a
co-clustering algorithm to simultaneously cluster the hidden state
space and word space, so that hundreds of hidden states can be
explored with ease.
Evaluation results of the RNN models are finally provided as a
co-clustering visualization (Fig. 1). Users first select a trained model
in the control panel (A), where architectural parameters of the model
are also listed. The main view (B-D) then presents hidden state
clusters as memory chips (C), word clusters as word clouds (D), and
layouts them in a way that highly correlated co-clusters are closely
positioned. To analyze the sequential behavior of the model, users
can utilize sequence visualization (B) through inputting sentences
in the control panel. Users can further explore the distribution of
model’s responses to particular words by clicking words or memory
cells in the main view and examine the detail view (E). The control
panel also provides controls to help users adjust visualization style.
Users can also learn the usage of the system by watching a guidance
video which introduces each view.
5

RNN E VALUATOR

Before presenting the interface and interaction design, we first discuss the techniques used in the RNN evaluator.
5.1

Interpreting hidden states

In CNNs, the features learned by neurons can be visually explained
using images derived by activation maximization [12] or code inversion [52], since the input space of CNN is continuous. However, the
input space of RNNs applied in NLP usually consists of discretized
words or characters, where these image-based methods fail to generalize. Although gradient-based methods [26] managed to provide
overall interpretations on model’s predictions, they are difficult to
be applied to intermediate hidden states h (t) , whose gradients are
very sensitive with regards to h (t−1) .
Inspired by the idea of using interpretable representations to
explain functions of network components, we propose a method
to intuitively interpret individual hidden state unit using words or
characters. For simplicity, we only use the word-level model for
illustration, but character-level models can also be used.
We first show that the numerator of probability pi in Equation 2
can be decomposed into a product of factors using the method proposed by Murdoch and Szlam [31] (we always set h (0) = 0 ):
!
exp(uuTi h (T ) ) = exp

T

∑ u Ti (hh(t) − h (t−1) )

t=1

T

= ∏ exp(uuTi ∆hh(t) ).
t=1

(3)
Here, exp(uuTi ∆hh(t) ) can be interpreted as the multiplicative contribution of word t to the predicted probability of class i, and
∆hh(t) = h(t) − h(t−1) can be regarded as model’s response to input
word t.
Although h (t) is calculated by a non-linear transformation of
(t−1)
h
and x (t) , ∆hh(t) is deterministic to the input x (t) when the previous history h (t−1) is given. Thus, ∆hh(t) can reflect to what degree
the model’s hidden state is influenced by the input x(t) . However,
given the same word x, ∆hh(t) might vary due to its dependence on

h (t−1) . Consequently, we formulate x (t) and h (t) as random variables, and use model’s expected response to a word w as a more
stable measure of the word’s importance on hidden state units. We
will also show that this formulation is empirically effective.
In NLP, where inputs are sequences, it is common to regard input
words w(t) as random variables [23, Chapter 4]. Thus, the corresponding word embedding vectors x (t) can be regarded as discrete
random variables. Since the hidden state h (t) is deterministically
computed from a sequence of random variables x (0) , · · · , x (t) , it is
also eligible to consider h (t) as random variables [16, p. 389].
The expected response to a word w is then computed by Adam’s
Law:
s(xx) = E(∆hh(t) | x ) = E(E(∆hh(t) | x , h (t−1) ))
(4)
where x is the embedding vector of w, and s(xx)i represents the
relation between the ith hidden state unit hi and w. Note that with the
tanh activation function, the response s(xx)i can have either positive
or negative value. A larger absolute value of s(xx)i indicates that
x is more “salient” or important to the hidden state unit hi . The
advantage of Equation 4 is that, with enough data, we can easily
estimate the expected response using all observations of ∆hh(t) on the
word w:
1
(5)
ŝ(xx) =
∑ ∆hh(t) .
∑x (t) =xx 1 x (t) =xx
The explanation of a hidden state unit i is then formulated as m
words with top m absolute expected responses. In our prototype,
users can adjust the parameter m.
For an LSTM, which has two state vectors, we calculate the
model’s expected response based on the update of cell states, i.e.,
s(xx) = E(∆cc(t) | x ). The reason of this specification is that cell
state c (t) is considered to maintain long-term memory, while h (t)
is directly computed from cell state and used for output. However,
based on the above formulation, RNNVis can also be used to analyze
the behavior of h (t) of an LSTM.
In Fig. 5, we show the distributions of a two-layer LSTM’s responses, ∆cc(t) , given three different words. The LSTM has 600 cell
state units per layer and is trained on the Penn Tree Bank (PTB)
dataset [29]. We can see that the hidden state units in the left and
right end are highly responsive to “he” and “she”. In addition, we
can see that the model’s response patterns differ a lot between prepositions (“for”) and pronouns (“he” and “she”).
response
2.5
2.0
1.5

1.0
0.5
0.0
-0.5
-1.0
-1.5
-2.0

hundred to a few thousand. Showing the interpretations of one
hidden state unit at a time creates cognitive burdens to users, and
does not offer clues about RNN’s high-level behavior of all hidden
states as a whole.
To generate an overview for hidden units or neurons, the machine
learning community has widely adopted projection methods (e.g.,
t-SNE and PCA) to explore or illustrate the patterns inside model’s
learned features [35]. For example, in NLP, projection is a common
approach in demonstrating model’s effectiveness in mastering wordlevel semantic relationships such as analogy. However, projecting
word embedding alone only provides an overall structure of the
learned representations. Analyzing a large number of hidden states
still requires a more concise overview.
Given the expected response s(xx) derived from Equation 4, for
each word, we have the expected responses of n hidden units, and
for each hidden unit, we have its expected responses to N words.
By viewing words and hidden units as nodes, we can model this
multi-to-multi relation as a bipartite graph G = (Vw ,Vh , E), where
Vw and Vh are sets of word nodes and hidden unit nodes, respectively.
The relations between hidden state units and words are then treated
as weighted edges E = {ei, j = s(xx(wi )) j | wi ∈ Vw , h j ∈ Vh }, where
x (wi ) is the embedding vector of word wi , and h j is the jth hidden
unit of hidden state vector h . As a natural way of analyzing bigraphs, co-clustering (or bipartite graph partitioning) [51] is used to
structure hidden state space and word space for easier exploration
while preserving the relation between two spaces. In the prototype
of RNN evaluator, the spectral co-clustering algorithm [8] with kmeans++ initialization is used to simultaneously cluster hidden state
units and words. Other co-clustering algorithms can also be applied.
In the presented case, structured hidden state clusters and word
clusters also facilitate the provision of overview visualization as
required in R2. Our method outperforms projection-based methods
in that it shows the word-level semantics directly with clusters. For
instance, the highlighted word cluster (in the form of a word cloud)
in Fig. 1D roughly presents “preposition”. In addition, the semantics
formed by word clusters also serves as concise interpretations for
corresponding hidden state clusters.
5.3

Sequence Analysis

To support sequence-level analysis as required in R3, we discussed
with the experts and formulated a few aggregate measurements to
profile RNN’s behavior at sequence-level. All the measurements
are defined as cluster-level summaries at each step of p hidden state
clusters {H1 , H2 , · · · , H p } obtained from Sect. 5.2. We will show the
usefulness of these measurements in Sect. 6.1.4.
Aggregate Information of a cluster Hi at a step t is defined as
(t)
(t) (t)
(t)
(t)
αi = (αi+ , αi− ), where αi+ and αi− are the sums of positive and
negative hidden units in cluster Hi :
(t)

αi+ =

(t)

hj ,

∑

(t)

αi− =

h j ∈Hi ,h j >0
0

50

100

150

he

200

250

300

she

350

400

450

500

550

600

dim

for

Figure 5: Distribution of an LSTM’s response to input words “he” and
“she” and “for”. Horizontal axis represents 600 dimensions of the
hidden states of the last layer. The dimensions are sorted according
to the model’s estimated expected response to “he”, ŝ(“he”). The
darker area denotes 25% to 75% range of the distribution, while the
lighter area denotes 9% to 91% range.

5.2 Co-clustering Hidden States and Words
In NLP, state-of-the-art RNN models have about two to four layers,
with the size of hidden states vector per layer ranges from several

∑

(t)

hj .

(6)

h j ∈Hi ,h j <0

Since a larger absolute value of a hidden unit |h j | represents that it
is more activated or stores more information, αi+ and αi− represent
to what degree a cluster of hidden units is positively or negatively
activated. We separate the information into positive αi+ and negative
αi− because they generally refer to different semantics. For example,
in sentiment analysis, if a hidden unit encodes positive sentiment
with positive value, it will very likely encode negative sentiment
with negative value. Besides, the sign of αi+ + αi− can flag the
characteristics (positive or negative) of the information stored in
cluster Hi .
(t)
(t)
Updated information of Hi is defined as δ (t) = (∆αi+ , ∆αi− ),
(t)

(t)

(t−1)

where ∆αi+ = αi+ −αi+

is the change of positive aggregate infor(t)

mation of cluster Hi at step t, whereas ∆αi− is the change of negative

(t)

(t)

information. The absolute updated information |∆αi+ + ∆αi− | can
be used to measure the sensitiveness of hidden state cluster Hi with
the current input. This measure is useful for examining the most
responsive hidden state clusters to the current input. Particularly, a
(t)
(t)
high value of |∆αi+ + ∆αi− | presumes that Hi is highly correlated
with x (t) .
Preserved information measures how much information in a
hidden state cluster Hi has been retained after processing a new
(t)
input xt . The preserved information of Hi , βi , is defined as:
(t)

(t)

βi

=

∑

(t−1)

|h j

|min(1, max(0,

h j ∈Hi

hj

(t−1)

))

(7)

hj

(t)

(t−1)

where the latter min-max term is used to clip the value of h j /h j
(t)

into range (0, 1). βi

can be thought as the intersection volume
(t)

(t−1)

between current and previous aggregate information αi , αi
.
For LSTM or GRU, gate information can be directly used instead
of the zero to one clipper in Equation 7. This index is useful for
examining the overall hidden state updating characteristics.
6

V ISUALIZATION D ESIGN

In this section, we discuss the design choices and interaction designs
of RNNVis based on the design requirements (Sect. 4.1).
6.1

Visual Representation

Hidden state clusters and word clusters have different intrinsic characteristics (i.e., one is abstract components while the other is interpretable texts). Thus, to present a better overview of RNN models
(R2), we visualize word clusters as word clouds [14] and hidden
state clusters as memory chips. Together with the sequence-level
representation, we organize the main view with a three-part layout,
as shown in Fig. 1. PivotPath [11], ConVis [21] and NameClarifier [39] have inspired our layout design, which reduces visual clutter
across different entities and provides an easy interface for interaction
design.
6.1.1

Hidden State Clusters as Memory Chips

We visualize each hidden state unit as a small square-shaped memory
cell and pack memory cells in the same cluster into a rectangular
memory chip to allow exploration of details (R4). All the memory
chips are vertically stacked and aligned by their centers (Fig. 1C)
to present an overview of RNN’s hidden state mechanisms (R2).
By default, memory chips have the same heights so that larger
hidden state clusters will be visualized with larger width. In case
of unbalanced clustering results, we also provide an alternative
layout in which memory chips are aligned with equal widths to
achieve a space efficient arrangement. This design is inspired by
the association between RNN and Turing machine, where hidden
state units can be associated with memory cells, which is intuitive
for computer science researchers.
With the memory-like layout, we also use a divergent blue to orange color to represent the response value of hidden state units based
on interaction. For example, with a given word with embedding x
selected, each hidden unit h j is rendered with the value of estimated
expected response s(xx) j .
Since our experiments show that a hidden state cluster of a typical RNN can contain up to a few hundred units, we adaptively
pack memory cells into a few rows according to the size of hidden
state cluster and the current window size for scalable visualization.
The current implementation can handle up to 20 clusters with two
thousand hidden state units.

6.1.2 Word Clusters as Word Clouds
Each word cluster is visualized as a word cloud, and all the word
clouds are arranged in a circular layout as shown in Fig. 1D. To
utilize the space more efficiently, word clouds are bounded by rectangles rather than circles to show more words. Word clouds present
summary interpretations (R1) of hidden state clusters, and allow
users to navigate through clusters to validate the clustering quality.
When users click on a hidden state cluster, highly correlated word
clouds are highlighted to present users with an overall interpretation of the cluster. Word clouds also serve as an entry interface for
analyzing an RNN’s detailed behavior on individual words.
In a word cluster Wi , the size of each word wi j is proportional
to the Euclidean distance d(ci , wi j ) between wi j and the cluster’s
centroid ci = ∑ j wi j /|Wi |. Thus, more centered words will have
larger sizes in the word cloud. This design allows users to quickly
identify the most representative words of a cluster. To maintain the
completeness of the interpretation rather than only showing a few
representing words, we linearly scaled the words to readable size.
Users can also turn on the grammar mode, which rendered words
into different colors according to their part of speech (POS) tags. For
words that have multiple possible POS tags, we render them according to its most frequent POS tag in the datasets. For implementation
details, we use a pre-trained Greedy Averaged Perceptron tagger in
NLTK [3] to first tag words with the PTB tag set, then convert the
POS tags into universal tag set [33] which contains only 12 tags.
6.1.3 Co-Clustering Layout
As shown in Fig. 1, the memory chips and word clouds are separately
positioned and linked (R2). To avoid visual clutter, we aggregate
the bipartite connections between words and hidden state units into
cluster-to-cluster edges, which is a similar practice of Xu et al.
[51]. The width of a cluster-to-cluster edge indicates the aggregate
correlation between a word cluster Wi and a hidden state cluster H j ,
which is computed as the average of bi-connected edge weights:
e(Wi , H j ) =

1
|Wi | × |H j | w

∑

s(xx(wx ))y

(8)

x ∈Wi ,hy ∈H j

where |Wi | and |H j | are cardinalities of Wi and H j . To distinguish
the positive and negative correlations, we use blue and orange to
represent negative and positive edge weights. The blue-orange diverging color scheme is consistently used to visualize negative and
positive values throughout our design.
With spectral co-clustering, hidden states and words are clustered
into one-to-one clusters. Thus word clouds can be easily positioned
with the same order as hidden state clusters to minimize visual clutter.
When using other co-clustering algorithms, where co-clusters does
not form pairs, force-based layouts can be applied to position word
clouds.
Here the circular layout of word clouds is designed to reduce
visual clutter of the edges between bipartite clusters [21]. Moreover,
the circular arrangement is cleaner and more time efficient than a
pure force-based layout as proposed by Xu et. al [51].
6.1.4 Glyph Design of Sequence Nodes
To enable sequence-level analysis of RNN (R3), we design a glyphbased sequence visualization. Here a sequence refers to a sentence
or a paragraph, and the nodes denote words of the sentence. The
measurements used in the glyph design are discussed in Sect. 5.3.
As shown in Fig. 6(c), the bar chart based glyph is designed to
help experts understand the sequence-level behavior of an RNN.
The bold rectangle bounded part of each bar denotes the average
(t)
aggregate information αi /|Hi | of a hidden state cluster Hi , split by
(t)

the horizontal zero line into positive information αi+ /|Hi | (upper
part) and negative information

(t)
αi− /|Hi |

(lower part). The top and

0.5

sold
-0.5

(a)

(b)

(c)

Figure 6: Alternatives of the glyph design for sequence node. (a) a
pie chart based design. (b) a bar chart design with bar width encoding
cluster size and (c) the composite bar chart glyph used in RNNVis.

bottom colored “hat” of each bar represents the updated information
(t)
(t)
∆αi+ /|Hi | and ∆αi− /|Hi | respectively. To keep a consistent design
style, we use orange color to encode an increase in the positive
information or a decrease in the negative information, and use blue
color to encode the opposite. Note that the increased information
is always a part of the aggregate information, while the decreased
information is placed outside the bounding box. The order of the bar
is consistent with the order of hidden state clusters in the co-cluster
layout. Users can also hover on any bar to highlight its corresponding
hidden state cluster. At the top of the squared glyph is a control chart
showing the percentage of information that flows from the previous
step. The position of the cursor on each vertical line represents the
preserved information normalized by the corresponding aggregate
(t)
(t)
(t)
information βi /(αi+ − αi− ) in a hidden state cluster.
During the design process, we have considered several design
alternatives as shown in Fig. 6. Although a pie chart based glyph
(Fig. 6(a)) is aesthetic, it cannot be used to compare the aggregate
information and updated information simultaneously. As for the
variable-width bar chart (Fig. 6(b)) which encodes cluster sizes
as bar widths, the experts found it inconvenient to identify small
clusters. Also, the choices of categorical colors are limited when the
number of clusters is large. Comparing all three glyph designs, the
experts felt design (c) is the most helpful one, since they can easily
tell the direction (positive or negative) and extent of the information
change in each cluster by simply identifying the major color of the
corresponding bar.
Based on the glyph design, a sentence is visualized as a sequence
of glyph nodes, as shown in Fig. 1(B). Users can click on a single
node to highlight the links to its most responsive hidden state clusters.
Here the link width between a node wt and hidden state cluster Hi is
proportional to the absolute value of average updated information
(t)
(t)
|∆αi+ + ∆αi− |/|Hi |. The color of the link denotes the updated
(t)

(t)

information ∆αi+ + ∆αi− in this cluster is positive (orange) or
negative (blue).
6.1.5

Layout Alternatives

Several alternative designs were considered before the co-cluster
based bipartite visualization was finalized.
As shown in Fig. 7(a), an alternative layout that we proposed
at the beginning uses t-SNE to project and fix hidden state units
in 2D space and then arranges the positions of words using force
simulation. Though this design shows hidden states projections
and words in the same space, this design is not scalable for a large
number of states or words where details are messed with numerous
links.
Another alternative design visualizes word clusters as word
clouds and positions them inside the ring with force-directed layout
(Fig. 7(b)). This design handles bi-connections between hidden
states and words better than the previous one, but it is difficult to use
in general comparative analysis (R5). The experts also felt difficult

Figure 7: Alternative layouts of the main view. (a) a design that
combines t-sne projection of hidden states with force-directed layout
of words. (b) a design that visualizes hidden state clusters as arcs
and word clusters as word clouds.

to associate the circular arcs with hidden states, which introduces
confusion to users.
6.2

Interactions

To assist user’s exploration, we develop rich interactive features
through the main view to the detailed view for RNNVis.
6.2.1

Detail on Demand

To avoid overloading users with too much information, all interaction
designs are guided by visual information seeking mantra: “overview
first, zoom and filter, then details-on-demand” [40].
When users select a hidden state cluster, edges to its highly correlated word clusters are highlighted. Meanwhile, these word clouds
are expanded to show the complete word collections. Other word
clouds are contracted to only display a few representative words.
Except for cluster-level interaction, detailed statistics of individual hidden state units and words are displayed in the detail view
upon interaction. When users select a hidden state unit, the activation distribution of a hidden state unit on different words will be
displayed as box plots, as shown in Fig. 1(E). Users are also allowed
to click on individual word w in the word cloud and see the model’s
expected response s(xx(w)) visualized as a heat map in the memory
chips visualization.
In sequence analysis, users are allowed to zoom in and out certain parts of the sequence with brushing. Similar to the co-cluster
layout, the links between a sequence node and the memory chips are
highlighted when users click on the node.
6.2.2

Comparative Interactions

RNNVis uses three levels of comparison to help experts understand
RNNs (R5), namely, hidden state unit and word-level comparison,
sentence-sentence comparison and model-model comparison.
For word-level comparison, users can interactively select two or
three words in the word clouds, and compare the distributions of
model’s responses to these words in an overlay manner. Since a hidden state vector may have hundreds of dimensions, the dimensions
can be sorted according to the expected response to a selected word
s(xx(w)) j for a clearer comparison. For example, as shown in Fig. 5,
the dimensions are sorted according to the word “he”. We can see
that “he” and “she” result in a similar response distribution, while
the response distribution of “for” is very different from “he”. This
condition suggests that the RNN model has learned to distinguish
pronouns and prepositions. Similar comparisons can be performed
for hidden states using a side-by-side box plots layout.
To compare how an RNN treats different sentences, we provide a
side-by-side comparison as shown in Fig. 10. Users can easily ana-

A

response
2.5
2.0
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
-2.0

they

VERB
NOUN
PRON
ADJ
ADV
ADP
CONJ
DET
NUM
PRT
X
.

he
is

she
closed 're

while

while and with for

because that in if

was

and with for

where

because that in if
where

more

an

her
those several ourmany

other no a

his some

such

these itsany
mr.

most exchange

about back due

0

20

40

60

80 100 120 140 160 180 200

when

words

has
$
such
rates
increase
sell
buy
share

make
buy
sell
say
operations
stock
average
bush
-2

-1

0

1

2

Dim: 48

earlier

overoff

only down up out N
make chairman
very used far

two

stock fund
bond
program
futures
plans
him rates cents
money rate
prices industry
markets income
management

bill quarter system
shares

bank plan growth
share house issue
past yield market
government number
own year time board

billion

economic both american

good

vice cash big net

friday next trade well
japan oil all how

-2 -1 0 1 2 3

Dim: 123

(1)

like
as
since officer by

more

an

her
those several ourmany

other no a

his some

such

(2)
these itsany
mr.

get

B
C

where

words

added made report $

dim

when

through among

when

through among

like
as
since officer by

million

international

city
lower high yesterday
president
public

r

what

then so

-1.5

0

1.5

still

does
will co.
can would
'
according
could & n't

may
corp. should
now even
to not

stock fund
bond
program
futures
plans
him rates cents
money rate
(3)
prices industry
markets income
management

bill quarter system
shares

does then so

still

will co.
can would
'
according
could & n't

may
corp. should
now even

(4)

to not

Figure 8: Co-cluster visualization of the last layer of LSTM-Small. (A): The co-Clustering layout with words colored according to POS tags. (B):
The detail view showing model’s responses to “when” and “where” in overlay manner. (C): Words interpretations of the 48th and 123rd cell states.

lyze whether different sentence contexts may dramatically influence
the information inside the hidden states.
As shown in Fig. 9, a side-by-side layout of the main view is
employed to compare different models.
7 E VALUATION
In this section, we first demonstrate how RNNVis can be effectively
used to help experts understand and explore the behavior of different
RNN models through two case studies. Then we present the feedback
gathered through one-to-one interviews with the domain experts.
Table 1: Configuration and performance of RNN models for language
modeling (lower perplexity represents better performance).

Model
Model
Layer
LSTM-Small
2
LSTM-Medium
2
LSTM-Large
2
RNN
2
GRU
2

Size
200
600
1500
600
600

Perplexity
Validation Set Test Set
118.6
115.7
96.2
93.5
91.3
88.0
123.3
119.9
119.1
116.4

7.1 Case Study: Language Modeling
The first case study is collaborated with expert E1 to understand
and compare different RNN models that were trained for language
modeling, which is a basis of machine translation and image captioning. Note that for simplicity, the expected response s(xx) that we
used in this case study are estimated on hidden state observations in
the test set, though the training set or the validation set may also be
evaluated.
A language model assigns probabilities to sequences. Its target
is to predict the conditional probability of the next word given the
sequence of previous words P(w(t) | w(0) · · · w(t−1) ). In this case
study, we used the Penn Tree Bank (PTB) dataset [29], which is a
widely used benchmark dataset for language modeling. We trained
a vanilla RNN, a GRU, and three LSTMs on the PTB dataset. The
detailed parameter settings and performance of these models are
shown in Table 1.
An ablation of an LSTM. We started with an LSTM model
(LSTM-Small in Table 1) to introduce the basic usage of RNNVis to
expert E1. The number of clusters was set to 10, and the bipartite

links with absolute weight less than 20% of the maximum value
were filtered. The visualization shows the cell states of the last layer.
At the first glance (Fig. 8A), the expert noticed that words with
similar functions were likely to be clustered together. For example,
prepositions like “with”, “for”, “by” and “as” were grouped into the
word cloud (1), and modal verbs like “would”, “could” “should” and
“will” were clustered to the word cloud (4). After focusing on the
word cloud (1), the first and fourth memory chips are highlighted. As
mentioned in Sect. 6.1.2, word clouds visualize a batch of words as
a summary interpretation of hidden state clusters. The hidden units
in the first and fourth memory chips can be regarded to be able to
capture the information of prepositions (R2). The expert then turned
on the POS mode to color words according to their most frequent
POS tags to further evaluate the quality of the word clusters. The
result clearly showed that LSTM-Small was able to distinguish a
large number of nouns and prepositions from other words. Although
the word cloud (2) was messed with adjectives, determinants, and
pronouns (especially adjectival possessive pronouns like “her”, “his”,
“our” and “its”), these words actually contain similar decorative
semantics for specifying the successive nouns.
To further validate his hypothesis on the model’s ability of recognizing POS tags, the expert clicked on several pairs of words
(e.g., “those” and “these”, and “his” and “her”) and compared the
detailed distribution of model’s responses shown in Fig. 8B ( R4).
The results showed that the distributions of words in each pair are
only slightly different from each other, indicating the model can
capture their similar linguistic functions. Then the expert explored
among detailed interpretations of specific hidden units by selecting
his interested cells on the memory chips. The box plots in the detail
view (Fig. 8C) showed that both the 123rd and 48th dimensions
can catch information of verbs like “buy” and “sell”, although their
information is stored with opposite signs in cell states. Since the estimated expected responses ŝ(xx) were directly computed from hidden
states observations, the similarity between word pairs was a result
of model’s hidden state updating mechanism rather than similarity
in word embedding alone. This suggests that LSTM-Small can well
recognize grammatic functions of words, although it only has 200
cell states, and it was not intentionally trained for POS tagging.
Comparison between vanilla RNN and Gated RNNs. Despite the interesting findings on the interpretable representations
of LSTMs, the expert was also curious about whether these visualizations can reveal the difference between the vanilla RNN and other

A

B

D
1

to

-1

-0.6

1

0.6

oﬀer

you

he
him average
i

E

F
no any

0.6

to

we

C

their

our '

-1

-0.6

1

0.6

foreign

those a its these his

vice very

can iswas were
just 're or does

are
even do
has have
corp.

court

offer funds costs
bid cents bonds

firm
day

according

rates way plan

prices rate time
companies

assets

investors we

bush

analysts

issue bank co.
number plan
day
sale way
firm board offer
corp. rate group

say
how
what in that
but and on

a

a

products yen

<eos>
byatwhile
under as among

-1

-0.6

public federal

1

0.6

G

stock own u.s.

who up may
willagothere
now if to

1.0
0.0

50

oﬀer

100

150

200

250

300

350

400

450

500

550

oﬀer(sentence)

600

dim

-1
1

u.s. general

friday

1.0

recent operating

said yen up

0.0

end report
made
about

-1.0

executive

-2.0

-0.6

0

50

100

oﬀer

0.6

H

less more
sales chief $
net cash most

2.0

0.6

chairman

own &
oil
earlier
early new
current

response

of

0

-0.6

of

so very

-2.0

-1
1

exchange bush
third business
city yesterday

-1.0

number

few

number

good foreign net
major n't financial
responseinternational

japanese

state

money house

economic federal
american both

2.0

mr. it

which i

since

if

bill

months

VERB
NOUN
PRON
ADJ
ADV
ADP
CONJ
DET
NUM
PRT
X
.

n't 's
an

oﬀer

VERB
did
NOUN
PRON
during
like<eos>over in of
ADJ
underby including
against
ADV
ADP
mr. more a our
his
CONJ
her
their '
the less anno
its several
DET
NUM
end
report
PRT
such
buy
X
because sold year
.

150

200

250

300

350

over

out
400

450

500

550

600

dim

oﬀer(sentence)

Figure 9: Comparison of an RNN and an LSTM. Left (A-C): co-cluster visualization of the last layer of RNN (Table 1). Right (D-F): visualization of
the cell states of the last layer of LSTM-Medium. Bottom (GH): two models’ responses to the same word “offer”. The orange line represents the
actual response in the sentence.

-0.6

0.4

0.6

though

-0.4

-0.6

0.4

0.6

-0.4

B1

A1

-0.6

after new $ though
big not i free

full never why
're
she went if ca
work n't away price
eat

we

help

B2

every

C1
C2

chicken both where

experienceplaces
still 'mnexthelpful
around
love
bar should then
well like year
say awesome
years

i

challenge

helpful

1 http://www.yelp.com/dataset

-0.4

the

Case Study: Sentiment Analysis

Besides language modeling, we also conducted another case study
to understand RNN’s behavior in sentiment analysis with expert
E2. The dataset we used is retrieved from Yelp Data Challenge
1 . The Yelp dataset contains over 4 million reviews for restaurants
with labels from one to five. For simplicity, we pre-process the five
labels into only two labels by mapping one and two to “negative”
and four and five to “positive”. The data entries with label three
are not included. For simplicity, we only used a small subset of 20

0.6

though

7.2

0.4

food

gated RNNs (e.g., LSTM and GRU).
The expert first compared an RNN with an LSTM (LSTMMedium) using the compare mode of the main view (see Fig. 9,
R5). The memory of the RNN model (B) has significantly higher
saturation than that of the LSTM (E). This indicates the expected
response strength of the RNN tends to be much stronger. The expert
then added a sentence: “The company said it planned to offer a number of common shares in exchange,” for the two models to run and
compare their response histories (Fig. 9C and D). The colored part of
each bar chart glyph of C is much larger than that of D, indicating the
RNN updates its hidden memories more intensively than the LSTM.
Or, the LSTM’s response to inputs is generally much sparser than
the RNN (see Fig. 9). LSTM’s lazy behavior in updating cell states
might contribute to its long-term memory advantage over RNN. The
idea was further validated on detailed model response distribution in
the detail view, where the response distribution of the RNN (G) is
more dispersive and unstable than that of the LSTM (H). Such evidence also provided empirical explanations of the widely accepted
claim that LSTMs can handle long-term dependencies better than
vanilla RNNs.
The expert also compared GRU with LSTM and RNN. Although
the response distribution of the GRU model is dispersive, the shape
and range of the distribution of hidden states’ response are closer to
those of the LSTM. This finding may serve as an empirical explanation of the similar performances of GRUs and LSTMs, which agrees
with a recent empirical result [22].

A2

put different
friendly customer

Figure 10: Comparison of two sentences on an GRU. A1 and A2:
two evaluated sentences, focusing on the word “though”. B1 and B2:
hidden unit clusters. C1 and C2: word clusters. B1 is positively related
to negative words, and B2 is positively related to positive words.

thousands reviews with length less than 100 words with 0-1 labels.
The small dataset was split into train/validation/test set with 80/10/10
percentages. We trained a single layered GRU with 50 cell states
on the train set and achieved accuracies of 89.5% and 88.6% on the
validation and test set.
Sentiments of words. The expert first explored how the trained
GRU handled the sentiment of each word. Expecting to have hidden
state clusters that respectively catch positive, negative information,
the expert set the cluster number to two. As shown in Fig. 10, the
co-clustering visualization clearly represented two word clouds that
present different sentiments. For example, the upper word cloud
contained negative words such as “never” and “not”, and the lower
word cloud contained positive ones such as “love”, “awesome” and
“helpful”. From the color of the link, we can infer that the hidden
units in the upper memory chip are positively related to negative
words, while the lower memory chip reacts positively to positive
words. Note that such relation is coupled, i.e., hidden states use
positive and negative values to encode opposite sentiments.
Sentiments in contexts. The expert was also very interested in
how the GRU dealt with subtle sentiment changes in a sentence (R3).
Then the expert used the sentence-level comparison visualization to
investigate how different context may influence the sentiment of the

A

response
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0
-1.2

0

5

10

15

20

25

worst

30

35

40

45

50

C

someone decor

instead though 'll
cold run $ used walked

priced nothing
n't rude not bad

'' never
left worst
maybe
ok ? if told
no disappointed
however ago
looked to phone

dim

asked wrong
closed
tasted friend

excellent

super
enjoyed
wonderful tasty
awesome helpful a
delicious great

B

words
priced
worst
part
prices
highly
excellent
perfect
delicious
-0.5

0.0

0.5

r

1.0

Dim: 36

-1

0

1

excellent

!

best fantastic
favorite amazing
easy perfect
highly reasonable
us

-1.0

often

cool always

Figure 11: The GRU model trained on the balanced dataset. A: the
model’s responses to word “worst” and “excellent”. B: most responsive
words of the 36th hidden unit. C: co-cluster visualization of the model.

same word. The expert compared two sentences: “I love the food,
though the staff is not helpful” and “The staff is not helpful, though I
love the food”. As shown in Fig. 10, the updated information of the
first hidden state cluster of “though” in sentence (a) is much larger
than in sentence (b), denoting a larger response towards negative
sentiment. Note the previous information of the first sentence (at
word “food”) has more positive sentiment than that of the second
sentence (at word “helpful”). These results suggest that the GRU had
learned to treat the sentiment of the same word differently according
to different contexts.
Diagnosing sentiment models. Though the visualization in
Fig. 10 indicates the GRU model is capable of distinguishing different sentiments, many negative words such as “bad” and “worst”,
which can be found in the dataset, are not shown in the visualization.
The expert suspected that the model might have different performance in positive and negative reviews. We then examined the code
base and found that the dataset was unbalanced, with approximately
3:1 positive to negative reviews, which caused the uneven performance of the model. After over-sampling to achieve a balanced
dataset, a re-trained GRU model achieved accuracies of 91.52%
and 91.91% on the validation set and test set. The visualization of
the new model (Fig. 11) shows more words with strong sentiments,
such as “rude”, “worst” and “bad” in the negative word cloud, and
“excellent”, “delicious” and “great” in the positive word cloud. By
clicking words “worst” and “excellent”, the expert compared the
response distributions of the two words in the detail view (A). We
can see that they have nearly opposite response patterns (i.e., the
dimensions with negative response to “worst” always respond positively to “excellent”). The expert commented, “the visualization
provides clues on what’s going on inside the model” and “allows me
to explore the model’s behavior from different perspectives”.
7.3

Expert Reviews

To evaluate the effectiveness and usefulness of our system, we conducted one-to-one interviews with four domain experts (P1, P2, P3
and P4). P1 and P2 are machine learning researchers who have
published papers on prestigious conferences in related fields. P1
specializes in sentiment analysis while P2 is interested in NLP and
transfer learning. P3 and P4 are graduate students with knowledge
in both machine learning and visualization. We first introduce the
proposed system and the used method through the tour guide and
the presentation of the first case study. Then we asked the experts
to explore and compare the two GRU models discussed in Sect. 7.2.
Finally, we asked them to guess which model has better performance
and state their findings and suggestions. The result and feedback are
summarized as follows.
Effectiveness. After exploring the two models using RNNVis,
all the interviewees made correct guess on the performances of the
models. They can also clearly state that the second model can capture more obvious negative words than the first one, which indicates
the second model is more likely to provide better predictions on the

sentiments of reviews. Besides, they all agreed on the effectiveness
and usability of the system. P2 commented that given the success
of neural models, a recent concern in the community is their interpretability. RNNVis provides him with the possibility to flexibly
explore RNNs’ hidden behavior, and easily validate his hypothesis
on these models. The comparison of vanilla RNNs and LSTMs
reveals the long-term dependency characteristics of LSTMs intuitively and convincingly. P1 added that this tool could help her make
more sense on the model: “it provides intuitive information on what
is happening in sentiment models, which is helpful for debugging
compared with merely looking at training logs”. P3 mentioned that
“tuning deep learning models is like metaphysics to him, and this
visualization can be very helpful when introducing the advantages
of RNNs to non-experts”.
Cognitive Burdens. All the experts agreed that they could fully
understand the visual encoding of the system after our introduction and the guidance tour. They agreed that the cognitive load
is acceptable and the interaction is fluent. P4 commented that the
visualization design is quite aesthetic and he enjoys using it for
exploration. However, at the beginning of the interview, P2 has
mistakenly regarded different memory chips as different layers of
the network. He also suggested us to design a simple document for
users to learn the techniques behind the system quickly.
Suggestions. The experts have also provided valuable suggestions on improving our work. P1 and P3 suggested that we make
the system programmable so that other visualizations such as loss
curves and gradient distributions can be shown during the training
process. P4 also suggested we should add model’s inference results during the sequence visualization as additional information.
P2 mentioned the need for a simpler visualization for explaining
RNNs’ behavior during presentation. P1 and P2 pointed out that we
can explore more applications such as machine reading and more
advanced RNN-based models such as attention mechanisms and
memory networks.
8 C ONCLUSION
In this paper, we presented a visual analytic method for understanding and diagnosing RNNs for text applications. A technique based
on RNN’s expected response to inputs is proposed to interpret the
information stored in hidden states. Based on this technique, we
designed and implemented an interactive co-clustering visualization
composed of memory chips and word clouds, which allows domain
users to explore, understand and compare the internal behavior of
different RNN models. Our evaluation, including two case studies
and expert interviews, demonstrated the effectiveness of using our
system to understand and compare different RNN models and also
verified the completeness of design requirements.
To further improve our proposed visual analytic system, RNNVis,
we plan to deploy it online, and improve the usability by adding
more quantitative measurements of RNN models. Considering the
proposed technique is based on the discrete input space of texts, RNNVis is only suitable to analyze RNN-based models for texts. The
analysis of RNN models for audio applications requires further efforts to construct interpretable representations. A current bottleneck
for RNNVis is the efficiency and quality of co-clustering, which
may results in delays during interaction. Other potential future work
includes the extension of our system to support the visualization
of specialized RNN-based models, such as memory networks or
attention models.
ACKNOWLEDGMENTS
The authors wish to thank anonymous reviewers for their constructive comments. The authors also wish to thank the domain experts
who participated in the studies and Miss Yuehan Liu who helped
record the video. This work was supported in part by the National
973 Program of China (2014CB340304).

R EFERENCES
[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S.
Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. J. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser,
M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. G. Murray,
C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar,
P. A. Tucker, V. Vanhoucke, V. Vasudevan, F. B. Viégas, O. Vinyals,
P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensorflow:
Large-scale machine learning on heterogeneous distributed systems.
CoRR, abs/1603.04467, 2016.
[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by
jointly learning to align and translate. CoRR, abs/1409.0473, 2014.
[3] S. Bird, E. Klein, and E. Loper. Natural Language Processing with
Python. O’Reilly Media, Inc., 1st edition, 2009.
[4] M. Brooks, S. Amershi, B. Lee, S. M. Drucker, A. Kapoor, and P. Y.
Simard. Featureinsight: Visual support for error-driven feature ideation
in text classification. In Proc. Visual Analytics Science and Technology
(VAST), pages 105–112. IEEE, 2015.
[5] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio. On the
properties of neural machine translation: Encoder-decoder approaches.
In Proc. 8th Workshop on Syntax, Semantics and Structure in Statistical
Translation, pages 103–111, 2014.
[6] J. Choo, H. Lee, J. Kihm, and H. Park. ivisclassifier: An interactive
visual analytics system for classification based on supervised dimension
reduction. In Proc. Visual Analytics Science and Technology (VAST),
pages 27–34. IEEE, 2010.
[7] J. Chung, Ç. Gülçehre, K. Cho, and Y. Bengio. Empirical evaluation
of gated recurrent neural networks on sequence modeling. CoRR,
abs/1412.3555, 2014.
[8] I. S. Dhillon. Co-clustering documents and words using bipartite spectral graph partitioning. In Proc. 7th SIGKDD Int. Conf. on Knowledge
Discovery and Data Mining, pages 269–274. ACM, 2001.
[9] J. Donahue, L. A. Hendricks, M. Rohrbach, S. Venugopalan, S. Guadarrama, K. Saenko, and T. Darrell. Long-term recurrent convolutional
networks for visual recognition and description. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 39(4):677–691, 2017.
[10] A. Dosovitskiy and T. Brox. Inverting visual representations with convolutional networks. In Conf. Computer Vision and Pattern Recognition
(CVPR, pages 4829–4837. IEEE, 2016.
[11] M. Drk, N. H. Riche, G. Ramos, and S. Dumais. Pivotpaths: Strolling
through faceted information spaces. IEEE Transactions on Visualization and Computer Graphics, 18(12):2709–2718, 2012.
[12] D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higherlayer features of a deep network. Technical Report 1341, University of
Montreal, 2009.
[13] P. Fiaux, M. Sun, L. Bradel, C. North, N. Ramakrishnan, and A. Endert.
Bixplorer: Visual analytics with biclusters. Computer, 46(8):90–94,
2013.
[14] A. G. Forbes, S. Savage, and T. Höllerer. Visualizing and verifying directed social queries. In Workshop on Interactive Visual Text Analytics.
IEEE, 2012.
[15] M. Gleicher, D. Albers, R. Walker, I. Jusufi, C. D. Hansen, and J. C.
Roberts. Visual comparison for information visualization. Information
Visualization, 10(4):289–309, 2011.
[16] I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT
Press, 2016. http://www.deeplearningbook.org.
[17] A. Graves, A. Mohamed, and G. E. Hinton. Speech recognition with
deep recurrent neural networks. In Int. Conf. Acoustics, Speech and
Signal Processing, pages 6645–6649. IEEE, 2013.
[18] K. Greff, R. K. Srivastava, J. Koutnk, B. R. Steunebrink, and J. Schmidhuber. LSTM: A search space odyssey. IEEE Transactions on Neural
Networks and Learning Systems, PP(99):1–11, 2016.
[19] K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay,
M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages
1693–1701. Curran Associates, Inc., 2015.
[20] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural
Computation, 9(8):1735–1780, 1997.
[21] E. Hoque and G. Carenini. Convis: A visual text analytic system

[22]

[23]
[24]

[25]
[26]

[27]

[28]

[29]

[30]

[31]

[32]

[33]

[34]

[35]

[36]

[37]

[38]

[39]

[40]

[41]
[42]

[43]

for exploring blog conversations. In Proc. 16th Eurographics Conf.
Visualization, pages 221–230. Eurographics Association, 2014.
R. Józefowicz, W. Zaremba, and I. Sutskever. An empirical exploration
of recurrent network architectures. In Proc. 32nd Int. Conf. Machine
Learning, volume 37 of ICML’15, pages 2342–2350, 2015.
D. Jurafsky and J. H. Martin. Speech and Language Processing (2nd
Edition). Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 2009.
A. Kapoor, B. Lee, D. Tan, and E. Horvitz. Interactive optimization
for steering machine classification. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, pages 1343–
1352. ACM, 2010.
A. Karpathy, J. Johnson, and F. Li. Visualizing and understanding
recurrent networks. CoRR, abs/1506.02078, 2015.
J. Li, X. Chen, E. Hovy, and D. Jurafsky. Visualizing and understanding neural models in NLP. In Proc. NAACL: HLT, pages 681–691.
Association for Computational Linguistics, 2016.
M. Liu, J. Shi, Z. Li, C. Li, J. Zhu, and S. Liu. Towards better analysis of deep convolutional neural networks. IEEE Transactions on
Visualization and Computer Graphics, 23(1):91–100, 2017.
S. C. Madeira and A. L. Oliveira. Biclustering algorithms for biological
data analysis: a survey. IEEE/ACM Transactions on Computational
Biology and Bioinformatics, 1(1):24–45, 2004.
M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini. Building a
large annotated corpus of english: The penn treebank. Computational
Linguistics, 19(2):313–330, 1993.
T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur.
Recurrent neural network based language model. In 11th Annua. Conf.
Int. Speech, pages 1045–1048, 2010.
W. J. Murdoch and A. Szlam. Automatic rule extraction from long
short term memory networks. In Proc. 5th Int. Conf. on Learning
Representations (ICLR), 2017.
J. G. Paiva, L. Florian, H. Pedrini, G. Telles, and R. Minghim. Improved
similarity trees and their application to visual data classification. IEEE
Transactions on Visualization and Computer Graphics, 17(12):2459–
2468, 2011.
S. Petrov, D. Das, and R. T. McDonald. A universal part-of-speech
tagset. In Proc. 8th Int. Conf. Language Resources and Evaluation,
pages 2089–2096, 2012.
P. E. Rauber, S. G. Fadel, A. X. Falcão, and A. C. Telea. Visualizing
the hidden activity of artificial neural networks. IEEE Transactions on
Visualization and Computer Graphics, 23(1):101–110, 2017.
P. E. Rauber, S. G. Fadel, A. X. Falco, and A. C. Telea. Visualizing
the hidden activity of artificial neural networks. IEEE Transactions on
Visualization and Computer Graphics, 23(1):101–110, 2017.
D. Ren, S. Amershi, B. Lee, J. Suh, and J. D. Williams. Squares:
Supporting interactive performance analysis for multiclass classifiers.
IEEE Transactions on Visualization and Computer Graphics, 23(1):61–
70, 2017.
M. T. Ribeiro, S. Singh, and C. Guestrin. ”why should i trust you?”:
Explaining the predictions of any classifier. In Proc. 22nd SIGKDD Int.
Conf. on Knowledge Discovery and Data Mining, pages 1135–1144.
ACM, 2016.
X. Rong and E. Adar. Visual tools for debugging neural language
models. In ICML 2016: Workshop on Visualization for Deep Learning,
2016.
Q. Shen, T. Wu, H. Yang, Y. Wu, H. Qu, and W. Cui. NameClarifier:
A visual analytics system for author name disambiguation. IEEE
Transactions on Visualization and Computer Graphics, 23(1):141–150,
2017.
B. Shneiderman. The eyes have it: a task by data type taxonomy for
information visualizations. In Proc. IEEE Symp. Visual Languages,
pages 336–343, 1996.
V. Singh and S. Pathak. Feature selection using classifier in high
dimensional data. CoRR, 2014.
H. Strobelt, S. Gehrmann, B. Huber, H. Pfister, and A. M. Rush. Visual
analysis of hidden state dynamics in recurrent neural networks. CoRR,
abs/1606.07461, 2016.
M. Sun, C. North, and N. Ramakrishnan. A five-level design framework
for bicluster visualizations. IEEE Transactions on Visualization and
Computer Graphics, 20(12):1713–1722, 2014.

[44] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning
with neural networks. In Proc. 27th Int. Conf. Neural Information
Processing Systems, NIPS’14, pages 3104–3112, 2014.
[45] J. Talbot, B. Lee, A. Kapoor, and D. S. Tan. Ensemblematrix: interactive visualization to support machine learning with multiple classifiers.
In Proc. SIGCHI Conf. on Human Factors in Computing Systems, pages
1283–1292. ACM, 2009.
[46] D. Tang, B. Qin, and T. Liu. Document modeling with gated recurrent
neural network for sentiment classification. In Proc. Conf. Empirical
Methods in Natural Language Processing, pages 1422–1432, 2015.
[47] Z. Tang, Y. Shi, D. Wang, Y. Feng, and S. Zhang. Memory visualization
for gated recurrent neural networks in speech recognition. CoRR,
abs/1609.08789, 2016.
[48] S. T. Teoh and K.-L. Ma. Starclass: Interactive visual classification
using star coordinates. In SDM, pages 178–185. SIAM, 2003.
[49] Y. Wang, J. Li, F. Nie, H. Theisel, M. Gong, and D. J. Lehmann. Linear
discriminative star coordinates for exploring class and cluster separation of high dimensional data. Computer Graphics Forum (EuroVis),
2017.
[50] K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S.
Zemel, and Y. Bengio. Show, attend and tell: Neural image caption
generation with visual attention. In Proc. 32nd Int. Conf. Machine
Learning, pages 2048–2057, 2015.
[51] P. Xu, N. Cao, H. Qu, and J. Stasko. Interactive visual co-cluster
analysis of bipartite graphs. In IEEE Pacific Visualization Symp., pages
32–39, 2016.
[52] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Proc. 13th European Conf. Computer Vision, pages
818–833, 2014.

A PPENDIX A M ODEL D EFINITIONS
A.1 Long Short-Term Memory
LSTM was designed by Hochreiter & Schmidhuber [20] to deal
with the training difficulties of RNN. Except for hidden state h (t) , an
LSTM maintains another memory vector c (t) called cell state. Also,
LSTM uses input gate i (t) , forget gate f (t) and output gate o (t) to
explicitly control the update of h (t) and c (t) . The three gate vectors
i (t) , f (t) , o (t) ∈ Rn are computed using:
W i h (t−1) +V
V i x (t) )
i (t) = σ (W
W f h (t−1) +V
V f x (t) )
f (t) = σ (W
o(t)

o

h(t−1)

W oh
= σ (W

(9)

x(t)

V ox )
+V

and cell state and hidden state are updated by:
W c h (t−1) +V
V c x (t) )
c̃c(t) = tanh(W
c (t) = f (t)

c (t−1) + i (t)

h(t)

c(t)

h

o(t)

=o

c̃c(t)

(10)

c

where c̃c(t) is called candidate cell state, and W ∗ and V ∗ are trainable weight matrices. With the sigmoid function, the values of
i (t) , f (t) , o (t) have range between 0 and 1, representing how much
we update the c (t) , forget the c (t−1) and output to h (t) . Such an architecture was argued to have the advantage to maintain information
over a relatively long term.
A.2 Gated Recurrent Unit
GRU is a simpler model proposed by Cho et al. [5]. A GRU only uses
a single hidden state vector h (t) . Behaving like a simpler variation
of an LSTM, a GRU uses two gates, update gate z(t) and reset gate
r (t) to control the update of hidden memory:
W z h (t−1) +V
V z x (t) )
z (t) = σ (W

(11)

W r h (t−1) +V
V r x (t) ).
r (t) = σ (W
(t)

Then a candidate hidden state h̃h is computed using reset gate
vector before updating hidden state:
h̃h

(t)

W h h (t−1) +V
V h (rr (t)
= tanh(W

h (t) = (1 − z (t) )

h (t−1) + z (t)

x (t) ))
(t−1)

h̃h

(12)

.

The computation of GRU can be thought as updating the hidden
states vector h (t) by interpolating between previous h (t−1) and a
(t)
candidate h̃h with ratio z (t) .
A.3 Multi-layer Models
To increase representation capability, the aforementioned RNN models can be easily extended to multi-layered version by simply stack(t)
(t)
ing. A k-layered RNN associates k hidden states vector h 1 , · · · , h k
(t)

for each layer, where each h l ∈ Rn . At step t, layer l takes the
(t)

(t)

output h l−1 of previous layer and update its hidden states vector h l
using:
(t)
(t−1)
(t)
hl = f (W
W l hl
V l hl−1 )
+V
(13)
(t)

where we use h0 = x(t) the model input at step t for the first layer.
LSTMs and GRUs can also be extended to multi-layered models
with similar procedure. But we will neglect these details considering
the page limits.

A PPENDIX B

C ASE S TUDY:
SPEARE

T HE L ANGUAGE

S HAKE -

OF

We demonstrate how new models and new datasets can be added to
RNNVis, and how the system can be used to validate hypotheses
related to the behavior of hidden states.
Training a language model on plain texts. RNNVis provides a
pre-defined pipeline to process plain texts and build up a tokenized
data set. Users can easily modify configuration files to specify the
vocabulary size of the new dataset and the hyper parameters of
the new model (e.g., the type of RNN, the number of layers and
the size of each layer). In this case study, we used a collection of
Shakespeare’s work2 (containing one million words) to build up a
language modeling dataset with a vocabulary size of 15 thousand.
The dataset is split into 80/10/10 train/valid/test sets. We trained
three two-layer LSTM models with different sizes per each layer
(see Table 2). For the validation of the proposed hypotheses in
the following paragraphs, we only illustrate the results on LSTMMedium, though the presented phenomenons can be found in all
three models.
Table 2: Configuration and performance of LSTM models trained on
the Shakespeare corpus (lower perplexity represents better performance).

Model
Model
Layer
LSTM-Small
2
LSTM-Medium
2
LSTM-Large
2

Size
256
512
1024

Perplexity
Validation Set Test Set
154.0
153.3
135.8
134.9
130.4
129.7

on
near seen
without
god after
between
' bid with
mark o'er from to let at before
about for than
pardon in

further these theiray even
my thy
full very so the whose our
being his an such your
three both
new many

get

have can were do be art
was
's
hath wilt am go has are
dost

look need

shalt <eos>c will n't if ne'er ;
must
should does
or i 'll did till : doth
shall as may
ye ere could since
ha

VERB
NOUN
PRON
ADJ
ADV
ADP
CONJ
DET
NUM
PRT
X
.

proud

master sir mine
out down
forth
another up all off ogone

poor

lost

right less ill mad
known sweet
good

woman pleasure law
lady lords duke sar
earth
highness
. friend
maid king queen gentleman
letter son sea crown
word husband
charge

meet sit kill swear help
play welcome talk fear
warrant answer
prithee think
call fight say flycry
marry
loves seek pray lose return see
well means half now still
then worth said't! lie heard, far

light

him says
back been
but doubt
lives pity wrong
looks heaven
what myself

-2

0

2

Figure 12: The LSTM-Medium model trained on the Shakespeare
corpus. The POS tags of ancient words may be incorrect because the
NLTK POS tagger is trained on modern English corpus.

A mixture of the old and the new. Shakespeare’s language is
a mixture of the old and the new. For example, “thou” and “you”,
“hath” and “has”, and “ay” and “yes” are mixed in his plays. Thus,
one interesting hypothesis that we want to validate is whether the
language model can capture the similar meaning of these words
despite their different spellings.
We began with the co-cluster visualization of the LSTM-Medium
model (Table 2), which is shown in Fig. 12. We quickly found that
2 The

dataset is downloaded
http://www.gutenberg.org/ebooks/100

from

the

Project

Gutenberg:

“hath”, “dost” and “art” are settled in the same word cloud with
“have”, “do” and “are”, which indicates the model is able to learn
the similar usage of these auxiliary verbs, no matter they are from
ancient or modern English. To further validate the model’s ability in
learning the similarity between old and new words, we compared
a variety of old-new word pairs in Fig. 13. All the new words
have similar response distributions as their old versions, except for
“thou” and “you”. After consulting an expert in classic literature, we
learned that such anomaly may result from their different usages in
Shakespeare’s time. Although “thou”, “ye” and “you” are all Early
Modern English second person pronouns, “thou” is the singular form
and often used in informal cases, while “ye” and “you” are plural
forms and are considered to be formal and polite.
Comparing different layers in one model. From the visualization research of CNNs [52], we learned that CNNs use multiple
layers to capture different levels of structure in images. Initial layers
capture small features such as edges and corners, while the layers
close to the output learn to identify more complex structures which
formulate different classes. Little research has explored how features in RNNs are composed through multiple layers. A natural
hypothesis is that the initial layers learn abstract representations,
while the latter layers learn more task-specific representations. Next
we illustrate how RNNVis can be used to compare the response of
different layers in the LSTM model.
Comparing the hidden state clusters of the two layers, we found
that the sizes of memory chips of the layer 0 are more balanced
than those of the layer 1. We also found that in layer 1, the color
within each word cloud is more consistent. In another word, the
quality of the word clouds in layer 1 is generally higher that those
in layer 0. This indicates that the layer 0 is “fuzzier” in treating the
usage of different words, while the layer 1 learns a “clearer” way
in identifying the different functions of words. Then we clicked
on the same word “thou” of each side. As shown in Fig. 14C1,
the color of memory chips of layer 1 has lower saturation than that
of the layer 0, indicating the hidden states of layer 1 have sparser
response to the word “thou”. Similar phenomenon could be found
for other words. For language modeling, such sparsity is helpful in
the last layer, where the output is directly projected through softmax
to a large space containing thousands of words. To summarize,
the comparative visualization of the two layers provides a way of
explaining how multi-layer RNNs utilize multiple layers of hidden
states to formulate the learned representations. That is, the model
uses the first layer to construct abstract representations, while uses
the last layer to formulate more task-specific representations.

response

response

2.5

2.0
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
-2.0

0

50

100

150

ye

200

250

300

350

400

450

500

dim

0

you

50

100

150

thou

response

200

250

300

350

400

450

500

250

300

350

400

450

500

250

300

350

400

450

500

dim

you

response
2.5

2.0

2.0

1.5

1.5

1.0

1.0

0.5

0.5
0.0
-0.5

0.0
-0.5
-1.0
-1.5
-2.0

3.0
2.5
2.0
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
-2.0
-2.5

0

50

100

150

doth

200

250

300

350

400

450

500

dim

0

does

50

100

150

hast

response

200

dim

have

response

2
1
0
-1
-2
-3
-4
0

-1.0
-1.5
-2.0

50

art

100

150

200

are

250

300

350

400

450

500

dim

2.0
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
-2.0
-2.5
-3.0
-3.5
-4.0
-4.5

0

50

ay

100

150

200

yea

yes

Figure 13: The response distributions to ancient-modern word pairs of the LSTM-Medium model.

dim

A. Layer 0 - Hidden State

VERB
NOUN
they o thou
PRON
to
are
ADJ
'll
let has
ADV
had is were if
ADP
shalt
as or into
by on
: ' ;
CONJ
off and <eos>
,
DET
in at upthantill ne'erfrom
NUM
so
PRT read prithee stand
farewell yes nay
X
do be
am go why
run lay sit
come
.

i

B. Layer 1 - Hidden State

c

every

alas

your

a

thyour
use near

play seen see think
lose
wear beseech lost
sent please all
talk serve

no

state war sword
lettern'tmatter house

mind
land worldsea
thing

each ill his an any a no
these ay thy
the
ha sir our
two yea one

duke
queen face body
run

swear
beseech see
out know thank
serve say pray lose
hear hold

he

was

nor
or
did if can todoth let
will 'll hath
makes
didst hast

men
boy which it france
what

england

comes

break thank bring
is
am take
put draw
get
keep read give lay

part

mean

so

madam ay ho
alas o
farewell yes i thou

VERB
NOUN
PRON
ADJ
ADV
ADP
CONJ
DET
NUM
PRT
X
.

.

she 't
that him
them

as

house seaname
sun
sword body duke
matter sar mind

home

thee
heaven

thing face
times state queen

she
you

france him us

god

letter majesty world

unto <eos>; :' into
,
than fromnot of on

england
.

rome
them it yours
gold sorrow

less ye

young save
free
back though because
looks
ere
made when
late own new years

though

off cold
art
much far but need
knows even
thousand
gone honour
old

thousand need
since
same

-1

response

0

C1

1.5
1.0

1

response

C2

1.5
1.0

0.5

0.5

0.0

0.0

-0.5

-0.5

-1.0

-1.0

-1.5
0

20

40

thou (layer 0)

60

80

100

thou (layer 1)

120

140

160

180

200

220

240

260

dim

-1.5

0

20

40

alas (layer 0)

60

80

100

120

140

160

180

200

220

240

260

dim

alas (layer 1)

Figure 14: The comparison of two layers of the LSTM model. A and B: the co-cluster visualization of the hidden states in layer 0 and layer 1,
respectively. C1 and C2: the distribution of model’s response to words “thou” and “alas”, respectively.

