Recurrent Ladder Networks

arXiv:1707.09219v4 [] 18 Dec 2017

Isabeau Prémont-Schwarz, Alexander Ilin, Tele Hotloo Hao,
Antti Rasmus, Rinu Boney, Harri Valpola
The Curious AI Company
{isabeau,alexilin,hotloo,antti,rinu,harri}@cai.fi

Abstract
We propose a recurrent extension of the Ladder networks [22] whose structure
is motivated by the inference required in hierarchical latent variable models. We
demonstrate that the recurrent Ladder is able to handle a wide variety of complex
learning tasks that benefit from iterative inference and temporal modeling. The
architecture shows close-to-optimal results on temporal modeling of video data,
competitive results on music modeling, and improved perceptual grouping based
on higher order abstractions, such as stochastic textures and motion cues. We
present results for fully supervised, semi-supervised, and unsupervised tasks. The
results suggest that the proposed architecture and principles are powerful tools
for learning a hierarchy of abstractions, learning iterative inference and handling
temporal information.

1

Introduction

Many cognitive tasks require learning useful representations on multiple abstraction levels. Hierarchical latent variable models are an appealing approach for learning a hierarchy of abstractions.
The classical way of learning such models is by postulating an explicit parametric model for the
distributions of random variables. The inference procedure, which evaluates the posterior distribution
of the unknown variables, is then derived from the model – an approach adopted in probabilistic
graphical models (see, e.g., [5]).
The success of deep learning can, however, be explained by the fact that popular deep models focus
on learning the inference procedure directly. For example, a deep classifier like AlexNet [19] is
trained to produce the posterior probability of the label for a given data sample. The representations
that the network computes at different layers are related to the inference in an implicit latent variable
model but the designer of the model does not need to know about them.
However, it is actually tremendously valuable to understand what kind of inference is required by
different types of probabilistic models in order to design an efficient network architecture. Ladder
networks [22, 28] are motivated by the inference required in a hierarchical latent variable model. By
design, the Ladder networks aim to emulate a message passing algorithm, which includes a bottom-up
pass (from input to label in classification tasks) and a top-down pass of information (from label to
input). The results of the bottom-up and top-down computations are combined in a carefully selected
manner.
The original Ladder network implements only one iteration of the inference algorithm but complex
models are likely to require iterative inference. In this paper, we propose a recurrent extension
of the Ladder network for iterative inference and show that the same architecture can be used for
temporal modeling. We also show how to use the proposed architecture as an inference engine in
more complex models which can handle multiple independent objects in the sensory input. Thus, the
proposed architecture is suitable for the type of inference required by rich models: those that can
learn a hierarchy of abstractions, can handle temporal information and can model multiple objects in
the input.
31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

z
l+1
el sl
l

el
el−1

dl+1
dl

el

dl

el−1

dl+1

zt+1

yt−1 yt

yt+1

xt

xt+1

y

el

t−1

zt−1 zt

x

dl

x̃

t
(a)

(b)

(c)

Figure 1: (a): The structure of the Recurrent Ladder networks. The encoder is shown in red, the
decoder is shown in blue, the decoder-to-encoder connections are shown in green. The dashed line
separates two iterations t − 1 and t. (b)-(c): The type of hierarchical latent variable models for which
RLadder is designed to emulate message passing. (b): A graph of a static model. (c): A fragment of
a graph of a temporal model. White circles are unobserved latent variables, gray circles represent
observed variables. The arrows represent the directions of message passing during inference.

2

Recurrent Ladder

Recurrent Ladder networks
In this paper, we present a recurrent extension of the Ladder networks which is conducive to iterative
inference and temporal modeling. Recurrent Ladder (RLadder) is a recurrent neural network whose
units resemble the structure of the original Ladder networks [22, 28] (see Fig. 1a). At every iteration
t, the information first flows from the bottom (the input level) to the top through a stack of encoder
cells. Then, the information flows back from the top to the bottom through a stack of decoder cells.
Both the encoder and decoder cells also use the information that is propagated horizontally. Thus, at
every iteration t, an encoder cell in the l-th layer receives three inputs: 1) the output of the encoder
cell from the level below el−1 (t), 2) the output dl (t − 1) of the decoder cell from the same level from
the previous iteration, 3) the encoder state sl (t − 1) from the same level from the previous iteration.
It updates its state value sl (t) and passes the same output el (t) both vertically and horizontally:
sl (t) = fs,l (el−1 (t), dl (t − 1), sl (t − 1))
el (t) = fe,l (el−1 (t), dl (t − 1), sl (t − 1)) .

(1)
(2)

The encoder cell in the bottom layer typically sends observed data (possibly corrupted by noise) as
its output e1 (t). Each decoder cell is stateless, it receives two inputs (the output of the decoder cell
from one level above and the output of the encoder cell from the same level) and produces one output
dl (t) = gl (el (t), dl+1 (t)) ,

(3)

which is passed both vertically and horizontally. The exact computations performed in the cells can
be tuned depending on the task at hand. In practice, we have used LSTM [15] or GRU [8] cells in the
encoder and cells inspired by the original Ladder networks in the decoder (see Appendix A).
Similarly to Ladder networks, the RLadder is usually trained with multiple tasks at different abstraction levels. Tasks at the highest abstraction level (like classification) are typically formulated at the
highest layer. Conversely, the output of the decoder cell in the bottom level is used to formulate
a low-level task which corresponds to abstractions close to the input. The low-level task can be
denoising (reconstruction of a clean input from the corrupted one), other possibilities include object
detection [21], segmentation [3, 23], or in a temporal setting, prediction. A weighted sum of the costs
at different levels is optimized during training.
Connection to hierarchical latent variables and message passing
The RLadder architecture is designed to mimic the computational structure of an inference procedure
in probabilistic hierarchical latent variable models. In an explicit probabilistic graphical model,
inference can be done by an algorithm which propagates information (messages) between the nodes
of a graphical model so as to compute the posterior distribution of the latent variables (see, e.g., [5]).
2

For static graphical models implicitly assumed by the RLadder (see Fig. 1b), messages need to be
propagated from the input level up the hierarchy to the highest level and from the top to the bottom,
as shown in Fig. 1a. In Appendix B, we present a derived iterative inference procedure for a simple
static hierarchical model to give an example of a message-passing algorithm. We also show how that
inference procedure can be implemented in the RLadder computational graph.
In the case of temporal modeling, the type of a graphical model assumed by the RLadder is shown
in Fig. 1c. If the task is to do next step prediction of observations x, an online inference procedure
should update the knowledge about the latent variables yt , zt using observed data xt and compute the
predictive distributions for the input xt+1 . Assuming that the distributions of the latent variables at
previous time instances (τ < t) are kept fixed, the inference can be done by propagating messages
from the observed variables xt and the latent variables y, z bottom-up, top-down and from the past to
the future, as shown in Fig. 1c. The architecture of the RLadder (Fig. 1a) is designed so as to emulate
such a message-passing procedure, that is the information can propagate in all the required directions:
bottom-up, top-down and from the past to the future. In Appendix C, we present an example of the
message-passing algorithm derived for a temporal hierarchical model to show how it is related to the
RLadders’s computation graph.
Even though the motivation of the RLadder architecture is to emulate a message-passing procedure,
the nodes of the RLadder do not directly correspond to nodes of any specific graphical model.1 The
RLadder directly learns an inference procedure and the corresponding model is never formulated
explicitly. Note also that using stateful encoder cells is not strictly motivated by the message-passing
argument but in practice these skip connections facilitate training of a deep network.
As we mentioned previously, the RLadder is usually trained with multiple tasks formulated at
different representation levels. The purpose of tasks is to encourage the RLadder to learn the right
inference procedure, and hence formulating the right kind of tasks is crucial for the success of training.
For example, the task of denoising encourages the network to learn important aspects of the data
distribution [1, 2]. For temporal modeling, the task of next step prediction plays a similar role. The
RLadder is most useful in problems that require accurate inference on multiple abstraction levels,
which is supported by the experiments presented in this paper.
Related work
The RLadder architecture is similar to that of other recently proposed models for temporal modeling
[10, 11, 9, 27, 20]. In [9], the recurrent connections (from time t − 1 to time t) are placed in the
lateral links between the encoder and the decoder. This can make it easier to extend an existing
feed-forward network architecture to the case of temporal data as the recurrent units do not participate
in the bottom-up computations. On the other hand, the recurrent units do not receive information from
the top, which makes it impossible for higher layers to influence the dynamics of lower layers. The
architectures in [10, 11, 27] are quite similar to ours but they could potentially derive further benefit
from the decoder-to-encoder connections between successive time instances (green links in Fig. 1b).
The aforementioned connections are well justified from the message-passing point of view: When
updating the posterior distribution of a latent variable, one should combine the latest information
from the top and from the bottom, and it is the decoder that contains the latest information from the
top. We show empirical evidence to the importance of those connections in Section 3.1.

3

Experiments with temporal data

In this section, we demonstrate that the RLadder can learn an accurate inference algorithm in tasks
that require temporal modeling. We consider datasets in which passing information both in time and
in abstraction hierarchy is important for achieving good performance.
3.1

Occluded Moving MNIST

We use a dataset where we know how to do optimal inference in order to be able to compare the
results of the RLadder to the optimal ones. To this end we designed the Occluded Moving MNIST
1

To emphasize this, we used different shapes for the nodes of the RLadder network (Fig. 1a) and the nodes
of graphical models that inspired the RLadder architecture (Figs. 1b-c).

3

t=1

t=2

t=3

t=4

t=5

observed frames

frames with
occlusion
visualized

optimal temporal
reconstruction
Figure 2: The Occluded Moving MNIST dataset. Bottom row: Optimal temporal recombination for a
sequence of occluded frames from the dataset.
dataset. It consists of MNIST digits downscaled to 14 × 14 pixels flying on a 32 × 32 white
background with white vertical and horizontal occlusion bars (4 pixels in width, and spaced by 8
visible pixels apart) which, when the digit flies behind them, occludes the pixels of the digit (see
Fig. 2). We also restrict the velocities to be randomly chosen in the set of eight discrete velocities
{(1, ±2), (−1, ±2), (2, ±1), (−2, ±1)} pixels/frame, so that apart from the bouncing, the movement
is deterministic. The digits are split into training, validation, and test sets according to the original
MNIST split. The primary task is then to classify the digit which is only partially observable at any
given moment, at the end of five time steps.
In order to do optimal classification, one would need to assimilate information about the digit identity
(which is only partially visible at any given time instance) by keeping track of the observed pixels
(see the bottom row of Fig. 2) and then feeding the resultant reconstruction to a classifier.
In order to encourage optimal inference, we add a next step prediction task to the RLadder at the
bottom of the decoder: The RLadder is trained to predict the next occluded frame, that is the network
never sees the un-occluded digit. This thus mimics a realistic scenario where the ground truth is
not known. To assess the importance of the features of the RLadder, we also do an ablation study.
In addition, we compare it to three other networks. In the first comparison network, the optimal
reconstruction of the digit from the five frames (as shown in Fig. 2) is fed to a static feed-forward
network from which the encoder of the RLadder was derived. This is our gold standard, and obtaining
similar results to it implies doing close to optimal temporal inference. The second, a temporal baseline,
is a deep feed-forward network (the one on which the encoder is based) with a recurrent neural
network (RNN) at the top only so that, by design the network can propagate temporal information
only at a high level, and not at a low level. The third, a hierarchical RNN, is a stack of convolutional
LSTM units with a few convolutional layers in between, which is the RLadder amputated of its
decoder. See Fig. 3 and Appendix D.1 for schematics and details of the architectures.
Fully supervised learning results. The results are presented in Table 1. The first thing to notice
is that the RLadder reaches (up to uncertainty levels) the classification accuracy obtained by the
network which was given the optimal reconstruction of the digit. Furthermore, if the RLadder does
not have a decoder or the decoder-to-encoder connections, or if it is trained without the auxiliary
prediction task, we see the classification error rise almost to the level of the temporal baseline. This
means that even if a network has RNNs at the lowest levels (like with only the feed-forward encoder),
or if it does not have a task which encourages it to develop a good world model (like the RLadder
without the next-frame prediction task), or if the information cannot travel from the decoder to the
encoder, the high level task cannot truly benefit from lower level temporal modeling.
Next, one notices from Table 1 that the top-level classification cost helps the low-level prediction
cost in the RLadder (which in turn helps the top-level cost in a mutually beneficial cycle). This
mutually supportive relationship between high-level and low-level inferences is nicely illustrated by
the example in Fig. 4. Up until time step t = 3 inclusively, the network believes the digit to be a five
4

xt−1

xt−1

xt

Temporal baseline network

xt−1

xt

x̂t

Hierarchical RNN

xt

x̂t+1

RLadder

Figure 3: Architectures used for modeling occluded Moving MNIST. Temporal baseline network is a
convolutional network with a fully connected RNN on top.
Table 1: Performance on Occluded Moving MNIST
Classification error (%)
Optimal reconstruction and static classifier

0.71 ± 0.03

Temporal baseline
Hierarchical RNN (encoder only)
RLadder w/o prediction task
RLadder w/o decoder-to-encoder conn.
RLadder w/o classification task
RLadder

2.02 ± 0.16
1.60 ± 0.05
1.51 ± 0.21
1.24 ± 0.05
0.74 ± 0.09

Prediction error, ·10−5

156.7 ± 0.4
155.2 ± 2.5
150.1 ± 0.1

(Fig. 4a). As such, at t = 3, the network predicts that the top right part of the five which has been
occluded so far will stick out from behind the occlusions as the digit moves up and right at the next
time step (Fig. 4b). Using the decoder-to-encoder connections, the decoder can relay this expectation
to the encoder at t = 4. At t = 4 the encoder can compare this expectation with the actual input
where the top right part of the five is absent (Fig. 4c). Without the decoder-to-encoder connections
this comparison would have been impossible. Using the upward path of the encoder, the network can
relay this discrepancy to the higher classification layers. These higher layers with a large receptive
field can then conclude that since it is not a five, then it must be a three (Fig. 4d). Now thanks to the
decoder, the higher classification layers can relay this information to the lower prediction layers so
that they can change their prediction of what will be seen at t = 5 appropriately (Fig. 4e). Without a
decoder which would bring this high level information back down to the low level, this drastic update
of the prediction would be impossible. With this information the lower prediction layer can now
predict that the top-left part of the three (which it has never seen before) will appear at the next time
step from behind the occlusion, which is indeed what happens at t = 5 (Fig. 4f).
Semi-supervised learning results. In the following experiment, we test the RLadder in the semisupervised scenario when the training data set contains 1.000 labeled sequences and 59.000 unlabeled
ones. To make use of the unlabeled data, we added an extra auxiliary task at the top level which
was the consistency cost with the targets provided by the Mean Teacher (MT) model [26]. Thus,
the RLadder was trained with three tasks: 1) next step prediction at the bottom, 2) classification
at the top, 3) consistency with the MT outputs at the top. As shown in Table 2, the RLadder
improves dramatically by learning a better model with the help of unlabeled data independently and in
addition to other semi-supervised learning methods. The temporal baseline model also improves the
classification accuracy by using the consistency cost but it is clearly outperformed by the RLadder.
3.2

Polyphonic Music Dataset

In this section, we evaluate the RLadder on the midi dataset converted to piano rolls [6]. The dataset
consists of piano rolls (the notes played at every time step, where a time step is, in this case, an eighth
note) of various piano pieces. We train an 18-layer RLadder containing five convolutional LSTMs
and one fully-connected LSTM. More details can be found in Appendix D.2. Table 3 shows the
5

t=1

t=2

t=3

t=4

t=5

ground-truth
unoccluded
digits
f
observed
frames

c
b

e

predicted
frames
a

probe of internal representations

d

Figure 4: Example prediction of an RLadder on the occluded moving MNIST dataset. First row: the
ground truth of the digit, which the network never sees and does not train on. Second row: The actual
five frames seen by the network and on which it trains. Third row: the predicted next frames of a
trained RLadder. Fourth row: A stopped-gradient (gradient does not flow into the RLadder) readout
of the bottom layer of the decoder trained on the ground truth to probe what aspects of the digit are
represented by the neurons which predict the next frame. Notice how at t = 1, the network does
not yet know in which direction the digit will move and so it predicts a superposition of possible
movements. Notice further (red annotations a-f), that until t = 3, the network thought the digit was a
five, but when the top bar of the supposed five did not materialize on the other side of the occlusion
as expected at t = 4, the network immediately concluded correctly that it was actually a three.
Table 2: Classification error (%) on semi-supervised Occluded Moving MNIST
1k labeled
1k labeled & 59k unlabeled
w/o MT
MT
3.50 ± 0.28
10.86 ± 0.43
10.49 ± 0.81

Optimal reconstruction and static classifier
Temporal baseline
RLadder

3.50 ± 0.28
10.86 ± 0.43
5.20 ± 0.77

1.34 ± 0.04
3.14 ± 0.16
1.69 ± 0.14

negative log-likelhoods of the next-step prediction obtained on the music dataset, where our results
are reported as mean plus or minus standard deviation over 10 seeds. We see that the RLadder is
competitive with the best results, and gives the best results amongst models outputting the marginal
distribution of notes at each time step.
The fact that the RLadder did not beat [16] on the midi datasets shows one of the limitations of
RLadder. Most of the models in Table 3 output a joint probability distribution of notes, unlike
RLadder which outputs the marginal probability for each note. That is to say, those models, to output
the probability of a note, take as input the notes at previous time instances, but also the ground truth
of the notes to the left at the same time instance. RLadder does not do that, it only takes as input the
past notes played. Even though, as the example in 3.1 of the the digit five turning into a three after
seeing only one absent dot, shows that internally the RLadder models the joint distribution.

4

Experiments with perceptual grouping

In this section, we show that the RLadder can be used as an inference engine in a complex model which
benefits from iterative inference and temporal modeling. We consider the task of perceptual grouping,
that is identifying which parts of the sensory input belong to the same higher-level perceptual
6

Table 3: Negative log-likelihood (smaller is better) on polyphonic music dataset
Piano-midi.de

Nottingham

Muse

JSB Chorales

6.48
5.54
6.01
5.60
5.03
4.34
3.90

8.51
7.59
6.27
5.56
6.10
5.92
5.86

7.43

8.76

5.69 ± 0.02

5.64 ± 0.02

Models outputting a joint distribution of notes:
NADE masked [4]
NADE [4]
RNN-RBM [6]
RNN-NADE (HF) [6]
LSTM-NADE [16]
TP-LSTM-NADE [16]
BALSTM [16]

7.42
7.05
7.09
7.05
7.39
5.49
5.00

3.32
2.89
2.39
2.31
2.06
1.64
1.62

Models outputting marginal probabilities for each note:
RNN [4]
LSTM [17]
MUT1 [17]
RLadder

7.88
6.866
6.792
6.19 ± 0.02

3.87
3.492
3.254
2.42 ± 0.03

components (objects). We enhance the previously developed model for perceptual grouping called
Tagger [13] by replacing the originally used Ladder engine with the RLadder. For another perspective
on the problem see [14] which also extends Tagger to a recurrent neural network, but does so from an
expectation maximization point of view.
4.1

Recurrent Tagger

Tagger is a model designed for perceptual grouping. When applied to images, the modeling assumption is that each pixel x̃i belongs to one of the K objects, which is described by binary variables zi,k :
zi,k = 1 if pixel i belongs to object k and zi,k = 0 otherwise. The reconstruction of the whole image
using object k only is µ k which is a vector with as many elements µi,k as there are pixels. Thus, the
assumed probabilistic model can be written as follows:
K
Y
Y
p(zk , µ k |hk )p(hk )
(4)
p(x̃, µ , z, h) =
N (x̃i |µi,k , σk2 )zi,k
k=1

i,k

where zk is a vector of elements zi,k and hk is (a hierarchy of) latent variables which define the shape
and the texture of the objects. See Fig. 5a for a graphical representation of the model and Fig. 5b
for possible values of model variables for the textured MNIST dataset used in the experiments of
Section 4.2. The model in (4) is defined for noisy image x̃ because Tagger is trained with an auxiliary
low-level task of denoising. The inference procedure in model (4) should evaluate the posterior
distributions of the latent variables zk , µ k , hk for each of the K groups given corrupted data x̃.
Making the approximation that the variables of each of the K groups are independent a posteriori
Y
p(z, µ , h|x̃) ≈
q(zk , µ k , hk ) ,
(5)
k

the inference procedure could be implemented by iteratively updating each of the K approximate
distributions q(zk , µ k , hk ), if the model (4) and the approximation (5) were defined explicitly.
Tagger does not explicitly define a probabilistic model (4) but learns the inference procedure directly.
The iterative inference procedure is implemented by a computational graph with K copies of the same
Ladder network doing inference for one of the groups (see Fig. 5c). At the end of every iteration, the
inference procedure produces the posterior probabilities πi,k that pixel i belongs to object k and the
point estimates of the reconstructions µ k (see Fig. 5c). Those outputs, are used to form the low-level
cost and the inputs for the next iteration (see more details in [13]). In this paper, we replace the
original Ladder engine of Tagger with the RLadder. We refer to the new model as RTagger.
4.2

Experiments on grouping using texture information

The goal of the following experiment is to test the efficiency of RTagger in grouping objects using
the texture information. To this end, we created a dataset that contains thickened MNIST digits with
7

h1

hk

h2

µ1
µk

µ2

z1

z2

zk
K

K
x

x̃

x

x̃
(a)

(b)

π, µ

x̃

π, µ

(c)

Figure 5: (a): Graphical model for perceptual grouping. White circles are unobserved latent variables,
gray circles represent observed variables. (b): Examples of possible values of model variables for the
textured MNIST dataset. (c): Computational graph that implements iterative inference in perceptual
grouping task (RTagger). Two graph iterations are drawn. The plate notation represent K copies of
the same graph.

(a)

(b)

(c)

(d)

Figure 6: (a): Example image from the Brodatz-textured MNIST dataset. (b): The image reconstruction m0 by the group that learned the background. (c): The image reeconstruction m1 by the group
that learned the digit. (d): The original image colored using the found grouping π k .
20 textures from the Brodatz dataset [7]. An example of a generated image is shown in Fig. 6a. To
create a greater diversity of textures (to avoid over-fitting), we randomly rotated and scaled the 20
Brodatz textures when producing the training data.
The network trained on the textured MNIST dataset has the architecture presented in Fig. 5c with
three iterations. The number of groups was set to K = 3. The details of the RLadder architecture are
presented in Appendix D.3. The network was trained on two tasks: The low-level segmentation task
was formulated around denoising, the same way as in the Tagger model [13]. The top-level cost was
the log-likelihood of the digit class at the last iteration.
Table 4 presents the obtained performance on the textured MNIST dataset in both fully supervised
and semi-supervised settings. All experiments were run over 5 seeds. We report our results as mean
plus or minus standard deviation. In some runs, Tagger experiments did not converge to a reasonable
solution (because of unstable or too slow convergence), so we did not include those runs in our
evaluations. Following [13], the segmentation accuracy was computed using the adjusted mutual
information (AMI) score [29] which is the mutual information between the ground truth segmentation
and the estimated segmentation π k scaled to give one when the segmentations are identical and zero
when the output segmentation is random.
For comparison, we trained the Tagger model [13] on the same dataset. The other comparison method
was a feed-forward convolutional network which had an architecture resembling the bottom-up pass
(encoder) of the RLadder and which was trained on the classification task only. One thing to notice is
that the results obtained with the RTagger clearly improve over iterations, which supports the idea that
iterative inference is useful in complex cognitive tasks. We also observe that RTagger outperforms
Tagger and both approaches significantly outperform the convolutional network baseline in which the
classification task is not supported by the input-level task. We have also observed that the top-level
classification tasks makes the RTagger faster to train in terms of the number of updates, which also
supports that the high-level and low-level tasks mutually benefit from each other: Detecting object
8

Table 4: Results on the Brodatz-textured MNIST. i-th column corresponds to the intermediate results
of RTagger after the i-th iteration. In the fully supervised case, Tagger was only trained successfully
in 2 of the 5 seeds, the given results are for those 2 seeds. In the semi-supervised case, we were not
able to train Tagger successfully.
50k labeled
1k labeled + 49k unlabeled
Segmentation accuracy, AMI:
Segmentation accuracy, AMI:
RTagger 0.55 0.75 0.80 ± 0.01
Tagger
−
− 0.73 ± 0.02
RTagger 0.56 0.74 0.80 ± 0.03
Classification error, %:
RTagger
Tagger
ConvNet

18.2
−
–

8.0
−
–

Classification error, %:
5.9 ± 0.2
12.15 ± 0.1
14.3 ± 0.46

RTagger
ConvNet

63.8
–

28.2
–

22.6 ± 6.2
88 ± 0.30

Figure 7: Example of segmentation and generation by the RTagger trained on the Moving MNIST.
First row: frames 0-9 is the input sequence, frames 10-15 is the ground truth future. Second row:
Next step prediction of frames 1-9 and future frame generation (frames 10-15) by RTagger, the colors
represent grouping performed by RTagger.

boundaries using textures helps classify a digit, while knowing the class of the digit helps detect the
object boundaries. Figs. 6b-d show the reconstructed textures and the segmentation results for the
image from Fig. 6a.
4.3

Experiments on grouping using movement information

The same RTagger model can perform perceptual grouping in video sequences using motion cues. To
demonstrate this, we applied the RTagger to the moving MNIST [25]2 sequences of length 20 and the
low-level task was prediction of the next frame. When applied to temporal data, the RTagger assumes
the existence of K objects whose dynamics are independent of each other. Using this assumption,
the RTagger can separate the two moving digits into different groups. We assessed the segmentation
quality by the AMI score which was computed similarly to [13, 12] ignoring the background in the
case of a uniform zero-valued background and overlap regions where different objects have the same
color. The achieved averageAMI score was 0.75. An example of segmentation is shown in Fig. 7.
When we tried to use Tagger on the same dataset, we were only able to train successfully in a single
seed out of three. This is possibly because speed is intermediate level of abstraction not represented
at the pixel level. Due to its reccurent connections, RTagger can keep those representations from
one time step to the next and segment accordingly, something more difficult for Tagger to do, which
might explain the training instability.

5

Conclusions

In the paper, we presented recurrent Ladder networks. The proposed architecture is motivated by
the computations required in a hierarchical latent variable model. We empirically validated that the
recurrent Ladder is able to learn accurate inference in challenging tasks which require modeling
dependencies on multiple abstraction levels, iterative inference and temporal modeling. The proposed
model outperformed strong baseline methods on two challenging classification tasks. It also produced
competitive results on a temporal music dataset. We envision that the purposed Recurrent Ladder
will be a powerful building block for solving difficult cognitive tasks.
2

For this experiment, in order to have the ground truth segmentation, we reimplemented the dataset ourselves.

9

Acknowledgments
We would like to thank Klaus Greff and our colleagues from The Curious AI Company for their
contribution in the presented work, especially Vikram Kamath and Matti Herranen.

References
[1] Alain, G., Bengio, Y., and Rifai, S. (2012). Regularized auto-encoders estimate local statistics. CoRR,
abs/1211.4246.
[2] Arponen, H., Herranen, M., and Valpola, H. (2017). On the exact relationship between the denoising
function and the data distribution. arXiv preprint arXiv:1709.02797.
[3] Badrinarayanan, V., Kendall, A., and Cipolla, R. (2015). Segnet: A deep convolutional encoder-decoder
architecture for image segmentation. arXiv preprint arXiv:1511.00561.
[4] Berglund, M., Raiko, T., Honkala, M., Kärkkäinen, L., Vetek, A., and Karhunen, J. T. (2015). Bidirectional
recurrent neural networks as generative models. In Advances in Neural Information Processing Systems.
[5] Bishop, C. M. (2006). Pattern Recognition and Machine Learning (Information Science and Statistics).
Springer-Verlag New York, Inc., Secaucus, NJ, USA.
[6] Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2012). Modeling temporal dependencies in
high-dimensional sequences: Application to polyphonic music generation and transcription. In Proceedings
of the 29th International Conference on Machine Learning (ICML-12), pages 1159–1166.
[7] Brodatz, P. (1966). Textures: a photographic album for artists and designers. Dover Pubns.
[8] Cho, K., Van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machine
translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259.
[9] Cricri, F., Honkala, M., Ni, X., Aksu, E., and Gabbouj, M. (2016). Video Ladder networks. arXiv preprint
arXiv:1612.01756.
[10] Eyjolfsdottir, E., Branson, K., Yue, Y., and Perona, P. (2016). Learning recurrent representations for
hierarchical behavior modeling. arXiv preprint arXiv:1611.00094.
[11] Finn, C., Goodfellow, I. J., and Levine, S. (2016). Unsupervised learning for physical interaction through
video prediction. In Advances in Neural Information Processing Systems 29.
[12] Greff, K., Srivastava, R. K., and Schmidhuber, J. (2015). Binding via reconstruction clustering. CoRR,
abs/1511.06418.
[13] Greff, K., Rasmus, A., Berglund, M., Hao, T., Valpola, H., and Schmidhuber, J. (2016). Tagger: Deep
unsupervised perceptual grouping. In Advances in Neural Information Processing Systems 29.
[14] Greff, K., van Steenkiste, S., and Schmidhuber, J. (2017). Neural expectation maximization. In ICLR
Workshop.
[15] Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735–
1780.
[16] Johnson, D. D. (2017). Generating polyphonic music using tied parallel networks. In International
Conference on Evolutionary and Biologically Inspired Music and Art.
[17] Jozefowicz, R., Zaremba, W., and Sutskever, I. (2015). An empirical exploration of recurrent network
architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15).
[18] Kingma, D. and Ba, J. (2015). Adam: A method for stochastic optimization. In The International
Conference on Learning Representations (ICLR), San Diego.
[19] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional
neural networks. In Advances in neural information processing systems.

10

[20] Laukien, E., Crowder, R., and Byrne, F. (2016). Feynman machine: The universal dynamical systems
computer. arXiv preprint arXiv:1609.03971.
[21] Newell, A., Yang, K., and Deng, J. (2016). Stacked hourglass networks for human pose estimation. In
European Conference on Computer Vision. Springer.
[22] Rasmus, A., Berglund, M., Honkala, M., Valpola, H., and Raiko, T. (2015). Semi-supervised learning with
Ladder networks. In Advances in Neural Information Processing Systems.
[23] Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image
segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention.
[24] Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2014). Striving for simplicity: The all
convolutional net. arXiv preprint arXiv:1412.6806.
[25] Srivastava, N., Mansimov, E., and Salakhudinov, R. (2015). Unsupervised learning of video representations
using LSTMs. In International Conference on Machine Learning, pages 843–852.
[26] Tarvainen, A. and Valpola, H. (2017). Mean teachers are better role models: Weight-averaged consistency
targets improve semi-supervised deep learning results. In Advances in neural information processing systems.
[27] Tietz, M., Alpay, T., Twiefel, J., and Wermter, S. (2017). Semi-supervised phoneme recognition with
recurrent ladder networks. In International Conference on Artificial Neural Networks 2017.
[28] Valpola, H. (2015). From neural PCA to deep unsupervised learning. Advances in Independent Component
Analysis and Learning Machines.
[29] Vinh, N. X., Epps, J., and Bailey, J. (2010). Information theoretic measures for clusterings comparison:
Variants, properties, normalization and correction for chance. Journal of Machine Learning Research, 11(Oct),
2837–2854.

11

A

Cells used in the decoder of the RLadder

Decoder cells receive two inputs: vector vtop from the top and vector h from the encoder and produce
output y.
A.1

G1 and convG1 cells

u = BN(Avtop + b)
s = f (u, ws )
y = s h + (1 − s) f (u, w) ,
where BN() is batch normalization and
f (u, w) = w0 sigmoid(w1 u + w2 ) + w3 u + w4
(6)
with wi being the elements of vector w and the element-wise product. In the convolutional version
ConvG1 of that cell, u in the first formula is computed with the convolution operation ∗:
u = A ∗ vtop + b.
A.2

convG2 cell

µ 1 = f (LN(A1 ∗ vtop ) + LN(B1 ∗ h) + c1 , w1 )
µ 2 = f (LN(A2 ∗ vtop ) + LN(B2 ∗ h) + c2 , w2 )
s = f (LN(As ∗ vtop ) + LN(Bs ∗ h) + cs , ws )
y = s µ 1 + (1 − s) µ 2
where f is defined in (6) and LN() is layer normalization.
A.3

convG3 cell

u = relu(LN(A ∗ vtop ) + LN(B ∗ h) + c)
s = sigmoid(Ws ∗ u)
y = s (D ∗ u) + (1 − s) (E ∗ u) ,
where LN() is layer normalization.
A.4

Detailes of normalizations used

Encoder: For all the experiments all non-recurrent layers in the encoder, whether convolutional or not,
are normalized using batch normalization. In the convolution LSTM, there is a layer normalization
after the all the first convolutions of the inputs in the RLadder experiments and batch normalization
in the RTagger experiments, these normalizations were important to get things working.
Decoder: In the decoder gating functions, there is always a layer normalization after the first
convolution over the concatenation of the inputs in the RLadder experiments, and this normalization
was replaced by batch normalization in the RTagger experiments. A normalization at this point was
critical for optimal performance.

B

Example of approximate inference in a static model

The structure of the RLadder is designed to emulate a message-passing algorithm in a hierarchical
latent variable model. To illustrate this, let us consider a simple hierarchical latent variable model
with three one-dimensional variables whose joint probability distribution is given by
p(x, y, z) = p(z)p(y|z)p(x|y)
(7)
p(x|y) = N (x|wx y, σx2 )
p(y|z) =
p(z) =

N (y|wyz z, σy2 )
N (z|µz , σz2 ) ,
12

(8)
(9)
(10)

2
my wyz
,
wyz σy2

z
y

y

x

x

x̃

x̃

(a)

wyz mz , σ12
y

wyz mz , σ12

2
mx wx
2
wx , σx

y

2
mx wx
2
wx , σx

wy my , σ12

x

wy my , σ12

x̃, σ12

x

x̃, σ12
t

(b)

(c)

2
my wyz
,
wyz σy2
2
mx wx
2
wx , σx
2
mx wx
2
w x , σx

x̃, σ12
x̃, σ12
t+1

Figure 8: (a): Simple hierarchical latent variable model. A filled node represents an observed variable.
(b): Directions of message propagation in an inference procedure. (c): Computational graph which
implements an iterative inference procedure has the RLadder architecture.
where N (·|m, v) denotes the Gaussian probability density function with mean m and variance v. We
want to derive a denoising algorithm that recovers clean observation x from its corrupted version x̃,
where the corruption is also modeled to be Gaussian:
p(x̃|x) = N (x̃|x, σ 2 ) .
(11)
The reason we look at denoising is because denoising is a task which can be used for unsupervised
learning of the data distribution [1, 2]. The graphical representation of the model shown in Fig. 8a.
In order to do optimal denoising, one needs to evaluate the expectation of p(x|x̃), which can be
done by learning the joint posterior distribution of the unknown variables x, y and z. For this linear
Gaussian model, it is possible to derive an inference algorithm that is guaranteed to produce the exact
posterior distribution in a finite number of steps (see, e.g., [5]). However, in more complex models,
the posterior distribution of the latent variables is impossible to represent exactly and therefore a
neural network that learns the inference procedure needs to represent an approximate distribution. To
this end, we derive an approximate inference procedure for this simple probabilistic model.
Using the variational Bayesian approach, we can approximate the joint posterior distribution by a
distribution of a simpler form. For example, all the latent variables can be modeled to be independent
Gaussian variables a posteriori:
p(x, y, z|x̃) ≈ q(x, y, z) = q(x)q(y)q(z),
(12)
q(x) = N (x|mx , vx ),
(13)
q(y) = N (y|my , vy ),
(14)
q(z) = N (z|mz , vz ) .
(15)
and the goal of the inference procedure is to estimate parameters mx , vx , my , vy , mz , vz of the
approximate posterior (12)–(15) for the model in (7)–(11) with fixed parameters wx , σx2 , wyz , σy2 , µz ,
σz2 , σ 2 .
The optimal posterior approximation q(x, y, z) can be found by minimizing the lower bound of the
Kullback-Leibler divergence between the true distribution and the approximation:
Z
q(x)q(y)q(z)
CVB = log
q(x)q(y)q(z)dydz = hlog q(x)q(y)q(z)i − hlog p(x, y, z, x̃)i (16)
p(x, y, z, x̃)
where h·i denotes the expectation over q(x, y, z). This can be done with the following iterative
procedure:
1
1
mx = sx x̃ + (1 − sx )wy my
sx = sigmoid(log(σx2 /σ 2 ))
vx−1 = 2 + 2 (17)
σ
σx
mx
w2
1
my = sy
+ (1 − sy )wyz mz
sy = sigmoid(log(σy2 wx2 /σx2 ))
vy−1 = 2x + 2 (18)
wx
σx
σy
mz = sz

my
+ (1 − sz )µz
wyz

2
sz = sigmoid(log(σz2 wyz
/σy2 ))

13

vz−1 =

2
wyz
1
+ 2.
2
σy
σz
(19)

zt−1 zt

zt+1

zt−1 zt

zt+1

m̃z,t−1

m0z,t
mz,t−1

yt−1 yt

yt+1

yt−1 yt

yt+1

m̃y,t−1

m0y,t
my,t−1

xt

xt+1

xt

xt+1

xt−1

(a)

x̂t
t−1

(b)

xt

x̂t+1

(c)

t

Figure 9: (a): Fragment of a graph of a temporal model relevant for updating the distributions of
unknown variables after observing xt . Light-gray circles represent latent variables whose distribution
is not updated. (b): Directions of information propagation needed for inference. (c): The structure of
the RLadder network can be seen as a computational graph implementing information flow in (b).
The dotted arrows are the skip connections that would be needed if we forced the activations to be
literally interpreted as the distribution parameters of the latent variables.

Thus, in order to update the posterior distribution of a latent variable, one needs to combine the
information coming from one level above and from one level below. For example, in order to
update q(y), one needs information from below (mx /wx and wx2 /σx2 ) and from above (wyz mz and
1/σy2 ). We can think of the information needed for updating the parameters describing the posterior
distributions of the latent variables as ‘messages’ propagating between the nodes of a probabilistic
graphical model (see Fig. 8b). Since there are mutual dependencies between mx , my and mz in
(17)–(19), the update rules need to be iterated multiple times until convergence. This procedure can
be implemented using the RLadder computational graph with the messages shown in Fig. 8c. Note
that in practice, the computations used in the cells of the RLadder are not dictated by any particular
explicit probabilistic model. The original Ladder networks [22] contain only one iteration of the
bottom-up and top-down passes. The RLadder extends the model to multiple passes. Note also that
the computations used in the decoder (top-down pass) of the Ladder networks are inspired by the
gating structure of the update rules (17)–(19) in simple Gaussian models.

C

Example of approximate inference in a simple temporal model

In this section we consider a simple hierarchical and temporal model and look at the relationship
between temporal inferance in that model and the computational structure of RLadder. Consider a
simple probabilistic model with three levels of hierarchy xt , yt , zt in which variables vary in time
(see Fig. 9a). The conditional distributions given the latent variables in the past yt−1 , zt−1 are defined
as follows:
p(xt |yt ) = N (xt |wx xt−1 + wxy yt , σx2 )

(20)

N (yt |wy yt−1 + wyz zt , σy2 )
N (zt |wz zt−1 , σz2 )

(21)

p(yt |yt−1 , zt ) =
p(zt |zt−1 ) =

(22)

At time t − 1 the observed variables are x1..t−1 = (x1 , ..., xt−1 ). Using the variational Bayesian
approach, we can approximate the joint posterior distribution of the latent variables y and z by a
distribution of a simpler form:
p(yt−1 , zt−1 |x1..t−1 ) ≈ q(yt−1 )q(zt−1 )
q(yt−1 ) = N (yt−1 |my,t−1 , vy,t−1 )
q(zt−1 ) = N (zt−1 |mz,t−1 , vz,t−1 ).
The cost function minimized in the variational Bayesian approach is:
CVB = hlog q(yt )q(zt ) − log p(xt |yt )p(yt |yt−1 , zt )p(zt |zt−1 ) − log p(yt−1 )p(zt−1 )i
14

where the expectation is taken over q(yt , zt , yt−1 , zt−1 , . . . , y1 , z1 ) =
elaborate the terms of CVB . The first term is

Qt

s=1

q(ys )q(zs ). Let us



1
1
1
1
− log 2πvy,t −
(yt − my,t )2 − log 2πvz,t −
(zt − mz,t )2
2
2vy,t
2
2vz,t
1
1 1
1
= − log 2πvy,t − − log 2πvz,t − .
2
2 2
2



Cq =

The second term is


1
1
1
1
2
2
2
2
Cp = − − log 2πσx − 2 (xt − wx yt ) − log 2πσy − 2 (yt − wy yt−1 − wyz zt )
2
2σx
2
2σy


1
1
− − log 2πσz2 − 2 (zt − wz zt−1 )2
2
2σz
1
1
= log 2πσx2 + 2 (x2t − 2xt wx my,t + wx2 (m2y,t + vy,t ))
2
2σx
1
1
2
(m2z,t + vz,t )
+ log 2πσy2 + 2 (m2y,t + vy,t + wy2 (m2y,t−1 + vy,t−1 ) + wyz
2
2σy
− 2my,t wy my,t−1 − 2my,t wyz mz,t + 2wy my,t−1 wyz mz,t )
1
1
+ log 2πσz2 + 2 (m2z,t + vz,t − 2mz,t wz mz,t−1 + wz2 (m2z,t−1 + vz,t−1 )) .
2
2σz
The last term is a function of variational parameters my,t−1 , mz,t−1 , vy,t−1 , vz,t−1 which we do not
update at time instance t. Taking the derivative of CVB wrt the variational parameters yields:
∂CVB
1
1
= 2 (−xt wx + wx2 my,t ) + 2 (my,t − wy my,t−1 − wyz mz,t )
∂my,t
σx
σy
1
∂CVB
1
w2
=−
+ x2 + 2
∂vy,t
2vy,t
2σx
2σy
1
∂CVB
wyz
= 2 (wyz mz,t − my,t + wy my,t−1 ) + 2 (mz,t − wz mz,t−1 )
∂mz,t
σy
σz
2
wyz
∂CVB
1
1
=−
+ 2 + 2.
∂vz,t
2vz
2σy
2σz

Equating the derivatives to zero yields:
−1
wx2
1
+
σx2
σy2


xt wx
wy my,t−1 + wyz mz,t
= vy,t
+
σx2
σy2
!−1
2
wyz
1
=
+ 2
2
σy
σz


(my,t − wy my,t−1 )wyz
wz mz,t−1
= vz,t
+
.
σy2
σz2


vy,t =
my,t
vz,t
mz,t

These computations can be done by first estimating my,t , mz,t before observing xt and then correcting
using the observation. Let us show how it is done for y. We define m0y,t to be the posterior mean
before observing xt (next step decoder prediction), m̃y,t (encoder posterior) to be the posterior means
after observing xt but before updating the approximate posterior of higher latent variables (z in
this case), and finally my,t to be the posterior mean after observing xt and updating all posteriors
15

(decoder, same step prediction). Thus, we can write
m0y,t = wy my,t−1 + wyz m0z,t


m0y,t
xt
xt w x
+ 2
= sy,t
+ (1 − sy,t )m0y,t
m̃y,t = vy,t
2
σx
σy
wx
sy,t = sigmoid(log(σy2 wx2 /σx2 ))

wyz vy,t
mz,t − m0z,t ,
my,t = m̃y,t +
2
σy

(23)
(24)
(25)
(26)

which if we eliminate the tilded and primed m’s, give exactly the same equations as previously.
Generalizing the results for an arbitrary number of variables in the chain we have that for a generic
level l in a hierarchical chain, we have:
m0l,t = wl ml,t−1 + wl,l+1 m0l+1,t
m̃l,t

(27)

(1 − sl )
= m0l,t +
(m̃l−1,t − m0l−1,t )
wl−1,l

sl =

2
wl−1,l
2
σl−1

1
σl2

+

2
wl−1,l
2
σl−1

2
2
= sigmoid(log(σl2 wl−1,l
/σl−1
))

ml,t = m̃l,t + sl wl,l+1 (ml+1,t − m0l+1,t )
= m̃l,t + sl wl,l+1 (ml+1,t −

m0l,t

− wl ml,t−1
)
wl,l+1

(28)

(29)
(30)
(31)

These equations suggest a computational graph as shown in Fig. 9c. This is, if we literally interpret
m̃ to be the activations of the encoder and m and m0 to be the activations of the decoder, then we
would conclude that we need upward-diagonal (the grey dotted arrow in Fig. 9c) decoder to encoder
connections, and a recurrent decoder (curved dotted arrow in Fig. 9c). But we impose no such
constraint, and the dotted lines are in fact only skip connections. So as long an information can be
copied through accross neural network layers (which it can), and as long as each layer has sufficient
capacity, we can remove those skip connections and retain the same computational capabilities. This
is what is done in RLadder. However, it would be interesting to see if adding those skip connections
help with training on temporal tasks.
Using gatings in the bottom-up computations are justified by (28). Even though the top-down updates
(27) and (31) are simple summations in the case of this linear Gaussian model, the more general
gatings in the top-down pass provides more flexibility for nonlinear models. The activations in the
level above can select one of a few possible evolution scenarios on the level below.
As we discussed previously, RLadder networks are usually trained by introducing tasks that encourage
the network to learn the optimal inference of the latent variables. For temporal models, the natural
low-level task is predicting the input at the next time instance. Thus the cost is − log(p(xt+1 |x1..t )):
Cpred = (x̂t+1 − xt+1 )2
if we assume a Gaussian distribution for x̂.

D
D.1

Details of experiments
Detailes of experiments with Occluded Moving MNIST

The RLadder architecture trained for the occluded moving MNIST dataset is presented in Table 5.
The architecture of the baseline temporal model and the static feed-forward network trained on
optimal digit reconstructions are shown in Tables 6–7. Initially, we trained the network only on the
training set using the Adam optimizer [18] with an initial learning rate of 0.001. Every time the
classification error on the validation set rose, we divided the learning rate by 2 until a minimum value
of 0.0001. We then trained the network using both the training and validation sets following the
learning rate schedule learned previously.
16

Table 5: Structure of RLadder for Occluded Moving MNIST
Encoder
Layer

# channels (size)

1
1
1
1
2
1
1
1
1
2
1
1
1
2

1
32
32
32
32
MAX
32
64
64
64
MAX
128
64
32
AVG
16
10

Input
ConvLSTM
Conv
Conv
Conv
Pool
ConvLSTM
Conv
Conv
Conv
Pool
Conv
Conv
Conv
Pool
LSTM
SoftMax

Table 6: Temporal baseline network for
Occluded Moving MNIST
Layer
Input
Conv
Conv
Conv
Pool
Conv
Conv
Conv
Pool
Conv
Conv
Conv
Pool
LSTM
SoftMax

Decoder

Stride

Stride

# channels
(size)

1
1
1
2
1
1
1
2
1
1
1
2

1
32
32
32
MAX
64
64
64
MAX
128
64
32
AVG
16
10

filter
size

filter size
(3, 3)
(3, 3)
(3, 3)
(3, 3)
(2, 2)
(3, 3)
(3, 3)
(3, 3)
(3, 3)
(2, 2)
(3, 3)
(1, 1)
(1, 1)
(2, 2)

filter size
convG3
convG3
convG3
convG3
convG3
convG3
convG3
convG3
convG3
convG3
convG3
convG3
convG3
G1
G1
-

(9,9)
(3, 3)
(3, 3)
(3, 3)
(6, 6)
(9, 9)
(3, 3)
(3, 3)
(3, 3)
(6, 6)
(3, 3)
(3, 3)
(3, 3)

Table 7: Feed-forward network in which
the optimal temporal reconstruction of Occluded Moving MNIST is inputed for classification
Layer

(3, 3)
(3, 3)
(3, 3)
(2, 2)
(3, 3)
(3, 3)
(3, 3)
(2, 2)
(3, 3)
(1, 1)
(1, 1)
(2, 2)

Input
Conv
Conv
Conv
Pool
Conv
Conv
Conv
Pool
Conv
Conv
Conv
Pool
SoftMax

17

Stride

# channels
(size)

filter
size

1
1
1
2
1
1
1
2
1
1
1
2

1
32
32
32
MAX
64
64
64
MAX
128
64
32
AVG
10

(3, 3)
(3, 3)
(3, 3)
(2, 2)
(3, 3)
(3, 3)
(3, 3)
(2, 2)
(3, 3)
(1, 1)
(1, 1)
(2, 2)

D.2

Details of experiments with Polyphonic Music data

The RLadder architecture trained for the music dataset is presented in Table 8. We trained using
the Adam optimizer with a learning rate of 0.01. Because the network overfited very rapidly, we
determined the optimal stopping time using the validation set and then trained on the training and
validation set for the same number of epochs (between 3 and 10 depending on the hyperparameters).
Table 8: Structure of RLadder for Polyphonic Music dataset
Encoder
Layer

Stride

# channels (size)

1
2
2
2
2
2

1
32
64
96
128
160
AVG
96
19

Input
ConvLSTM
ConvLSTM
ConvLSTM
ConvLSTM
ConvLSTM
Pool
LSTM
SoftMax

D.3

Decoder
filter size
(1, 3)
(1, 3)
(1, 3)
(1, 3)
(1, 3)
(1, 2)

filter size
ConvG2
ConvG2
ConvG2
ConvG2
ConvG2
G1
G1
Identity

(1,3)
(1,3)
(1,3)
(1,3)
(1,3)

Details of experiments with textured MNIST

The RTagger architecture used to train on Brodatz-textured MNIST data is in Table 9. The baseline
convolutional network is model C from [24]. The networks were trained using the Adam optimizer
with learning rate 0.0004. All training results were evaluated at update 160000. The size of the
mini-batch was 16 for all the models. Batch normalization was applied to encoder and decoder layers.
Table 9: Structure of RTagger for textured MNIST dataset
Encoder
Layer
Input
ConvLSTM
Conv
Conv
ConvLSTM
Conv
Conv
ConvLSTM
Conv
Conv
ConvLSTM
Pool
FC

Decoder

Stride

# channels (size)

filter size

2
1
1
2
1
1
2
1
1
2
-

1
96
96
96
192
192
192
192
192
192
11
11

(8, 8)
(5, 5)
(5, 5)
(4, 4)
(5, 5)
(5, 5)
(4, 4)
(5, 5)
(5, 5)
(4, 4)
(6,6)
-

18

ConvG1
ConvG1
ConvG1
ConvG1
ConvG1
ConvG1
ConvG1
ConvG1
ConvG1
ConvG1
AVERAGE
G1

