Teaching Machines to Code:
Neural Markup Generation with Visual Attention

Sumeet Singh 1

arXiv:1802.05415v1 [cs.LG] 15 Feb 2018

Abstract
We present a deep recurrent neural network model
with soft visual attention that learns to generate
LATEX markup of real-world math formulas from
their images. Applying neural sequence generation techniques that have been very successful in
the fields of machine translation and image/handwriting/speech captioning, recognition, transcription and synthesis; we construct an image-tomarkup model that learns to produce syntactically
and semantically correct LATEX markup code of
over 150 words long and achieves a BLEU score
of 89%; the best reported so far for the Im2Latex
problem100 . We also visually demonstrate that
the model learns to scan the image left-right /
up-down much as a human would read it.

1. Introduction
In the past decade, deep neural network models based on
RNNs1 and/or CNNs2 have been shown to be very powerful
sequence modelers. Their ability to model joint distributions of real-world data has been demonstrated through
remarkable achievements in a broad spectrum of generative tasks such as; image synthesis (van den Oord et al.,
2016b;c; Salimans et al., 2017; Theis & Bethge, 2015),
image description (Karpathy & fei Li, 2015; Xu et al.,
2015; Johnson et al., 2016; Pedersoli et al., 2016; Vinyals
et al., 2015), video description (Donahue et al., 2015),
speech and audio synthesis (van den Oord et al., 2016a),
handwriting recognition (Graves & Schmidhuber, 2008;
Bluche, 2016), handwriting synthesis (Graves, 2013), machine translation (Cho et al., 2014; Bahdanau et al., 2014;
Kalchbrenner et al., 2016; Sutskever et al., 2014), speech
1

Saratoga, California, USA. Correspondence to: Sumeet Singh
<sumeet@singhonline.info>.
100

https://openai.com/requests-for-research/#im2latex
Recurrent Neural Network
2
Convolutional Neural Networks and variants such as dilated
CNNs (Yu & Koltun, 2015)

recognition (Graves et al., 2006; Chan et al., 2015; Graves
et al., 2013), etc. (Graves, 2008)
One class of sequence models employ the so-called encoderdecoder (Cho et al., 2014) or sequence-to-sequence
(Sutskever et al., 2014) architecture, wherein an encoder
encodes a source sequence into feature vectors, which a decoder employs to produce the target sequence. The source
and target sequences may either belong to the same modality
(e.g. in machine translation use-cases) or different modalities (e.g. in image-to-text, text-to-image, speech-to-text);
the encoder / decoder sub-models being constructed accordingly. The entire model is trained end-to-end using
supervised-learning techniques3 . In recent years, this architecture has been augmented with an attention and alignment model which selects a subset of the feature vectors
for decoding. It has been shown to help with longer sequences (Bahdanau et al., 2014; Luong et al., 2015). Among
other things, this architecture has been used for imagecaptioning.We employ a variant of this architecture to map
images of math formulas into LATEX markup code. The
contributions of this paper are: 1) Solves the Im2Latex
problem100 and achieves the best reported BLEU score.
2) Pushes the limits of the neural encoder-decoder architecture with visual attention. 3) Analyses variations of the
model and cost function. Specifically we note the changes
to the base model (Xu et al., 2015) and what impact those
had on performance.
1.1. The I M 2L ATEX Problem
The I M 2L ATEX Problem is a request for research proposed
by OpenAI. The challenge is to build a Neural Markup
Generation model that can be trained end-to-end to generate
the LATEX markup of a math formula given its image. Data
for this problem (see section 2.5) was produced by rendering
single-line real-world LATEX 2ε formulas obtained from the
KDD Cup 2003 dataset. The resulting grayscale images
were used as the input samples while the original markup
was used as the label/target sequence. Each training/test
sample s, is comprised of an image x(s) - the input - and
a corresponding target markup-sequence y (s) - the output

1

3

generally backprop or reinforcement learning

Neural Markup Generation with Visual Attention
ŷt

X 1
X 1
X 1
a
Tr φal φa−l +
Tr fla f−l
+
S0 =
Tr ψ̄ra ψra .
2
2
2∆l
2l
g
r
r

l } ˆ { 2 } } \mathrm { T r } \, f
\frac { 1 } { g

{ l } ˆ { 2 } } \mathrm { T r } \, \

{ − l } ˆ { a } + \sum { l } \frac { 1 } { 2 \epsilon
{ l }ˆ {a }f

{

{ − l } ˆ { a } + \sum { r }

{ r } } \mathrm { T r } \, \bar { \psi }

{ r } ˆ { a } \psi

{r

} ˆ { a } \, .

S

{ 0 } = \sum { l } { \frac { 1 } { 2 \Delta

} \, \phi
epsilon

{ l } ˆ { a } \phi

\sum { r } { \frac { 1 } { g
psi

{ l }ˆ {a }f

{ r } } } \mathrm { T r } \, \psi

{−i}ˆ {a }+
{r }ˆ {a }\

Ht
Ct

LSTM-Stack
Attention

Init State Model

Embedding

a

{ l } ˆ { 2 } } } \mathrm { T r

{ − l } ˆ { a } + \sum { l } { \frac { 1 } { 2 \

{ l } ˆ { 2 } } } \mathrm { T r } \, f

yt-1, Ct-1

Deep Output Layer

CALSTM

{ l } ˆ { a } \phi

Decoder RNN

{ 0 } = \sum { l } \frac { 1 } { 2 \Delta

Encoder

phi

pt

l

l

S

Beam Search or CTC Decoding

{ r } ˆ { a } \, .

Pooling
Á
CNN
I

Figure 1. A training sample: At the top is the input image x, middle the target sequence y (τ = 145) and bottom the predicted
sequence ŷ (τ = 148). Each space-separated word in y and ŷ ∈
V
(s)

- of length τ (s) . Each word of the sequence yt , belongs
to the vocabulary of the dataset plus two special tokens,
begin-of-sequence ‘bos’ and end-of-sequence ‘eos’ (Figure
1). Denoting image dimensions as HI , WI and CI and
the vocabulary as a set V of K words and dropping the
superscript (s) for brevity, we represent:
x := Array(HI , WI , CI )

(1a)

y := {y 1 , . . . , y τ } ;

(1b)

yt ∈ V

V := {dataset vocabulary, eos, bos}; |V | = K (1c)
The task is to generate markup that a LATEX 2ε compiler will
render back to the original image. Therefore, our model
needs to generate syntactically and semantically correct
markup, by simply ‘looking’ at the image: i.e. it should
jointly model vision and language.

2. Image to Markup Model
Our model (Figure 2) has the same basic architecture as Xu
et al. (2015) in the way the encoder, decoder and a visual
attention interact. 4
2.1. Encoder
I MAGE P REPROCESSING : All images are standardized to
the same size by centering the text and padding with white
pixels. Then they are linearly transformed (whitened) to lie
in the range [-0.5,0.5].
A deep CNN then encodes the whitened image into a rectangular visual feature map Á, of H́ × Ẃ visual feature
4
However there are significant differences in the details which
we notate in the remainder of this paper and in section A.2.

Figure 2. Model outline showing major parts of the model. Beam
search decoder is only used during inferencing, not training.
LSTM-Stack and Attention model jointly form a Conditioned
Attentive LSTM stack (CALSTM) which can itself be stacked. 3
shows more details of the decoder.

vectors a(h́,ẃ) of size D́. H́ and Ẃ are the visual feature
map’s dimensions along the height and width axes of the
image respectively. Table 1 shows the configuration of the
Encoder CNN. All convolution kernels have shape (3,3),
stride (1,1) and tanh non-linearity, whereas all maxpooling
windows have shape (2,2) and stride (2,2). See section A.1
for further comments around this architecture.
Layer
Input (Image)
Convolution
Maxpool
Convolution)
Maxpool
Convolution
Maxpool
Convolution
Maxpool
Convolution
Maxpool

Output Shape
128 × 1088
128 × 1088
64 × 544
64 × 544
32 × 272
32 × 272
16 × 136
16 × 136
8 × 68
8 × 68
4 × 34 = (H́ × Ẃ )

Channels
1
64
64
128
128
256
256
512
512
512
512 = (D́)

Table 1. Specification of the Encoder CNN.

Next, the visual feature vectors are pooled together in rectangular blocks of shape [SH , SW ]; resulting in pooled feature
vectors a(h,w) of size D = D́ · SH · SW . SH and SW
are strides by which the pooling operation moves along
the height and width of Á. Therefore, shape of the overall feature map reduces to [H, W ] where H = H́/SH and

Neural Markup Generation with Visual Attention

Model
I 2 L - NOPOOL
I 2 L - STRIPS

Pooling
Stride
[1,1]
[4,1]

Pooled
Map Shape
[4,34]
[1,34]

Num
Parameters
61,235,049
70,984,635

we define:
pt : {1, . . . , K} → [0, 1]
yt ∼ p t

(3)

Pr (y t |y <t , a) = pt (yt )

Table 2. Pooling configuration of two model variants.

Probability of the output sequence for image a follows:
W = SW /Ẃ :

Pr (y|a) =

τ
Y

Pr (y t |y <t , a) =

t=1



 a(1,1)
..
A :=
.


a(H,1)

...
..
.
...


a(1,W ) 

..
. ;

a(H,W )

a(h,w) ∈ R

D

Pooling allows varying the receptive fields of the feature
vectors by changing the stride. We share results of two
models, one with stride [1,1] (i.e. no pooling) and the other
[4,1] (Table 2). Each pooled feature vector is a window
into a distinct spatial region of the image; i.e. the rectangle
projected by its receptive field.5 The idea behind this is
to partition the image into spatially localized regional encodings and setup a decoder architecture (Section 2.2) that
selects/emphasizes only relevant regions at a given time-step
t, while ignoring/de-emphasizing the rest. Bahdanau et al.
2014 showed that such piecewise encoding enables modeling longer sequences as opposed to models that encode
the entire input into a single feature vector (Sutskever et al.,
2014; Cho et al., 2014). 6 We represent A as a flattened
sequence a of pooled feature vectors al (Equation 2):

a := a1

...

aL ;

al ∈ RD

pt (y t )

(4)

t=1

The decoder (Figure 3) is an LSTM (Hochreiter & Schmidhuber, 1997) based model with an internal state C t that
holds relevant information (features) regarding the output
sequence unfolded thus far, the image regions attended to
and an initial state generated at t = 0. In addition, it receives the previous word y t−1 and encoded image a as
inputs. This architecture enables it to condition its output on
the entire previous sequence and the image. Representing it
as a function fD , we have:
fD : {a; y t−1 ; C t−1 } → {pt ; C t }

(5)

It is comprised of the following sub-models (Figure 3):
pt

DRNN

Deep Output Layer
zt

Ht
CALSTM
LSTM Stack
ht-1Q, ct-1Q

h tQ
c tQ

LSTMQ

Ct-1

Ct
h t1

ht-11, ct-11

(2)

c t1

LSTM1

L=HW ; l=H(h−1)+w

zt
a

βt

2.2. Decoder
The decoder is a Recurrent Neural Network (RNN) that
models the discrete probably distribution of the output word
y t , conditioned on the sequence of previous words y <t and
the encoded image a7 . Denoting the decoder’s output by pt

τ
Y

ht-1Q

C0

αt

Attention Model

Init State Model

E

a

5

Neighboring regions overlap but each region is distinct overall.
That said, Bahdanau et al. 2014 employ a bidirectional-LSTM
(Graves, 2008) encoder whose receptive field does encompass the
entire input anyway! (Although that does not necessarily mean
that the bi-LSTM will encode the entire image). Likewise Deng
et al. 2017 who also solve the I M 2L ATEX problem also employ a
bi-directional LSTM stacked on top of a CNN-encoder in order to
get full view of the image. In contrast, our visual feature vectors
hold only spatially local information which we found are sufficient
to achieve good accuracy. This is probably owing to the nature
of the problem; i.e. transcribing a one-line math formula into
LATEXsequence requires only local information at each step.
7
This is now a very standard way to model sequence probabilities in neural sequence-generators. See (Sutskever et al., 2014) for
example.
6

Figure 3. Schematic of Decoder RNN showing its sub-models.
There are three nested RNN cells in all: The decoder RNN (DRNN)
at the top level, nesting the CALSTM which nests the LSTM-Stack.
The Init Model does not participate in recurrence, therefore its is
shown outside the box.

1) A LSTM-Stack responsible for memorizing C t and
producing a recurrent activation H t .
2) A Visual Attention and Alignment model responsible
for selecting relevant regions of the encoded image for
input to the LSTM-Stack.

Neural Markup Generation with Visual Attention

N OTE : The LSTM-Stack and Visual Attention and Alignment model jointly form a Conditioned Attentive
LSTM (CALSTM); H t and C t being its activation and internal state respectively.8
3) A Deep Output Layer (Pascanu et al., 2013) that produces the output probabilities pt .
4) Init Model: A model that generates the initial state C 0 .
5) An embedding matrix E (learnt through training) that
transforms y t into a dense representation ∈ Rm .
2.2.1. I NFERENCING
After the model is trained, the output sequence is generated
by starting with the word ‘bos’ and then repeatedly sampling
from pt until ‘eos’ is produced. The sequence of words thus
sampled is the predicted sequence ŷ (Figure 2).
ŷ t ∈ RK

ŷ := {ŷ 1 , . . . , ŷ τ̂ } ;

(6)

For this procedure (also called ‘decoding’9 ) we use beam
search decoding with a beam width of 10.
2.2.2. V ISUAL ATTENTION AND A LIGNMENT M ODEL
As previously alluded, the decoder selects/emphasizes relevant (encoded) image regions at each step. This is implemented via. a ‘soft attention’ mechanism10 which computes a weighted mean z t (equation 9) of the pooled feature
vectors al . The visual attention model fatt , computes the
weight distribution αt .
αt := {αt,1 , . . . , αt,L }
0 ≤ αt,l ≤ 1 ;

(7)
L
X

αt,l = 1

l

αt = fatt (a; H t−1 )

(8)

z t = αt · a

(9)

fatt is modeled by an MLP (Table 3):
Layer
3 (output)
2
1

Num Units
L
max(128, L)
max(256, L)

While there is a potential for αt to end up in a flat distribution over {a1 . . . aL }, in practice we see that most of its
mass gets concentrated on a 2-4 region neighborhood (see
Figure 4). That is, the model learns to focus its attention
on small focal-regions of the image, tending towards the
‘hard attention’ formulation described by (Xu et al., 2015);
except in this case the focal-regions are learnt and therefore adaptive to the dataset and problem. Also, notice that
for a given sample (Figure 4), the attention model is able
to utilize the extra granularity of feature-map available in
the I2L-NOPOOL case and consequently generates a much
smaller focal-region than I2L-STRIPS. It would be interesting to experiment with an even more granular image-map
to see if the attention model refines its focal-region even
further.
Furthermore, the model aligns its attention with the word it
is producing and scans the image left-to-right (I2L-STRIPS)
or left-right/up-down (I2L-NOPOOL) just like a human
would read it (Figure 5).
2.2.3. LSTM S TACK

Activation
softmax
tanh
tanh

Table 3. Specification of the Visual Attention Model MLP. L = 34
for I2L-STRIPS and and 136 for I2L-NOPOOL.
8

Figure 4. Focal-regions learnt by the attention model: at the top by
I2L-STRIPS and at the bottom by I2L-NOPOOL. Image darkness
is proportional to αt . The focal-regions are 3-4 blocks big and
surround the symbol (∆) whose markup is being produced at that
step. Also, the model is able to utilize the finer granularity of I2LNOPOOL and produces a tighter focal region than I2L-STRIPS.
More samples at our website.

Our source-code implements the CALSTM as a RNN cell
which may be used as a drop-in replacement for a RNN cell.
9
The term ’decoding’ is borrowed from Hidden Markov Models. For further details see for e.g. (Graves, 2008)
10
‘soft’ attention as defined by Xu et al. (2015) and originally
proposed by (Bahdanau et al., 2014)

The core sequence-generator of the decoder is a multilayer
LSTM (Hochreiter & Schmidhuber, 1997; Graves, 2013)
(Figure 3). Our LSTM cell implementation (Figure. 6)
follows Graves et al. (2013):
it = σ (Wxi xt + Whi ht−1 + Wci ct−1 + bi )
f t = σ (Wxf xt + Whf ht−1 + Wcf ct−1 + bf )
ct = f t ct−1 + it tanh (Wxc xt + Whc ht−1 + bc )
ot = σ (Wxo xt + Who ht−1 + Wco ct + bo )
ht = ot tanh(ct )
it , f t , ot , ct , ht ∈ Rn

(10)

Neural Markup Generation with Visual Attention

Output Gate

Input Gate

Cell

Forget Gate

Figure 6. LSTM Cell

where σ is the logistic sigmoid function, and it , f t , ot , ct
and ht are respectively the input gate, forget gate, output
gate, cell and hidden activation vectors of size n. The LSTM
cells are stacked in a multi-layer configuration (Zaremba
et al., 2014; Pascanu et al., 2013) as below:
LST M q :

{xqt ; hqt−1 ; cqt−1 } → {hqt ; cqt } (11)
1 ≤ q ≤ Q;

xqt = hq−1
t
x1t

hqt , cqt ∈ Rn

; q 6= 1

= {z t ; Ey t−1 }

(12)
(13)

LST M q is the LSTM cell at position q with xqt , hqt and cqt
being its input, hidden activation and cell state respectively.
LST M 1 receives the stack’s input: soft attention context
z t and previous output word Ey t−1 . LST M Q produces
the stack’s output H t = hQ
t , which is sent up to the Deep
Output Layer. Accordingly, the stack’s activation (H t ) and
state (C t ) are defined as:
H t := hQ
t
Q
1
C t := {c1t , . . . , cQ
t , ht , . . . , ht }

(14)

We do not use skip or residual connections between the cells.
Both of our models have two LSTM layers with n = 1500.
More discussion on this model can be found in section A.3.
2.2.4. D EEP O UTPUT L AYER
We use a a Deep Output Layer (Pascanu et al., 2013) to
produce the final output probabilities pt :
pt = fout (H t ; z t ; Ey t−1 )

Figure 5. Attention scan. The movement of focal-region is aligned
with the output word (above the image). Continued in Figure 11.

(15)

fout is modeled by an MLP (Table 4). Note that the output
layer receives skip connections from the LSTM-Stack input
(Equation 13). More discussion on this model can be found
in section A.4.

Neural Markup Generation with Visual Attention

Layer
3 (output)
2
1

Num Units
K
max(358, K)
max(358, K)

Activation
softmax
tanh
tanh

through time :

Table 4. Configuration of the Deep Output Layer MLP. K = 339
and 358 for I2L-140K and Im2latex-90k datasets respectively.

1
(17)
J = − log (Pr (y|a)) + λR R + λA A
τ
1
= − log (Pr (y|a)) + λR R
; λA = 0
τ
1X 2
R =
θ
(17a)
2
θ

2.2.5. I NIT M ODEL
The Init Model finit , produces the initial state C0 of the
LSTM-Stack, based on the pooled feature vectors:
Q
1
finit : {a} → {c10 , . . . , cQ
0 , h0 , . . . , h0 }

hq0 , cq0

∈R

(16)

n

A = (ASE N − ASE T )
100
ASE N = 2 L−1  · ASE
τ
L
L 
X
τ 2
ASE =
αl −
L
αl :=

l=1
τ
X

αt,l

(17b)
(17c)
(17d)
(17e)

t=1

finit is modeled as an MLP with common hidden layers
and 2Q distinct output layers, one for each element of C0 ,
connected as in Figure 7 and specified in Table 5. That
said, since it only provides a very small improvement in
performance in exchange for over 7 million parameters,
its presence could be questioned (see Section A.5). The

FC1

FC2Q
Hidden MLP

Figure 7. Init Model. FC = Fully Connected Layer.

Layers

Num

Units

Output
Hidden

2Q
1

n
100

Activation
Function
tanh
tanh

Table 5. Init Model layers.

idea behind the init model is to setup the decoder to extract
relevant features from the image (if any) before it starts
generating the sequence.
2.3. Training
The entire model was trained end-to-end by minimizing the
objective function J (Equation 17) using back propagation

The first term in equation 17 is the average (per-word) log
perplexity of the predicted sequence11 and is the main objective. R is the L2-regularization term, equal to L2-norm
of the model’s parameters θ (weights and biases). A is
an optional penalty term intended to bias the distribution
of αl (Equation 17e) - the amount of P
attention placed on
L
image-location l. Observe that while l αt,l = 1, there
is no constraint on how the attention is distributed across
the L locations of the image. The term λA A serves to
steer the variance of αl by penalizing any deviation from
a desired value. ASE (Alpha Squared Error) is the sum
of squared-difference between αl and its mean τ /L; and
ASEN is its normalized value 12 ∈ [0,100]13 . Therefore
ASEN ∝ ASE ∝ σα2 l . ASET which is the desired value
of ASEN , is a hyperparameter that needs to be discovered
through experimentation14 . λR and λA are also hyperparameters that need to be tuned. See section A.2 for more
discussion around this topic.
For experimentation we split the dataset into two fixed parts:
1) training dataset = 90-95% of the data and 2) test dataset
5-10%. At the beginning of each run, 5% of the training
dataset was randomly held out as the validation-set and the
remainder was used for training. Therefore, each such run
had a different training/validation data-split, thus naturally
cross-validating our learnings through the duration of the
project. We trained the model in minibatches of 56 using the
ADAM optimizer (Kingma & Ba, 2014) for updating param11
Also variously known as cross-entropy, negative loglikelihood or negative log-probability

12
It can be shown that τ 2 L−1
is the maximum theoretical
L
value of ASE
13
We normalize ASE so that it may be compared across batches,
runs and models
14
Start with ASET = 0, observe where ASEN settles after
training, then set ASET to that value and repeat until approximate
convergence.

Neural Markup Generation with Visual Attention

eters; periodically evaluating it over the validation set15 . For
efficiency, we batched the data such that each minibatch had
similar length samples. For the final evaluation however, we
fixed the training and validation dataset split and retrained
our models for about 100 epochs (2 12 days on I2L-140K
dataset). We then picked the model-snapshots with the best
validation BLEU score and evaluated the model over the
test-dataset for publication. We executed on two Nvidia
GeForce 1080Ti graphics cards in a parallel towers configuration. Our implementation102 uses the Tensorflow toolkit
and is distributed under AGPL license.
2.4. Results
Given that there are multiple possible LATEXsequences that
will render the same math image, ideally we should perform
a visual evaluation. However, since there is no widely accepted visual evaluation metric, we report corpus BLEU
(1,2,3 & 4 grams) and per-word Levenstein Edit Distance16
scores (see table 7). We also report a (non-standard) exact
visual match score103 which reports the percentage of exact visual matches, discarding all partial matches. While
the predicted and targeted images match in at least 70%103
of the cases, the model generates different but correct sequences (i.e. y 6= ŷ) in about 40% of the cases (Figure 8).
For the cases where the images do not exactly match, the
differences in most cases are minor (Figure 9). Overall,
our models produce syntactically correct sequences17 for at
least 99.85% of the test samples (Table 7 and Figure 10).
2.5. Datasets
Datasets were created from single-line LATEX math formulas
extracted from scientific papers and subsequently processed
as follows: 1) Normalize the formulas to minimize spurious ambiguity.18 2) Render the normalized formulas using pdflatex and discard ones that didn’t compile or render
successfully. 3) Remove duplicates. 4) Remove formulas with low-frequencey words (frequency-threshold = 24
for Im2latex-90k and 50 for I2l-140K). 5) Remove images
15

Evaluation cycle was run once or twice per epoch and/or
when a training BLEU score calculated on sequences decoded
using CTC-Decoding(Graves et al., 2006) jumped significantly.
102
Source Code: https://github.com/untrix/im2latex
16
i.e. edit distance divided by number of words in the target
sequence.
17
i.e. those that were successfully rendered by LATEX 2ε
103
We use the ’match without whitespace’ algorithm provided by
Deng et al. (2017) wherein two images count as matched if they
match pixel-wise discarding white columns and allowing for upto
5 pixel image translation (a pdflatex quirk). It outputs a binary
match/no-match verdict for each sample - i.e. partial matches
however close, are considered a non-match.
18
Normalization was performed using the method and software
used by (Deng et al., 2017) which parses the formulas into an AST
and then converts them back to normalized sequences.

bigger than 1086 × 126 and formulas longer than 150. Processing the Im2latex-100k dataset104 (103559 samples) as
above resulted in the Im2latex-90k dataset which has 93741
samples. Of these, 4648 were set aside as the test dataset
and the remaining 89093 were split into training (95%) and
validation (5%) sets before each run (section 2.3). We found
the Im2latex-90k dataset too small for good generalization
and therefore augmented it with additional samples from
KDD Cup 2003. This resulted in the I2L-140K dataset
with 114406 (training), 14280 (validation) and 14280 (test)
samples. Since the normalized formulas are already space
separated token sequences, no additional tokenization step
was necessary. The vocabulary was therefore produced by
simply identifying the set of unique space-separated words
in the dataset. Both datasets are available at our website105
and the data processing code is available in our source-code
repository102 .

References
Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua.
Neural machine translation by jointly learning to align
and translate. CoRR, abs/1409.0473, 2014.
Bluche, Théodore. Joint line segmentation and transcription
for end-to-end handwritten paragraph recognition. In
NIPS, 2016.
Bluche, Théodore, Ney, Hermann, and Kermorvant, Christopher. A comparison of sequence-trained deep neural
networks and recurrent neural networks optical modeling
for handwriting recognition. In SLSP, 2014.
Chan, William, Jaitly, Navdeep, Le, Quoc V., and
Vinyals, Oriol. Listen, attend and spell. CoRR,
abs/1508.01211, 2015. URL http://arxiv.org/
abs/1508.01211.
Cho, Kyunghyun, van Merrienboer, Bart, aglar Gülehre,
Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger,
and Bengio, Yoshua. Learning phrase representations
using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014.
Deng, Yuntian, Kanervisto, Anssi, Ling, Jeffrey, and Rush,
Alexander M. Image-to-markup generation with coarseto-fine attention. In ICML, 2017.
Donahue, Jeff, Hendricks, Lisa Anne, Guadarrama, Sergio,
Rohrbach, Marcus, Venugopalan, Subhashini, Saenko,
Kate, and Darrell, Trevor. Long-term recurrent convolutional networks for visual recognition and description.
CoRR, abs/1411.4389, 2015.
104
105

Im2latex-100k dataset is provided by (Deng et al., 2017)
Paper’s Website: https://untrix.github.io/i2l/

Neural Markup Generation with Visual Attention

0
1
2
3
4

Rendered Sequence
+q
+q
1−
+2i 1−
+2i 1−
1 i
T+2−2
= 12 γqi ṗ (Ω+2i
+2 ψ−2ṗ −Ω−2 ψ+2ṗ ), T+p±2 = 2 γq ṗ Ω+p ψ±2ṗ ,
i − + − i (p+ x−− k+ y − )
+
dk+ − 2 P x
dP − dp
e 2
σij (p+ ,k+ ;P − ),
4π
4π
4π e
(n+1) ˜(1)
(n−m) ˜(m+2)
(n−2)
(n−m) ˜(m)
(n)
n
,f }(φ)
,f
}(φ) +{Θ̃β
,f
}(q) +Σm=0 {Θ̃β
G(f )β =Σm=0 {Θ̃β
P 1
P 1
P 1
a a
a a
a a
S0 = l
Tr φl φ−l + l 2 Tr fl f−l + r gr Tr ψ̄r ψr .
2
2∆
2
l
l
2 )(t2 −r 2 )
(t2 +r−
2
r r
+
dr 2 .
ds2 =− 2 2 t 2 2 dt2 +t2 (dφ+ +t2 − dr)2 +
t2
(t +r )(t −r )
−
+
−

−

+

R
σij (x ,y ;x )=



0
0
0

5


1
H= 2E
U


6

ab
Dµν
(p,p3 )=

7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

0
∆m221
0

0
0
∆m231





a
η∗ b
0

 † 1 

U +
2E 


ηb 0
η0 b 0
0 0


,




)

(

)

(

(

(

)

(

p̃2 ·k
2

)

(

p̃3 ·k
2

)}

)

)

sin
sin
sin
=− 14 (sin p̃1 ·k+sin p̃2 ·k+sin p̃3 ·k)


~2
P0 /κ
[P0 ,X0 ]=i, [Pi ,Xj ]=−i δij 1− P
eP0 /κ , [P0 ,Xi ]=− 2i
κ Pi e
κ2
a1
a1 ...an
an
n NT 2
1
hJµ (P1 )...Jµn (Pn )iT ≈(−i) 12 δΓµ ...µn (P1 ,...,Pn )+O( f 2 ) ,
1
1
π 

|~
p|2
ũπ (~
k10 )† ũπ (~
k)c2 (~
k)+ 13 ~
k2 b1 (~
k)2 .
k1 )=Nπ2 cos2 χ(~
k)− 4 cosχ(~

(

)

P1

τ0 (y)= σ =0 .. σ =0 Yσ(0)..σ (t1 Z1 )σ1 (t2 Z2 )σ2 (t5 Z5 )σ3 (t8 Z8 )σ4
1
4
4
1
sin(2β)
sin(2β)
I(Ω̃)→ 32 π2 (I(Ω̃)+c22 c23 δ1 I(MG ,MG1 )+c22 s23 δ2 I(MG ,MG2 ))
32 π 2
c
W=Ye Lj E c H1i ij +Yd Qja Da
H1i ij +Yu Qja Uac H2i ij +µH1i H2j ij
µ
−iκ̄(α) γ κ(β) ∂µ =−iC̃αγ Γs (T I )γ β ΓAdj (u−1 )a I ea =−iC̃αγ Γs (T I )γ β k(I)
P1

155

150

151

150

150

145

148

147

147

147

147



{

p̃1 ·k
2

ŷ len

147



δ ab δ a3
−gµν +pµ pν (1−δp3 ,0 ) 12 +δp3 ,0 (1−ξ) p2 1+i
p2 −p2
p3
3 +i
2
V (H1 ,H2 )= 18 (g22 +g12 ) |H1 |2 −|H2 |2 +m21 |H1 |2 +m22 |H2 |2 −m23 (H1 H2 +h.c.)
(1)
µ
(3)
ν
λν
µν
λ
A30 (α0 →0)=2gd ελ ε(2)
η λµ (pν
pµ
pλ
.
µ εν
1 −p2 )+η
2 −p3
3 −p1 +η
†
†
†
l
∗
∗
∗
ν
∗
νT
ν
∗
UL Ml UR =Ml , UL ML UL =ML , UL MD UR =MD , UR MR UR =MR
.
√
1 µ1 µ2 ...µd−p ν1 ν2 ...νp
−gg µ1 ν1 g µ2 ν2 ···g µd−p νd−p F̃ν1 ν2 ...νd−p = p!

Fν1 ν2 ...νp ,


2
dEel
dErad
C2 αs 2
2
dE
3ET
9E
s
≈ π µ ln 2µ2 ln π3 T + 3π2µα
.
2 T
dz = dz +
dz
P
|h|
|h|
†
†
†
f ree
˜† ˜
L0 =(2n+1) 2 +|h|d†0 d0 − 2 −|h| ∞
.
k=1 dk dk −dk dk +ak ak −bk bk +L0
g
a
Q7γ = 8πe 2 mb q̄α σ µν (1+γ5 )bα Fµν , Q8G = 8π2 mb q̄α σ µν ta
b
G
,
(q=d
or
s).
αβ β µν


u2 h(u) γA
2
0
2
γC
2
γB
2
2 γD
2
u2
R2
ds = α
e dx0 + R2 e dxi + u2 h(u) e du +R e dΩ5 ,
R2

(

y len

,

139

145

144

145

146

145

149

145

147

145

130

144

149

144

141

143

143

143

133

143

139

143

143

143

142

142

143

142

142

142

141

141

145

141

150

141

140

140

25

i
hP
P
φi φ̄j
Leff = i,j 21 Kφi ,φ̄j ∇φi ·∇φ̄j −eK
(Dφi W )(Dφ̄j W )† −3|W |2
i,j K
R d4 pn
P6
0
T (l) = (2π)
Aj Mj )S(pp )Γd CS(pn )TY N S(pY )}u(p0N ) .
4 uY (pY ){(
 j=1




2
2
C
(π )
(0)
α (µ2 )
παs (µ2 )
CF − 2A + s π CF 29 + π3
.
σqq̄ (s,m2 )=σqq̄ (s,m2 )
2β

140

140

26

dx̄
21 dt2

140

139

27

−iδ ab
Gab
µν (p)= (2π)2ω (p2 +i)

139

139

140

138

138

138

138

138

138

138

133

138

23
24

28
29
30
31
32
33
34
35
36
37
38
39

+(2β0 x̄0 (t))x̄2 =−β2 x̄40 (t)−3β1 x̄20 (t)x̄1 (t)−β0 x̄21 (t), x̄2 (0)=0,

h

p n +p n
gµν − µ νp·n ν µ

n 2 pµ pν
+ (p·n)
2

i

2

, n >0, epsilon>0,


R 2
(1+16θ 2 g 4 )
2
S̃= d x g 2 (1+ϕ21 )2 (∂ ϕ˜0 )2 +8θg 2 ∂a ϕ˜0 ∂ a ϕ1 + 2
2 2 (∂ϕ1 )

g (1+ϕ1 )
p21 p22 −(p1 ·p2 )2 =− 41 (p41 +p42 +p45 −2p21 p22 −2p21 p25 −2p22 p25 )=0
−1
−1
−1


gij,kl = −q[2] Dsi R̂js,kl
, (g= −q[2] D1 P R̂12
= −q[2] im R̂jm,kt −1
) ,
lt
P
4GF
4GF
(u)
T
∗
2
P
∗ P6
√
√
H = 2 Vub Vuq i=1 ci (µ)Oi ,
H =− 2 Vtb Vtq i=3 ci (µ)Oi ,
m2
m̃N
G
0
W ē κ
LN (S)= √F2
(1+γ
)
+
κ
µe
5
µe (1−γ5 ) µ mW Ψ̄N ΨN ,
m2
S
ābi eµ
(abi ābi −hi )|ai=0 , ābi ābi |ai=0 , εij abi ābj |ai=0 ,
b DµL |ai=0 ,
2
a
dψ4
2
2
2
2
2
2
1 d
− ρ+
2 ψ4 −g1 η ψ4 (ψ4 +ψ6 −1)−g2 η ψ4 (ψ4 −ψ6 −c)=0,
ρ dρ ρ dρ
dg 2 dz2
(g2 dg 2 −g 2 dg2 )
i 1
A= 2 { 2 d ln( dz dg )+
},
(1−g2 g 2 )
2
2
e
d
u
W =Yab
L̂a Ĥ1 Êbc +Yab
Q̂a Ĥ1 D̂bc +Yab
Q̂a Ĥ2 Ûbc −µĤ1 Ĥ2

[

(

]

)

A 0
<p0 |Jµ
|p >=ψ p0 [g10 (k2 )γµ γ5 +g20 (k2 )iσµν γ5 kν +g30 (k2 )iγ5 kµ ]ψp0 .
G
∗ P2
∗ P10
Heff = √F
) i=1 (ci Qci )−(Vtb Vtd
) i=3 (ci Qi )] + h.c. ,
[(Vcb Vcd
2

G0 = 4π21x2 →

Γ( d )
2
d
(d−2)2π 2

( d2 −1)


1
x2

Γ(1+)
π

138
137

131

137

136

136

136

136

135

136

142

136

(1+)


= 4π1 2

138
137

1
x2

Figure 8. A sample of correct predictions by I 2 L - STRIPS. We’ve shown the long predictions hence lengths are touching 150. Note that at
times the target length is greater than the predicted length and at times the reverse is true (though the original and predicted images were
identical). All such cases would evaluate to a less than perfect BLEU score or edit-distance. This happens in about 40% of the cases. For
more examples visit our website.105

Neural Markup Generation with Visual Attention

y

ŷ

0

Ψ:S̃ 2 →{M1 ,M2 }

Ψ:S̃ 2 →{M1 ,M2 }

1

ln

2

( r`s0 )

3

ṗa +ba pb ωµ ẋIµ=0

4

α,β
α ,β
α,β
α ,β
[M(m)
,M(n)
]≡M(m)
M(n)
−(−)αβ

5

T̂ =V̂ +t̂ĜV̂ ,

E+

√ 2 2
E −m −m
√ 2 l2 π .
E −m −mπ
l

E−

d̃

ln

0

0

0 +βα0 2i(β 0 m−βn)x

e

0

R

9

13
14
15
16
17
18
19
20

)

(

22
23
24
25
26
27
28
29
30
31
32
34

(

{

)

}

mu , md  ms mc , mb , mt
|
{z
} | {z }

0

0 +βα0 2i(β 0 m−β−n)x

e

{

eα̇ ),
∂A =(∂α ,∂

P

perms (−1)
B
∂A ,ΘB =δA

p

0

0

α ,β
α,β
M(n)
M(m)

γ a1 ...γ ap

}

pele
mvot −kezτ (mr
)|
Ŷτ (MZ )|DG = rδ(M )|DEcsβ(M DH
Z) i
hZ
2
3.4β [2|2] (y)=−9y 2 1−22.2y+36.93y
1−28.2y+143.2y 2
α
b α β = Π2 +2eBS3
D
β
R dz̃dzd̄bd
V (f )=0
n
P
(s0 )
L 2 =G8 F 2 37
i=1 Ni Wi
∆S =1
Q
(0)
det(M (N0 ))=αQ (0) r,s [α(h−hr,s )]Pf (N0 −rs /K)

(

)

r2

2 2

β++ = 4M 2 −M y + 12
e2 N
2T

h

M2
M2

.

,

i
p0
2
(1−y)−1 rs+ +M 2 M
2s ,
¯ A X.
QA (ζ)X≡{Sm (ζ),X}A −i~∆

.

1−

ΩG ϕ
η= 8π
,
mu md <ims
ukakt raras

λ4r (T0 )
9 λ (T ) ln(T /T )
0
8π 2 tx 0

lav ,mb .mt dear)

.

quars

.

heavy quarks

+n−1/2

Pn


21 )
(Hν)12 = r1 eiϕ1 = 13 ei(θ11 −θ
+ ei(θ12 −θ22 ) + e

a
−2 a
−1 a
eb l T 
A  Rb − l
R̄B
=
,
−l−1 Tb
0
Tr [Φ+ (~
x)ρ(t)]
Tr [Φ− (~
x)ρ(t)]
=
Tr ρ(0)
Tr ρ(0)

j
η−1
1
(z1 ,...,ẑj ,...,zn ),
j=1 (−1) z−z −1 ψ
j
2
2
2
M (1−sin 2β) sin θW
cos θ
δ2 = Z 2µ
+ (µ+MW) ]
(µ+M1 )
2
1 i(θ11 −θ21 ) i(θ12 −θ22 ) i(θ13 −θ23 )
iϕ1
i(θ13 −θ23
)
+e
+e
(Hν )12 =r1 e
=3 e

,




A 
R̄B
=

φ(~
x,t)≡
αβ

ω = {η α , η β } = −J αβ
χαβ = Jαβ .
√ ω a g a
√ ω a g a
a†
αa
i=
2~g [q̂i +i ω p̂i ] , αi =
2~g [q̂i −i ω p̂i ]

Pn

[
Rba − l−2 eab
−l−1 Tb

l1 Taa
0

]


,

Tr [Φ+ (~
x)ρ(t)]
Tr [Φ− (~
x)ρ(t)]
=
Rr ρ(0)
Rr (0)

,

ω = 1η α , η β } = −J αβ
χαβ = Iαβ .
√ ω a g a
a† √ ω
g a
a
αa
i=
2~g [q̂i +i ω µ̂i ] , αi =
2~g [q̂i −i ω p̂i ]

,
,

 1
 10o−45−1  1
 19α−45−4γ  1
 −83−4 γ  1
4
1+ 3
2
α+2
α+2
Λ+2
10−83 −1
10
10
10
2 Mp ' (α+2)
Mp ' (α+2)
GeV . v (α+2)
GeV .
v ' (α+2)
140 αα
140 αα
140 αα
10 αα




0
0
ε̄=ΩD ε, p̄=ΩD p, ᾱ=ΩD α− ΩΩ T
ε̄=ΩD ε, p̄=ΩD p, ᾱ=ΩD α− ΩΩ T

R
ΠΩ (p) = i d4 xh0|T {ηµ (x), η µ (0)}|0iFαβ eip·x
= Π0 (p) + Π1 (p)(σ · F p̂ + p̂σ · F ) + · · · ,
1 i
b0i = a
w −v k b̃k i , bij =b̃ij .

 η1 −η2
4
det X 0
H(i(z12 ))= det X12
H(z̃⊕i(z12 ))
12
R
P
∞
0 ∞
(q)
1
dzχµ (z/a)Fm
(z,x),
qc (a,r)= 4π2 a4
m=0
0

37

σc =

39

Θ =(ϑ ,e
vα̇ ),

γ ≡

36

38

α

λ4r (T )=

n−1
j
1
(z1 ,...,ẑj ,...,zn ),
j=1 (−1) z−z −1 ψ
j
2
2
2
M (1−sin 2β) sin θW
cos θ
δ2 = Z 2µ
[ (µ+M ) + (µ+MW) ]
1
2

φ(~
x,t)≡
αβ

0

a
¯ A X.
¯a
QA (ζ)X≡{Sm (ζ),X}A −i~∆
Ka
m (ζ)X≡(Sm (ζ),X) −i~∆m X,

e gs
ĥV b,Ab .
sin ϑW cos ϑW m2
Z
0
0
limσi →σj GP (σi ,σj )=−2α ln d2 (|σij |)+f (σi ,σj )

35

A

)

a
¯a
Qa
m (ζ)X≡(Sm (ζ),X) −i~∆m X,
e2 N
γ ≡ 2T .
λ4r (T0 )
λ4r (T )=
.
1− 9 2 λ4r (T0 ) ln(T /T0 )
8π
Ωϕ
η= 8π ,

+n−1/2

0

1
γ (n) =γ a1 ...an =γ [a1 γ a2 ...γ an ] = n!

dzdzdbdb
V (f )=0
n
P
(p4 )
L∆S=1 =G8 F 2 37
i=1 Ni Wi
Q
(0)
det(M (N0 ))=αQ(0) r,s [α(h−hr,s )]P` (N0 −rs/K) ,
→2
h
i
2
2
2
2 p0
r2
1 M2
β++ = 4M
2 −M yv + 2 M 2 (1−y)−1 rs+ +M M 2 s+ ,

light quarks

21

0

T̂ =V̂ +fˆĜV̂ ,

12

8

0

(n)

(

E −m −mπ
l

∼gs2−k .

α ,β
α,β
α,β
α ,β
α,β
α ,β
M(n)
M(m)
[M(m)
,M(n)
]≡M(m)
M(n)
−(−)αβ

10

7

d̄

ṗa +ba pb ωµ ẋ µµ =0

0

P
p a1
1
γ
=γ a1 ...an =γ [a1 γ a2 ...γ an ] = n!
...γ an
perm0 s (−1) γ
B
A
α e
α̇
B
e
Θ = ϑ ,ϑα̇ , ∂A = ∂α ,∂ ,
∂A ,Θ
=δA
pole
pole
mτ
−<eΣτ (mτ
)|
DR
Ŷτ (MZ )|DR = v̂(M )|
cosβ(MZ )
h Z DR
2i
3.4β [2|2] (y)=−9y 2 1−22.21y+36.93y
1−28.21y+143.2y 2
b α β = Π2 +2eBS3 α β
D

6

√ 2 2
E −m −m
√ 2 t2 π .

E−

( r`s0 )

∼gs2−k .
0

E+

r<a,

hV b,Ab =

e2 N (T ) τ
T
m
A=−β0 log µ µ
opt



g(d):=


mµ
opt

35
−β0 log µ
( 36

q=ε,pi ,

Π1 (p) =

R
i d4 x(η|Th (x), η µ (0)1|Nσs eip·x
Π0 (p) + Π1 (p)(σ · F p̂ + p̂σ · F ) + · · · ,

1 i
b0i = a
w −v k b̃ki , bij =b̃ij .

 2n−1 µ
4
det X 0
H(i(z12 ))= det X12
H(z̃⊕i(z12 ))
12
R
P
∞
0 ∞
(q)
1
qc (a,r)= 4π2 a2
dzχµ (z/a)Fm
(z,x),
m=0
0
e gs
ĥV h,db .
sin ϑW cos ϑW m2
Z
0
0
limσ ,σ0 GP (σi ,σj )=−2α ln d2 (|σij |)+f (σi ,σj )
i j
e2 N (T ) τ
σc =
T
m
m
−β0 log µ µ ).
A=−β0 log µ µ ( 35
ept
νθr 36


r<a,

q=ε,pi ,

hV,b,4b =
,

).

d,
for 1 ≤ d ≤ t − 1 , and
d − a + 1, for t ≤ d − a ≤ b − 1 .


g(d):=


,

d,
for 1 ≤ d ≤ t − h and
d − a + 1, for t ≤ d − ≤ 5 − 1 .

Figure 9. A random sample of mistakes made by I 2 L - STRIPS. Observe that usually the model gets most of the formula right and the
mistake is only in a small portion of the overall formula (e.g. sample # 1; generating one subscript t instead of an l ). In some cases the
mistake is in the font and in some cases the images are identical but were incorrectly flagged by the image-match evaluation software
(e.g. sample # 0 & #17). In some cases the predicted formula appears more correct than the original! (sample # 10 where position of
the subscript β has been ‘corrected’ by I 2 L - STRIPS). Sample # 20 is clearly bad probably owing to very few training samples with
underbraces. For more examples visit our website.105

Neural Markup Generation with Visual Attention

Dataset

Model

I2L-140K

I2L-STRIPS
I2L-STRIPS
I2L-NOPOOL
I2L-NOPOOL
I2L-STRIPS
I2L-STRIPS

Im2latex-90k

Init
Model?
Yes
No
Yes
No
Yes
No

λA

β1

0.0
0.0
0.0
0.0
0.0
0.0005

0.5
0.5
0.5
0.1
0.5
0.5

Training
Epochs
104
75
104
119
110
161

Validation
BLEU@Epoch
0.8900@72∗
0.8874@62
0.8909@72∗
0.8820@92
0.8886@77∗
0.8810@118

Training
BLEU
0.9361
0.9300
0.9333
0.9348
0.9366
0.9386

Validation
ED
0.0677
0.0691
0.0684
0.0738
0.0688
0.0750

ASEN
5.3827
4.9899
4.5801
4.7099
5.1237
4.8291

Table 6. Training metrics. λR = 0.00005 and β2 = 0.9 for all runs. Training BLEU scores were calculated over 100 mini-batches after
decoding the sequences using CTC-decoding (Graves et al., 2006). ∗ denotes that the row corresponds to Table 7.

Dataset

Model

I2L-140K

I 2 L - NOPOOL
I 2 L - STRIPS

Im2latex-90k
Im2latex-100k

I 2 L - STRIPS
I M 2T EX

BLEU
Score
89.0%
89.0%
88.19%
87.73%

Edit
Distance
0.0676
0.0671
0.0725
-

Visual
Match103
70.37%
69.24%
68.03%
79.88%

Compiling
Predictions
99.94%
99.85%
99.81%
-

Table 7. Test results. I M 2T EX results are taken from Deng et al. (2017). Im2latex-90k dataset are the 90k samples of the Im2latex-100k
dataset (Deng et al., 2017) that remained after the formulas were normalized, rendered and filtered of duplicates, non-compiling and large
formulas. The last column is the percentage of successfully rendering predictions.

Graves, Alex. Supervised sequence labelling with recurrent
neural networks. In Studies in Computational Intelligence,
2008.
Graves, Alex. Generating sequences with recurrent neural
networks. CoRR, abs/1308.0850, 2013.
Graves, Alex and Schmidhuber, Jürgen. Offline handwriting
recognition with multidimensional recurrent neural networks. In NIPS, 2008.

Koray. Neural machine translation in linear time. CoRR,
abs/1610.10099, 2016.
Karpathy, Andrej and fei Li, Fei. Deep visual-semantic
alignments for generating image descriptions. 2015 IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 3128–3137, 2015.
Kingma, Diederik P. and Ba, Jimmy. Adam: A method for
stochastic optimization. CoRR, abs/1412.6980, 2014.

Graves, Alex, Fernández, Santiago, Gomez, Faustino J., and
Schmidhuber, Jürgen. Connectionist temporal classification: labelling unsegmented sequence data with recurrent
neural networks. In ICML, 2006.

Luong, Minh-Thang, Pham, Hieu, and Manning, Christopher D. Effective approaches to attention-based neural
machine translation. CoRR, abs/1508.04025, 2015. URL
http://arxiv.org/abs/1508.04025.

Graves, Alex, Mohamed, Abdel-rahman, and Hinton, Geoffrey E. Speech recognition with deep recurrent neural
networks. CoRR, abs/1303.5778, 2013. URL http:
//arxiv.org/abs/1303.5778.

Pascanu, Razvan, aglar Gülehre, Cho, Kyunghyun, and
Bengio, Yoshua. How to construct deep recurrent neural
networks. CoRR, abs/1312.6026, 2013.

Hochreiter, Sepp and Schmidhuber, Jürgen. Long shortterm memory. Neural Comput., 9(8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.
9.8.1735. URL http://dx.doi.org/10.1162/
neco.1997.9.8.1735.
Johnson, Justin, Karpathy, Andrej, and Fei-Fei, Li. Densecap: Fully convolutional localization networks for dense
captioning. CoRR, abs/1511.07571, 2016.
Kalchbrenner, Nal, Espeholt, Lasse, Simonyan, Karen,
van den Oord, Aäron, Graves, Alex, and Kavukcuoglu,

Pedersoli, Marco, Lucas, Thomas, Schmid, Cordelia, and
Verbeek, Jakob. Areas of attention for image captioning.
CoRR, abs/1612.01033, 2016. URL http://arxiv.
org/abs/1612.01033.
Salimans, Tim, Karpathy, Andrej, Chen, Xi, and Kingma,
Diederik P. Pixelcnn++: Improving the pixelcnn with
discretized logistic mixture likelihood and other modifications. CoRR, abs/1701.05517, 2017.
Simonyan, Karen and Zisserman, Andrew. Very deep convolutional networks for large-scale image recognition.
CoRR, abs/1409.1556, 2014.

Neural Markup Generation with Visual Attention

Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence to sequence learning with neural networks. CoRR,
abs/1409.3215, 2014.
Theis, Lucas and Bethge, Matthias. Generative image modeling using spatial lstms. In NIPS, 2015.
van den Oord, Aäron, Dieleman, Sander, Zen, Heiga, Simonyan, Karen, Vinyals, Oriol, Graves, Alex, Kalchbrenner, Nal, Senior, Andrew W., and Kavukcuoglu, Koray. Wavenet: A generative model for raw audio. CoRR,
abs/1609.03499, 2016a.
van den Oord, Aäron, Kalchbrenner, Nal, and Kavukcuoglu,
Koray. Pixel recurrent neural networks. In ICML, 2016b.
van den Oord, Aäron, Kalchbrenner, Nal, Vinyals, Oriol,
Espeholt, Lasse, Graves, Alex, and Kavukcuoglu, Koray.
Conditional image generation with pixelcnn decoders.
CoRR, abs/1606.05328, 2016c.
Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption
generator. In 2015 IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2015.
Xu, Kelvin, Ba, Jimmy, Kiros, Jamie Ryan, Cho,
Kyunghyun, Courville, Aaron C., Salakhutdinov, Ruslan,
Zemel, Richard S., and Bengio, Yoshua. Show, attend
and tell: Neural image caption generation with visual
attention. In ICML, 2015.
Yu, Fisher and Koltun, Vladlen. Multi-scale context aggregation by dilated convolutions. CoRR, abs/1511.07122,
2015.
Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol.
Recurrent neural network regularization.
CoRR,
abs/1409.2329, 2014.

A. Qualitative Analyses and Discussion
In this section we present further analyses and discussion of
our experiments and comparison with related work.
A.1. Encoder
We initially experimented with the output of the VGG16
model (Simonyan & Zisserman, 2014) - per Xu et al. (2015).
However (presumably since VGG16 was trained on a different dataset and a different problem) the BLEU score didn’t
improve beyond 40%. Then we started training VGG16
along with our model but the end-to-end model didn’t even
start learning (the log-loss curve was flat) - possibly due
to the large overall depth of the end-to-end model. Reducing the number of convolution layers to 6 and changing
the non-linearity to tanh (to keep the activations in check)

got us good results. Further reducing number of layers to
5 yielded the same performance, therefore we stuck with
that configuration (Table 1). In additon, we experimented
with I 2 L - STRIPS because it reduces the rectangular imagemap to a linear map, thereby presumably making the alignment model’s task easier because now it would only need
to scan in one-dimension. However, it performed around
the same as I 2 L - NOPOOL and therefore that hypothesis was
debunked.
A.2. Attention Model
Xu et al. 2015’s formulation of attention model (αt,l =
M LP (al ; H t−1 )) receives inputs from only a single
image location. In comparison, our formulation (equation
8) receives the full encoded image a in its input. This
change was needed because the previous formulation did
not progress beyond a point, presumably because this problem warranted a wider receptive field. The new formulation
works equally well with different pooling strides (and correspondingly different values of L).
Xu et al. 2015’s formulation of z t = βt · αt · a includes a
scalar βt = M LP (H t−1 ) which informs the LSTM how
much emphasis to place on the image v/s the language
model. Experimentally we found that it had no impact
on end-to-end performance, therefore we dropped it from
our model.
Xu et al. 2015 also use a simpler formula for A =
PL Pτ
2
l=1 (
t=1 αt,l − 1) which they call ‘doubly stochastic optimization’. Our formulation uses the true mean of
αl , τ /L instead of 1, normalizes it to a fixed range so that
it can be compared across models and more importantly,
includes a target-ASE term ASET . Without this term, i.e.
with ASET = 0, A would bias the attention model towards
uniformly scanning all the L image locations. This is undesirable since there are many empty regions of the images
where it makes no sense for the attention model to spend
much time. Conversely, there are some densely populated
regions (e.g. a symbol with complex superscript and subscripts) where the model would reasonably spend more time
because it would have to produce a longer output sequence.
In other words, the optimal scanning pattern would have to
be non-uniform - ASET 6= 0. Also, the scanning pattern
would vary from sample to sample, but ASET is set to a
single value (even if zero) for all samples. Therefore we preferred to remove the attention-model bias altogether from
the objective function by setting λA = 0 in all situations
except when the attention model needed a ’nudge’ in order
to ‘get off the ground’. In such cases we set ASET based
on observed values of ASEN (Table 6).

Neural Markup Generation with Visual Attention

y
0
1
2
3

ŷ

∂A0µ
∂t
i

∂A0µ
∂t
i

=−i[A0µ ,HF 0 ],

=−i[A0µ ,HF 0 ],

{Φ

(x),Φj (y)}=ij δ 2 (x−y).
2
P
Vtotal = i ∂W
+VD +Vsof t
∂zi
R 3
R
−ip·x λ
α†α
(p)=
d
x
e
[e ·(ωAa −iE a )+ Ω (f1 Πa +f2 φa )]
λ

{Φ (x),Φj (y)}=ij δ 2 (x−y).
2
P
Vtotal = i ∂W
+VD +Vsof t
∂zi
R 3
R
−ip·x λ
α†a
(p)=
d
x
e
[e ·(ωAa −iE a )+ Ω (f1 Πa +f2 φa )]
λ

4

i
Hstat (k)=P+ vk+i
,

5

(φ∗ Ps +φPs∗ ) 2∆
,
M2

(φ∗ Ps +φPs∗ ) 2∆
,
M2

i
Hstat (k)=P+ vk+i
,

6

αβ
αβ
1
HG/H = 21 (πα − i~
πβ +VG/H ,
(πβ + i~
2 Γα )g
2 Γβ )= 2 πα g

αβ
αβ
1
HG/H = 12 (πα − i~
πβ +VG/H ,
(πβ + i~
2 Γα )g
2 Γβ )= 2 πα g

7

S[Φ]=S[φ]+S[ϕ]+Sint [φ,ϕ]

S[Φ]=S[φ]+S[ϕ]+Sint [φ,ϕ]

8

κ
γ1 = 4π

κ
γ1 = 4π

9

1−r
Γ= 8πM

2

2

γ2 = λ
4
2

B

1−r
Γ= 8πM

(|M S |2 +|M P |2 ) .
2GEν m2
r

q q
± 1r 2

2

(|M S |2 +|M P |2 ) .

B

m S S
+ ν r1 2

mν S1 S2
r

11

χ(x1 ,x2 )=h0|T ψ(x1 )ψ̄(x2 )|P i.

12

L(2) θ=h̃1 v 2 φj (γ i θ)(θγ ij θ)+h1 v i v j φk (γ i θ)(θγ jk θ),

L(2) θ=h̃1 v 2 φj (γ i θ)(θγ ij θ)+h1 v i v j φk (γ i θ)(θγ jk θ),

4 |U |2 |Ue2 |2
sin2 2ϑ=sin2 2ϑsun = (|U e1
2
2 2
e1 | +|Ue2 | )
2
g
F1 =
MP l ' 1.5×1015 GeV.
5/2
192π

4|U |2 |Ue2 |2
sin2 2ϑ=sin2 2ϑsun = (|U e1
2
2 2
e1 | +|Ue2 | )
2
g
F1 =
MP l ' 1.5×1015 GeV.
5/2
192π

14
15

E(r)=−(

q1 q2
r

E(r)=−(

13

)

2GEν m2
r

10

19

1
P̃g (z)=∆ns z1 + 1−z
.

20

S≤SH , T ≥TH ,

21
22





Ec ≤EBH , for HR≥1
h
i
√
3/2 B0
1/2 mp 1/2
8 2
Γ∼N −1/2 1023 s−1 exp − 3·137
( −E
( me ) ,
me )
NB A
2|J|2
m0 2

2

m
5
∼
= m b 2 ∼2.5×10 .

.

P
b<i> = 0≤p<q≤p Xr(i) iτ (i) (z,z) .

Q
b<i> = 0≤p<q≤p Xτ p (i)τ q (i) (z,z) .

18

17

)

(2)

(2)

.

i
+
F1W D (x)=[dp (x)+ūp (x)+dn (x)+ūn (x)+2s(x)+2c̄(x)]/2.
u=q 2 b2 /2 or Q=Q0 u,
Q0 = Am1 b2
N
1
1
h̄v Γ hv = 2 Tr Γ Pv h̄v hv − 2 Tr γµ γ5 Pv Γ Pv h̄v γ µ γ5 hv

16

+

χ(x1 ,x2 )=h0|T ψ(x1 )ψ̄(x2 )|P i.
(2)

(2)

±

F1W

,

+D

(x)=[dp (x)+ūp (x)+dn (x)+ūn (x)+2s(x)+2c̄(x)]/2.

u=q 2 b2 /2 or Q=Q0 u,

Q0 = Am1

h̄v Γ hv = 12

Tr(γµ γ5 Pv Γ Pv )h̄v γ µ γ5 hv ,

Tr(Γ Pv )h̄v hv − 12
1
1
P̃g (z)=∆n z + 1−z
.

Nb

2

S≤SH , T ≥TH , Ec ≤EBH , for HR≥1
h
i
√
3/2 B0
1/2 mp 1/2
8 2
Γ∼N −1/2 1023 e−1 exp − 3·137
( −E
( me ) ,
me )
NB A
R 2
|
2
m0

2

2

m
5
∼
= m2b ∼2.5×10 .

25

d
∂
u= z` U −1/2 ∂t
,
ρ
Ω= ρc
e(2r+1)πiL(0) Y1 (v,x)e−(2r+1)πiL(0) =Y1 ((−1)L(0) v,−x),

d
∂
u= z` U −1/2 ∂t
,
ρ
Ω= ρc
e(2r+1)πiL(0) Y1 (v,x)e−(2r+1)πiL(0) =Y1 ((−1)L(0) v,−x),

26

R
A2 = d2 x A2 (x)∗δα(x) ,

R
A2 = d2 x A2 (x)∗δα(x) ,

27

ds2 =(k+f0 R02 )−1 dR2 +R2 dΩ2k −(k+f0 R02 )[dx5 +AR (R)dR]2

28

[ρ̂0 ,ρ̂0 ]=0,

29

dW
L4 =(F 1 −∂5 A2 ) dA
1 +··· .

30

Ψj Ψi =δij −q −1 R̂ikjl Ψl Ψk

23
24

R2

R2

[Ŝ0A ,

R2

R2

ds2 =(k+f0 R02 )−1 dR2 +R2 dΩ2k −(k+f0 R02 )[dx5 +AR (R)dR]2
[ρ̂0 ,ρ̂0 ]=0,

[Ŝ0A ,

dW
L4 =(F 1 −∂5 A2 ) dA
1 +··· .

Ψj Ψi =δij −q −1 R̂iklj Ψl Ψk
2

Θk =∂k Π0 +cm hk =0 ,

Πi =0 ,

32

s c23
sin δ= 23
s2 c 2

33

∆A =−2H sAA ,

34

P
−1
−1 αβ
−1 n
D̂µν
=D̂µα
η
ηβν + ∞
)βν ,
n=1 An (D

P
−1
−1 αβ
−1 n
D̂µν
=D̂µα
η
ηβν + ∞
)βν ,
n=1 An (D

35

HG (x2 )=−

1
[B(x2 )]−2 −1 ,
8πGe2
a2
− −
+
Tµν =Tµν + 2 Tµν .
a
+
1−d R +∞
dx
GH d+1 = l Σ
.
σ
sinhd x
d
l
(n)
δ (n) Γ(A0 )
Γ{µ} = δA0 (x )···δA0 (x )···δA0 (x ) ,
n
1
j
µ1
µj
µn

2 −2
1
HG (x2 )=− 8πGe
−1),
2 ([B(x )]

s c23
sin δ= 23
s2 c 2

sin δ13

37
38



(

π̇ab =

40

0
i
γπa
=0, γπa
=f bac πbi η2c , γπ0i =0, γπij =0,

41



)

39

2

sin δ13

∆A =−2H sAA ,



36

Πi =0 ,

Θk =∂k Π0 +cm2 hk =0 ,

31



a2
− −
Tµν .
a2
+
R +∞ dx
ll−d
GH d+1 = Σ
.
q
sinhd x
d
(n)
∂ n(n) Γ(A0 )
Γ{µ} = δA0 (x )···δA0 (x )···δAo (x ) ,
n
1
j
µ1
µµ
µn

+
Tµν =Tµν
+

π̇ab =
2

|Z1 | =|Z2 |

−η0
1
= (4G)
[(QR 1 )2 +(QR 2 )2 ]
2e

0
i
γπa
=0, γπa
=f bac πbi η2c , γπ0i =0, γπij =0,

,

−η0
1
|Z1 |2 =|Z2 |2 = (4G)
[(QR 1 )2 +(QR 2 )2 ] ,
2e

Figure 10. A random sample of predictions of I 2 L - STRIPS containing both good and bad predictions. Note that though this is a random
sample, prediction mistakes are not obvious and it takes some effort to point them out! For more examples visit our website.105

Neural Markup Generation with Visual Attention

A.3. LSTM Stack
During experimentation our penultimate LSTM-stack which
had 3 LSTM layers with 1000 units each, gave us a validation score of 87.45%. At that point experimental observations suggested that the LSTM stack was the accuracy
’bottleneck’ because other sub-models were performing very
well. Increasing the number of LSTM units to 1500 got us
better validation score - but a worse overfit. Reducing the
number of layers down to 2 got us the best overall validation
score. In comparison, Xu et al. (2015) have used a single
LSTM layer with 1000 cells.
A.4. Deep Output Layer
Note that the output layer receives skip connections from
the LSTM-Stack input (Equation 13). We observed a 2%
impact on the BLEU score with the addition of input-tooutput skip-connections. This leads us to believe that adding
skip-connections within the LSTM-stack may help further
improve model accuracy. Overall accuracy also improved by
increasing the number of layers from 2 to 3. Lastly, observe
that this sub-model is different from Xu et al. (2015) wherein
the three inputs are affine-transformed into D dimensions,
summed and then passed through one fully-connected layer.
After experimenting with their model we ultimately chose to
instead feed the inputs (concatenated) to a fully-connected
layer thereby allowing the MLP to naturally learn the inputto-output function. We also increased the number of layers
to 3, changed activation function of hidden units from relu
to tanh101 and ensured that each layer had at least as many
units as the softmax layer (K).
A.5. Init Model
We questioned the need for the Init Model and experimented
just using zero values for the initial state. That caused a
slight but consistent decline (< 1%) in the validation score,
indicating that the initial state learnt by our Initial State
Model did contribute in some way towards learning and
generalization. Note however that our Init Model is different
than (Xu et al., 2015), in that our version uses all L feature
vectors of a while theirs takes the average. We also added
a hidden layer and used tanh activation function instead
of relu. We did start off with their version but that did not
provide an appreciable impact to the bottom line (validation).
This made us hypothesize that perhaps taking an average
of the feature vectors was causing a loss of information;
and we mitigated that by taking in all the L feature vectors
without summing them. After making all these changes,
the Init Model yields a consistent albiet small performance
improvement (Table. 8). But given that it consumes ∼7.5

Figure 11. Attention scan continuing from Figure 5. The movement of focal-region is aligned with the output word (above the
image).

101
We changed from relu to tanh partly in order to remedy
‘activation-explosions’ which were causing floating-point overflow
errors

Neural Markup Generation with Visual Attention

million parameters, its usefulness remains in question.
Model
I 2 L - NOPOOL
I 2 L - NOPOOL
I 2 L - STRIPS
I 2 L - STRIPS

Init Model
Present?
Yes
No
Yes
No

Validation
BLEU
89.09%
88.20%
89.00%
88.74%

Num
Params
7,569,300
0
7,569,300
0

Table 8. Impact of the Init Model on overall performance. Since it
comprises 10-12% of the total params, it may as well be omitted
in exchange for a small performance hit.

A.6. Training and Dataset
Default values of β1 andβ2 of the ADAM optimizer - 0.9
and 0.99 - yielded very choppy validation score curves with
frequent down-spikes where the validation score would fall
to very low levels, ultimately resulting in lower peak scores.
Reducing the first and second moments (i.e. β1 andβ2 ) fixed
the problem suggesting that the default momentum was too
high for our ‘terrain’. We did not use dropout for regularization. However increasing the data-set size and raising
the minimum-word-frequency threshold from 24 to 50 did
yield better generalization and overall test scores (Table 7).
Finally, normalizing the data (Section 2.5) yielded about
25% more accuracy than without.

