Reading Scene Text with Attention Convolutional Sequence Modeling
Yunze Gao1,2 , Yingying Chen1,2 , Jinqiao Wang1,2 , Hanqing Lu1,2
National Lab of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China
2
University of Chinese Academy of Sciences, Beijing, China
{yunze.gao, yingying.chen, jqwang, luhq}@nlpr.ia.ac.cn

Abstract

Introduction
Scene text recognition is to reading text in natural images,
which has received considerable attention and plays an important role in a variety of computer vision tasks. Reading
text in the wild can extract rich semantic information that is
highly relevant to scene or object and therefore has been applied in plenty of real world applications, such as street sign
reading in the driveless vehicle, automatic car license plate
recognition, assistive technologies for the blind, robot navigation, scene understanding and image retrieval. However,
suffering from various appearance, distortion, low resolution, blur and disturbance of background noise, text recognition in unconstrained environment is still a challenging problem.
Traditional methods (Wang, Babenko, and Belongie
2011; Bissacco et al. 2013; Mishra, Alahari, and Jawahar
2012b) recognized scene text by first detecting individual
character and then recognizing each cropped character with
convolutional neural network separately. A large amount of
inter-character and intra-character confusion impact the per-

trip
CTC

hour

cream

Reading text in the wild is a challenging task in the field of
computer vision. Existing approaches mainly adopted Connectionist Temporal Classification (CTC) or Attention models based on Recurrent Neural Network (RNN), which is
computationally expensive and hard to train. In this paper, we
present an end-to-end Attention Convolutional Network for
scene text recognition. Firstly, instead of RNN, we adopt the
stacked convolutional layers to effectively capture the contextual dependencies of the input sequence, which is characterized by lower computational complexity and easier parallel computation. Compared to the chain structure of recurrent
networks, the Convolutional Neural Network (CNN) provides
a natural way to capture long-term dependencies between elements, which is 9 times faster than Bidirectional Long ShortTerm Memory (BLSTM). Furthermore, in order to enhance
the representation of foreground text and suppress the background noise, we incorporate the residual attention modules
into a small densely connected network to improve the discriminability of CNN features. We validate the performance
of our approach on the standard benchmarks, including the
Street View Text, IIIT5K and ICDAR datasets. As a result,
state-of-the-art or highly-competitive performance and efficiency show the superiority of the proposed approach.

rose
flavor
trip

arXiv:1709.04303v1 [cs.CV] 13 Sep 2017

1

t

r

i

p EOS

R

R

R

R

A

R

trip
CTC

RNN

RNN

CNN

…

…

…

CNN

CNN

CNN

CNN

(a)

(b)

(c)

(d)

…
1  1  classes

Figure 1: Three typical architectures and our network for
scene text recognition. (a) CNN + softmax. (b) RNN + CTC.
(c) RNN + Attention. (d) CNN + CTC (our approach).

formance of the entire recognition network greatly. Therefore, these approaches rely on an accurate character detector.
Recently, some works adopted an end-to-end CNN framework for scene text recognition. (Jaderberg et al. 2016) formulated the scene text recognition as an image classification
problem. As shown in Figure 1(a), each class corresponds to
one word in a pre-defined large lexicon composed of around
90k words. However, it is difficult to be generalized to other
situations that has huge number of classes due to the oversize
pre-refined dictionary and the requirement for large amount
of training samples.
To address this dilemma, recent studies (Shi, Bai, and Yao
2016; Lee and Osindero 2016; Shi et al. 2016) regarded
scene text recognition as a sequence recognition problem
to directly generate label sequence, by means of recurrent
neural network with CTC (Graves et al. 2006) or Attention schemes. As shown in Figure 1(b), RNN is leveraged
to model the sequential dependencies within the feature sequence produced by CNN and then CTC is to predict label sequence with arbitrary length. Afterwards, attention decoder (Shi et al. 2016; Lee and Osindero 2016) was proposed
to weight the feature sequence and perform soft features selection, as illustrated in Figure 1(c).
RNN is effective to learn the contextual information and
capture the long-term dependencies. However, depending on
the computation of the previous step, the recurrent connec-

Feature
sequence

Feature
maps

…

Attention feature encoder

C H

n

Convolutional sequence modeling

BREWERY

C

B-RR-E--W-EE-R--Y

H

...

Sequence
to map

...

Dense
block

...

Residual
attention

...

Dense
block

Convolution

Residual
attention

Convolution

Convolution

Dense
block

Probability
distribution

CTC

Figure 2: Overview of the attention convolutional network. The attention feature encoder is a small densely connected network
with residual attention for extracting feature sequence, in which each vector of the feature sequence corresponds to a local
region of the input image. Then the elements of the feature sequence are put together to form a 2D map, which serves as the
input of a convolutional neural network. Next, CNN captures the effective contextual information and learns the sequential
dependencies, which is characterized by lower computational complexity and easier parallel computation. Subsequently, the
output feature maps are restored to a sequence as the input of CTC to generate final label sequence. Specifically, “n” denotes
the number of classes.
tion is incapable of performing parallel operations. The process of sequence modeling is closely related to the length of
input sequence due to the chain structure, so recurrent network is computation intensive and time consuming. Besides,
sometimes RNN is difficult to train due to the problem of
gradient vanishing/exploding (Bengio, Simard, and Frasconi
1994). By contrast, CNN has the superiority of parallelism
and lower computational complexity. Convolutional layers
apply filters over the entire sequence simultaneously and
allow parallel computation. Recent works have used CNN
for sequence learning in machine translation (Gehring et al.
2017) and language modeling (Dauphin et al. 2016). In this
paper, we present an end-to-end attention convolutional network for scene text recognition, which combines CNN with
CTC to generate label sequence, without any recurrent connections, as shown in Figure 1(d). Firstly, a sequence-to-map
operation transforms the feature sequence into a 2D map as
the input of CNN to process the sequence simultaneously.
Then stacked convolutional layers can extract hierarchical
contextual representation of the input sequence to model the
long-term dependencies with a shorter path. At the same
time, the length of dependencies can be controlled by the
number of convolutional layers. This architecture is entirely
convolutional, so it is easily for parallel processing and independent of the sequence length.
Besides, in order to enhance the representation of the
text to be recognized and suppress the background noise,
we incorporate a residual attention mechanism into a small
densely connected network to extract the discriminative
attention-aware features. We evaluate our approach on challenging benchmarks including the Street View Text, IIIT5K and ICDAR datasets. It is observed that the proposed
method can improve the efficiency with the state-of-the-art
performance. The mainly contributions are summarized as

follows:
(1) A novel end-to-end attention convolutional network is
proposed for scene text recognition, which is entirely convolutional and performs well with both high accuracy and
efficiency.
(2) Instead of RNN, we adopt the stacked convolutional
layers to effectively capture the contextual dependencies of
the input sequence, which is characterized by lower computational complexity and easier parallel computation.
(3) The residual attention modules in the small densely
connected network could effectively suppress the response
of background noise while enhancing the representation of
foreground text.

Related Work
Traditional methods of scene text recognition first performed detection to generate multiple candidates of character locations, then applied a character classifier for recognition. (Wang, Babenko, and Belongie 2011) used Random
Ferns and HOG features to detect characters and then found
an optimal configuration of a particular word via a pictorial structure. (Mishra, Alahari, and Jawahar 2012b) detected
character candidates using sliding windows and integrated
both bottom-up and top-down cues in a unified Conditional
Random Field (CRF) model. (Shi et al. 2013) constructed
a part-based tree-structured model to recognize characters
in cropped images. (Yao et al. 2014) proposed an alternative way for character representation, denoted as Strokelets,
which was a combination of multi-scale mid-lever features.
Afterwards, the explorations of scene text recognition focus on the mapping from the entire image to word string
directly. (Almazán et al. 2014) embedded word images and
word labels into a common Euclidean space and the embed-

ding vectors were used to match images and labels. (Jaderberg et al. 2014a) constructed two CNNs to classify character at each position in the word and detect the N-grams contained within the word separately, following a CRF model
to combile their representations. Recently, there are increasing researches on treating scene text recognition as a sequence recognition problem. (Shi, Bai, and Yao 2016) proposed Convolutional Recurrent Neural Network (CRNN)
which combined convolutional network and recurrent network to model the spatial dependencies. In (Lee and Osindero 2016), a recurrent network with attention mechanism
was constructed to decode feature sequence and predict labels recurrently. (Shi et al. 2016) adopted a convolutionalrecurrent structure in the encoder to learn the sequential dynamics.
Since the lower computational complexity and greater
parallelism, CNN is a more efficient structure to capture
the sequential contextual information. Some attempts for applying CNN to sequence modeling have been made to replace RNN. (Dauphin et al. 2016) introduced a new neural language model that replaced the recurrent connections
typically used in RNN with gated temporal convolutions.
(Strubell et al. 2017) proposed the Iterated Dilated Convolutional Neural Networks (ID-CNNs), which was a faster alternative to recurrent network for obtaining per token vector representations in Natural Language Processing (NLP).
(Gehring et al. 2017) proposed an architecture based entirely
on convolutional neural network in machine translation. Although CNN has shown the superiority of parallelism and
efficiency, as far as we know, there is no research on using
CNN to perform sequence generation in the field of scene
text recognition. Our method incorporates CNN and CTC
into a unified framework without any recurrent connections,
which improves the efficiency while achieving good performance. Furthermore, the proposed algorithm is not limited
by a pre-refined dictionary and is available in both lexiconfree and lexicon-based setting. Without individual character
detection, the proposed network can be trained end-to-end
with the word level annotations and can effectively deal with
words with arbitrary length.

The Proposed Approach
The overview of the attention convolutional network for
scene text recognition is illustrated in Figure 2, which is
composed of an attention feature encoder, a convolutional
sequence modeling module and a CTC module. The attention feature encoder is a small densely connected network with residual attention for extracting feature sequence.
Specifically, The encoder takes the cropped word image as
input and extracts robust feature representation for the entire image, with a residual attention mechanism to suppress
background noise. Then the feature maps produced by the
dense attention network are converted into a feature sequence. Each element in the feature sequence, which corresponds to a local region of the word image, can be viewed
as the feature representation of the region. In the convolutional sequence modeling module, in order to process the sequence simultaneously, a sequence-to-map operation transforms the feature sequence into a 2D map to serve as the in-

put of the convolutional layers. Then stacked convolutional
layers are leveraged to capture the effective contextual information and learn the sequential dependencies. Next, the
same as the map-to-sequence operation in the encoder, the
output feature maps are restored to a sequence as the input
of CTC. Finally, CTC is used to perform sequence generation with arbitrary length. In the following sections, we will
describe the three components in detail.

Attention Feature Encoder
To improve the discriminability of CNN features, we design
a dense attention encoder network to extract the attentionaware representation. In dense blocks, dense connectivity
can improve the information flow between layers, and meanwhile, equipped with a residual attention mechanism, the
disturbance of background noise can be effectively reduced.
Furthermore, we convert the feature maps into a sequential
representation of the input image. Specifically, in a left-toright order, we take out the same columns of feature maps
and concatenate them into a vector, which corresponds to a
local rectangular region of the word image.
Dense Connectivity Taking advantage of the power of
Densenet (Huang et al. 2016), we utilize the dense connectivity to improve the flow of information and gradient
propagation in the encoder network. There exists direct connections between all layers in the dense block. Therefore,
each layer can get the information from all preceding layers and transmit its message to all subsequent layers. Additionally, instead of gradient back propagation from layer
to layer, each layer can obtain the deep supervision, which
eases the learning. The concatenation of feature maps produced by preceding layers severs as the input of the later
layers.
Residual Attention Attention mechanism plays an important guiding role in the process of feature learning, which
aims to selectively focus on the salient regions of the objects and enhance the representation of relevant parts. For
text in natural images, there often exists some disturbances,
including shadow, irrelevant symbols and background texture. The scene text with various appearance is often confused by these factors. Therefore, we introduce a residual
attention mechanism as (Wang et al. 2017) to enhance the
representation of foreground text and suppress background
noise.
The residual attention module serves as the transition between dense blocks, which is divided into two branches.
Specifically, the feature branch performs the feedforward
process and the attention branch generates the soft attention weights. The attention branch is designed with bottomup top-down structure so that the high-level semantic information can be captured to guide the discriminative feature selection. Max pooling followed by the convolutional
layer are stacked several times to increase the receptive field
rapidly and collect the global information. Then a symmetrical architecture with bilinear interpolation for upsampling is
applied to recover the resolution. Afterwards, the attention
maps as soft weights are added on corresponding feature
maps in each position. Considering the attention weights

range from zero to one, the element-wise product between
feature maps and attention maps may cause severe degradation of useful information. Therefore, the residual attention
learning is utilized to address this issue. Similar to the connection in Resnet (He et al. 2016), the output of residual
attention module is:
T = (1 + A) × F

(1)

where F and A denote the output of the feature branch
and the attention branch, respectively. In this way, the background noise can be suppressed efficiently while enhancing
the discriminability of original features. Furthermore, different attention modules generate the attention maps adapted
to the corresponding features. The low-level attention module mainly concentrates on the appearance including edge,
color and texture, while the high level attention module contains more semantic information. With the residual attention
mechanism, the feature encoder benefits from the noise suppression to obtain a more discriminative representation.

Convolutional Sequence Modeling
As the dominant approach of sequence to sequence learning,
RNN has been successfully applied in many computer vision
tasks, including speech recognition (Hannun et al. 2014;
Miao, Metze, and Rawat 2013), language modeling (Jozefowicz et al. 2016) and machine translation (Luong et al.
2014). However, depending on the computation of the previous step, the recurrent connection is incapable of performing parallel operations. Besides, sometimes RNN is difficult
to train due to the problem of gradient vanishing/exploding.
Therefore, in this paper, we adopt CNN to capture the sequential dependencies with both directions for scene text
recognition, which is a faster alternative of recurrent network. Given the feature sequence produced by encoder,
which is denoted as f = (f1 , f2 , · · · , fw ). In order to get
the contextual information c = (c1 , c2 , · · · , cw ), the recurrent neural network R generates the contextual representation through the recurrent connection ci = R(ci−1 , fi ),
which is a chain structure and is unable to perform parallel
computation.
Our approach models the sequential dependencies with
entirely convolutional operation. Firstly, the elements of the
feature sequence are put together to form a 2D map, where
each column is associated with a local region of original
word image from left to right. Then we convolve the input using a filter with width k, resulting contextual information over k elements of the input sequence. Stacked convolutional layers generate the hierarchical representation to
enlarge the size of receptive field rapidly. For example, 4
convolutional layers with kernel size k = 5 can capture the
information of 17 input elements. In this way, we can easily control the range of spatial dependencies to be modeled
through the number of convolutional layers. When there are
sufficient layers, the high-level features can obtain the requiring contextual information. Besides, the convolution operation does not depend on the state of the previous step and
is irrelevant to the length of input sequence. Therefore, the
computation over the entire sequence can be simultaneously

processed, which could greatly accelerate the process of sequence modeling. Furthermore, the convolutional network
consumes less memory space and running time due to fewer
parameters and lower computation complexity.
In the process of sequence modeling, we ensure that the
length of sequence remains unchanged, by means of convolution with zero padding. Afterwards, in order to get the
sequential representation to serve as the input of CTC, we
restore the output feature maps to a sequence again by the
same map-to-sequence operation in the feature encoder. Assuming the feature maps produced by CNN have the dimension of C × H × W , where C, H and W denote the channels, height and width, respectively. Specifically, we crop the
each channel of feature maps by column and then concatenate the same columns of all channels into a vector, which
has a dimension of C × H. Therefore, we can obtain a sequence with W vectors, which is the contextual information
c = (c1 , c2 , · · · , cw ). Finally, for the generated sequence,
we obtain the probability distribution over the label space
for per-frame in the sequence via a linear layer:
yt = sof tmax(W ∗ ct + b), t = 1, · · · , w

(2)

where W and b denote the weight matrix and bias separately.

Connectionist Temporal Classification
To generate the final label sequence for the input image, the output probability distribution sequence y =
(y1 , y2 , · · · , yw ) is interpreted as conditional probabilities
over possible label sequences, using the Connectionist Temporal Classification (CTC) proposed by (Graves et al. 2006).
Defining L as the set of 36 classes including all English al0
phanumeric characters, we get the final label space L =
L ∪ {blank}, in which the extra blank denotes the class for
observing no character. Given the probability distribution,
the conditional probability of the sequence π is
p(π|y) =

w
Y

yπt t

(3)

t=1

where yπt t denotes the probability of emtting label πt at step
t. Then a many-to-one mapping B is defined to transform
the sequence π to a shorter sequence as the final prediction.
The operation B is to merge the repeated continuous labels
to a single one and then remove the blank labels. For example, B maps the sequence “- -aa-b- -c-dd” to “abcd”, where
the ‘-’ represents the blank. Different sequences π may be
mapped to the same result, thus the probability of the final
output sequence is formulated as the sum of the conditional
probabilities of all π corresponding to it:
X
p(l|y) =
p(π|y)
(4)
π∈B−1 (l)

In genaral, for a given sequence, there are a large number
of mapping paths. Thus the computation of the sum in Equation 4 is time consuming. To remedy this issue, the forwardbackward algorithm based on dynamic programming is used
to calculate the conditional probability and the error differentials in an efficient way.

Given the training set D = {Ii , li }, where Ii and li represent the word image and the corresponding ground truth
label, respectively. The objective function is formulated as
the sum of the negative log likelihood of the probabilities
for target labels:
O=−

X

log p(li |yi )

(5)

(Ii ,li )∈D

Minimizing the objective function equals to maximize
the probability of producing the target labels, which can be
solved by dynamic programming. By means of CTC, we
can handle sequences with arbitrary length, requiring no presegmented training data.
Lexicon-free Recognition Our network can be used in the
lexicon-free setting, i.e., the predicted words are not constrained with a pre-defined dictionary. During inference, the
sequence with the highest conditional probability severs as
the final output. However, there is no a general and tractable
approach to find the optimal solution. So the approach of
the best path decoding (Graves et al. 2006) is applied to get
the output label sequence approximately, assuming that the
most probable path corresponds to the most probable label:
lp = B(argmax p(π|y))

(6)

π

Specifically, the most probable path π is produced by
emitting the label with the highest probability at each step.
Lexicon-based Recognition In the constrained condition,
the prediction is generated by selecting the word with the
highest conditional probability in the pre-defined lexicon.
However, the probabilities need to be computed for all words
in the dictionary, which is time consuming and infeasible. So
we adopt an approximate method by comparing the edit distance between the predicted sequence in the lexicon-free setting and words in the lexicon, then choosing the word with
the smallest edit distance as the output label.

Experiment
In this section, we will give the implementation details and
evaluate the performance of the proposed approach.

Dataset
Several public datasets are used for the evaluation, including
Street View Text, IIIT5K, ICDAR 2003 and ICDAR 2013.
• Street View Text (Wang, Babenko, and Belongie 2011)
contains 647 word images which are cropped from 249
street-view images collected from Google Street View.
For each image, there is a 50 words lexicon defined by
(Wang, Babenko, and Belongie 2011), denoted as SVT50.
• IIIT5K (Mishra, Alahari, and Jawahar 2012a) contains
3000 cropped word images collected from the Internet.
Each image has a 50 words lexicon and a 1000 words lexicon, denoted as IIIT5k-50 and IIIT5k-1k separately.

Table 1: Network architecture of the attention convolutional
network. The growth rate is 18 in the dense blocks. Stride
2×1 represents the filter with height 2 and width 1.
Module
Layer
Configurations
Convolution
3×3, 36, stride 1×1
Dense Block
[3×3, stride 1×1] × 4
Attention Module
Attention 1
Average Pooling
2×2, stride 2×2
Dense Block
[3×3, stride 1×1] × 4
Attention Module
Attention 1
Encoder
Average Pooling
2×2, stride 2×2
Dense Block
[3×3, stride 1×1] × 4
Convolution
3×3, 512, stride 1×1
Average Pooling
2×2, stride 2×1
Convolution
3×3, 512, stride 1×1
Convolution
3×3, 1, stride 2×1
Convolution
3×3, 1, stride 2×1
CNN
Convolution
3×3, 1, stride 2×1
Convolution
3×3, 1, stride 2×1
CTC
CTC
• ICDAR 2003 (Lucas et al. 2005) contains 251 scene images and 860 cropped word images. The 50 words lexicon for each image is defined by (Wang, Babenko, and
Belongie 2011), which is denoted as IC03-50. And a full
lexicon is composed of all the words that appear in the
test set, denoted as IC03-Full.
• ICDAR 2013 (Karatzas et al. 2013) contains 1015
cropped word images without any lexicon, which derives
from the ICDAR 2003.
For training data, our model is trained purely on the synthetic dataset released by (Jaderberg et al. 2014b), without
any extra data for fine-tuning. The synthetic dataset contains
around 8 millions images, which are generated by an synthetic engine. Following the evaluation protocol in (Wang,
Babenko, and Belongie 2011), we perform recognition on
word images that contain only alphanumeric characters and
at least three characters.

Implementation Details
The network architecture of our approach is shown in Table
1. The attention feature encoder is composed of three dense
blocks and two residual attention modules. For each dense
block, there are four convolutional layers and the growth rate
is 18. For residual attention, the feature branch has one convolutional layer and the attention branch has the bottom-up
top-down structure. In detail, 3,2 max pooling are used in the
two residual attention modules, respectively. And a sigmoid
function is applied to normalize the weights in attention
maps. Besides, the skip connection is added to fuse information with different scales. Additionally, the CNN for sequence modeling contains 4 convolutional layers, so that the
per-frame representation of the output sequence can cover
the contextual information over 9 input elements. All the
convolution are performed with zero padding, ReLU activation function and batch normalization (Ioffe and Szegedy

Table 2: Scene text recognition accuracies on the benchmark datasets. “50”, “1000” and “Full” represent the size of lexicon
used for constrained recognition. “∗”(Jaderberg et al. 2016) is not lexicon-free strictly, due to the output sequence is constrained
to a 90k dictionary.
Methods
SVT-50 SVT IIIT5k-50 IIIT5k-1k IIIT5k IC03-50 IC03-Full IC03 IC13
ABBYY(2011)
35.0
24.3
56.0
55.0
Wang et al.(2011)
57.0
76.0
62.0
Mishra et al.(2012a)
73.2
64.1
57.5
81.8
67.8
Wang et al.(2012)
70.0
90.0
84.0
Goel et al.(2013)
77.3
89.7
Bissacco et al.(2013)
90.4
78.0
87.6
Alsharif and Pineau (2013)
74.3
93.1
88.6
Almazán et al.(2014)
89.2
91.2
82.1
Yao et al.(2014)
75.9
80.2
69.3
88.5
80.3
Rodriguez-Serrano et al.(2015)
70.0
76.1
57.4
Jaderberg et al.(2014)
86.1
96.2
91.5
Su and Lu (2014)
83.0
92.0
82.0
Gordo (2015)
91.8
93.3
86.6
∗Jaderberg et al.(2016)
95.4
80.7
97.1
92.7
98.7
98.6
93.1 90.8
Jaderberg et al.(2014a)
93.2
71.7
95.5
89.6
97.8
97.0
89.6 81.8
Shi et al.(2016)
97.5
82.7
97.8
95.0
81.2
98.7
98.0
91.9 89.6
95.5
81.9
96.2
93.8
81.9
98.3
96.2
90.1 88.6
Shi et al.(2016)
96.3
80.7
96.8
94.4
78.4
97.9
97.0
88.7 90.0
Lee et al.(2016)
Ghosh et al.(2017)
95.2
80.4
95.7
94.1
92.6
Ours
97.4
82.7
99.1
97.9
81.8
98.7
96.7
89.2 88.0

2015).
In the process of training and testing, the word images are
resized to 32 × 100 with gray scale. We adopt the msra (He
et al. 2015) as the weight initialization method and train our
network using Adam (Kingma and Ba 2014) with a minibatch size of 64. Moreover, the gradient clipping is used at
the magnitude of 5. The proposed network is implemented
with Tensorflow (Abadi et al. 2016).

Ablation Study
In this section, we evaluate the contributions of the two main
components in the proposed approach, including residual attention modules for suppressing noise and convolutional network for sequence modeling.
Residual Attention Module To validate the effectiveness
of residual attention modules, we explore the performance
of densely connected network with BLSTM and CNN while
applying residual attention mechanism. As shown in Table
3, the networks with residual attention modules consistently
outperform the networks without attention, which proves
the effectiveness of the method. Besides, we observe that
the improvements brought by residual attention modules on
ICDAR datasets are not as significant as that on SVT and
IIIT5k. The possible reason is that the proportion of images
with background noise in SVT and IIIT5k is higher than that
in ICDAR datasets. Furthermore, we visualize the attention
maps of some examples in Figure 3. As shown in Figure 3,
the attention maps focus on the foreground text to be recognized and effectively reduce the response of background
noise including shadow, irrelevant symbols, and background
texture.

Table 3: Lexicon-free scene text recognition accuracies on
standard benchmarks. The growth rate is 18 for all networks.
Method
SVT IIIT5k IC03 IC13
Dense+BLSTM
81.9 80.4 90.7 89.5
Dense+BLSTM+attention 83.0 83.0 90.8 88.7
Dense+CNN
81.2 79.6 88.3 88.6
Dense+CNN+attention
82.7 81.8 89.2 88.0

Convolutional vs. Recurrent Model To compare the performance of recurrent network and convolutional network,
we replace the convolutional layers for sequence modeling
with 2 layers BLSTM which has 256 units per layer as the
baseline. As shown in Table 4, compared with BLSTM, the
convolutional neural network requires significantly fewer
parameters and less time to achieve comparable performance. Furthermore, the speed of sequence modeling with
CNN can run 9 times faster than BLSTM and the parameters
are greatly reduced. Therefore, the proposed CNN model
significantly improves the efficiency while maintaining good
performance. The convolutional sequence modeling module
also can be used in other networks to capture contextual dependencies and achieve high efficiency.

Comparisons with State-of-the-art Methods
We evaluate the attention convolutional network on the
above four public datasets and compare it with state-of-theart algorithms in Table 2. Most existing approaches were incapable of performing recognition without a dictionary and
did not report the results in the unconstrained condition. By
contrast, our method is available in both lexicon-free and

(a)

(b)

(c)

(d)

Figure 3: Visualization of the feature maps and attention
maps. (a) Original images. (b) Feature maps before attention. (c) Attention maps. (d) Feature maps after attention.
Table 4: Comparison between CNN and BLSTM. The SM
time represents the time for sequence modeling (SM) with
CNN or BLSTM.
Model
SVT Params Total time SM time
BLSTM-2L 83.0 141.5M
45.3ms
31.7ms
CNN-4L
82.7
75.4M
19.6ms
3.5ms
lexicon-based setting.
For lexicon-free recognition, our network achieves the
state-of-the-art or highly competitive performance. Specifically, we obtain the best result on SVT and the second best
result on IIIT5k. It is worthy noting that the model in (Jaderberg et al. 2016) only can deal with the words in its 90k dictionary, which is not lexicon-free recognition strictly. Different form (Jaderberg et al. 2016), our approach is able to
recognize random word strings, which is not restricted by
a fixed dictionary. Besides, we obtain the better or comparable results compared with (Shi et al. 2016), while (Shi et
al. 2016) used Spatial Transformer Network (STN) to rectify the irregular text. We do not perform any special operation aimed at irregular text, which shows the tolerance
of our network to spatial distortions of scene text. Furthermore, most methods used RNN to capture contextual dependencies, while we adopt convolutional sequence modeling
to achieve higher efficiency and competitive results.
For lexicon-based recognition, our method consistently
outperforms other approaches on several benchmarks. Especially, we outperform prior state-of-the-art (Shi, Bai, and
Yao 2016) by margin of nearly 3 percentages on IIIT5k with
1000 words lexicon. The significant improvement validates
the effectiveness of our method. Moreover, it is observed
that IIIT5k contains plenty of images suffering from background noise, which proves the superiority of our method
in suppressing noise. Besides, we only behind (Jaderberg
et al. 2016) on ICDAR03 with the full lexicon. However,
(Jaderberg et al. 2016) benefits from the pre-defined large
dictionary as mentioned before. Therefore, our results are
still competitive compared with the state-of-the-arts.

Conclusion
In this paper, we propose a novel attention convolutional network for scene text recognition, which contains a densely

connected network with residual attention modules for extracting features, stacked convolutional layers for sequence
modeling and CTC for sequence generation. Instead of
RNN, we introduce a convolutional neural network to capture the contextual information and model long-term dependencies with fewer parameters, which is 9 times faster than
BLSTM. In addition, the usage of residual attention mechanism significantly improves the performance and suppresses
background noise. Finally, the proposed network can be
trained end-to-end with the word level annotations. And our
method has capability of handling with word sequences with
arbitrary length in both lexicon-free and lexicon-based setting. The extensive experimental results on the challenging
datasets demonstrate the superiority of our approach compared with the state-of-the-art methods.

References
Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen,
Z.; Citro, C.; Corrado, G. S.; Davis, A.; Dean, J.; Devin,
M.; et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint
arXiv:1603.04467.
Almazán, J.; Gordo, A.; Fornés, A.; and Valveny, E. 2014.
Word spotting and recognition with embedded attributes.
IEEE transactions on pattern analysis and machine intelligence 36(12):2552–2566.
Alsharif, O., and Pineau, J. 2013. End-to-end text recognition with hybrid hmm maxout models. arXiv preprint
arXiv:1310.1811.
Bengio, Y.; Simard, P.; and Frasconi, P. 1994. Learning
long-term dependencies with gradient descent is difficult.
IEEE transactions on neural networks 5(2):157–166.
Bissacco, A.; Cummins, M.; Netzer, Y.; and Neven, H. 2013.
Photoocr: Reading text in uncontrolled conditions. In Proceedings of the IEEE International Conference on Computer
Vision, 785–792.
Dauphin, Y. N.; Fan, A.; Auli, M.; and Grangier, D. 2016.
Language modeling with gated convolutional networks.
arXiv preprint arXiv:1612.08083.
Gehring, J.; Auli, M.; Grangier, D.; Yarats, D.; and Dauphin,
Y. N. 2017. Convolutional sequence to sequence learning.
arXiv preprint arXiv:1705.03122.
Ghosh, S. K.; Valveny, E.; and Bagdanov, A. D. 2017. Visual
attention models for scene text recognition. arXiv preprint
arXiv:1706.01487.
Goel, V.; Mishra, A.; Alahari, K.; and Jawahar, C. 2013.
Whole is greater than sum of parts: Recognizing scene text
words. In Document Analysis and Recognition (ICDAR),
2013 12th International Conference on, 398–402. IEEE.
Gordo, A. 2015. Supervised mid-level features for word image representation. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2956–2964.
Graves, A.; Fernández, S.; Gomez, F.; and Schmidhuber, J.
2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.
In Proceedings of the 23rd international conference on Machine learning, 369–376. ACM.

Hannun, A.; Case, C.; Casper, J.; Catanzaro, B.; Diamos, G.;
Elsen, E.; Prenger, R.; Satheesh, S.; Sengupta, S.; Coates,
A.; et al. 2014. Deep speech: Scaling up end-to-end speech
recognition. arXiv preprint arXiv:1412.5567.
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Delving deep
into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, 1026–1034.
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the
IEEE conference on computer vision and pattern recognition, 770–778.
Huang, G.; Liu, Z.; Weinberger, K. Q.; and van der Maaten,
L. 2016. Densely connected convolutional networks. arXiv
preprint arXiv:1608.06993.
Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate
shift. In International Conference on Machine Learning,
448–456.
Jaderberg, M.; Simonyan, K.; Vedaldi, A.; and Zisserman,
A. 2014a. Deep structured output learning for unconstrained
text recognition. arXiv preprint arXiv:1412.5903.
Jaderberg, M.; Simonyan, K.; Vedaldi, A.; and Zisserman, A. 2014b. Synthetic data and artificial neural networks for natural scene text recognition. arXiv preprint
arXiv:1406.2227.
Jaderberg, M.; Simonyan, K.; Vedaldi, A.; and Zisserman,
A. 2016. Reading text in the wild with convolutional
neural networks. International Journal of Computer Vision
116(1):1–20.
Jaderberg, M.; Vedaldi, A.; and Zisserman, A. 2014. Deep
features for text spotting. In European conference on computer vision, 512–528. Springer.
Jozefowicz, R.; Vinyals, O.; Schuster, M.; Shazeer, N.; and
Wu, Y. 2016. Exploring the limits of language modeling.
arXiv preprint arXiv:1602.02410.
Karatzas, D.; Shafait, F.; Uchida, S.; Iwamura, M.; i Bigorda, L. G.; Mestre, S. R.; Mas, J.; Mota, D. F.; Almazan,
J. A.; and de las Heras, L. P. 2013. Icdar 2013 robust reading
competition. In Document Analysis and Recognition (ICDAR), 2013 12th International Conference on, 1484–1493.
IEEE.
Kingma, D., and Ba, J. 2014. Adam: A method for stochastic
optimization. arXiv preprint arXiv:1412.6980.
Lee, C.-Y., and Osindero, S. 2016. Recursive recurrent nets
with attention modeling for ocr in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2231–2239.
Lucas, S. M.; Panaretos, A.; Sosa, L.; Tang, A.; Wong, S.;
Young, R.; Ashida, K.; Nagai, H.; Okamoto, M.; Yamamoto,
H.; et al. 2005. Icdar 2003 robust reading competitions:
entries, results, and future directions. International journal
on document analysis and recognition 7(2):105–122.
Luong, M.-T.; Sutskever, I.; Le, Q. V.; Vinyals, O.; and
Zaremba, W. 2014. Addressing the rare word problem in
neural machine translation. arXiv preprint arXiv:1410.8206.

Miao, Y.; Metze, F.; and Rawat, S. 2013. Deep maxout
networks for low-resource speech recognition. In Automatic
Speech Recognition and Understanding (ASRU), 2013 IEEE
Workshop on, 398–403. IEEE.
Mishra, A.; Alahari, K.; and Jawahar, C. 2012a. Scene text
recognition using higher order language priors. In BMVC
2012-23rd British Machine Vision Conference. BMVA.
Mishra, A.; Alahari, K.; and Jawahar, C. 2012b. Top-down
and bottom-up cues for scene text recognition. In Computer
Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, 2687–2694. IEEE.
Rodriguez-Serrano, J. A.; Gordo, A.; and Perronnin, F.
2015. Label embedding: A frugal baseline for text recognition. International Journal of Computer Vision 113(3):193–
207.
Shi, B.; Bai, X.; and Yao, C. 2016. An end-to-end trainable
neural network for image-based sequence recognition and
its application to scene text recognition. IEEE transactions
on pattern analysis and machine intelligence.
Shi, C.; Wang, C.; Xiao, B.; Zhang, Y.; Gao, S.; and Zhang,
Z. 2013. Scene text recognition using part-based treestructured character detection. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition,
2961–2968.
Shi, B.; Wang, X.; Lyu, P.; Yao, C.; and Bai, X. 2016. Robust
scene text recognition with automatic rectification. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 4168–4176.
Strubell, E.; Verga, P.; Belanger, D.; and McCallum, A.
2017. Fast and accurate sequence labeling with iterated dilated convolutions. arXiv preprint arXiv:1702.02098.
Su, B., and Lu, S. 2014. Accurate scene text recognition
based on recurrent neural network. In Asian Conference on
Computer Vision, 35–48. Springer.
Wang, K.; Babenko, B.; and Belongie, S. 2011. End-to-end
scene text recognition. In Computer Vision (ICCV), 2011
IEEE International Conference on, 1457–1464. IEEE.
Wang, T.; Wu, D. J.; Coates, A.; and Ng, A. Y. 2012. End-toend text recognition with convolutional neural networks. In
Pattern Recognition (ICPR), 2012 21st International Conference on, 3304–3308. IEEE.
Wang, F.; Jiang, M.; Qian, C.; Yang, S.; Li, C.; Zhang, H.;
Wang, X.; and Tang, X. 2017. Residual attention network
for image classification. arXiv preprint arXiv:1704.06904.
Yao, C.; Bai, X.; Shi, B.; and Liu, W. 2014. Strokelets: A
learned multi-scale representation for scene text recognition.
In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 4042–4049.

