arXiv:1711.10123v1 [cs.DC] 28 Nov 2017

Homomorphic Parameter Compression for
Distributed Deep Learning Training

Jaehee Jang, Byunggook Na, Sungroh Yoon
Department of Electrical Engineering
Seoul National University
sryoon@snu.ac.kr

Abstract
Distributed training of deep neural networks has received significant research
interest, and its major approaches include implementations on multiple GPUs
and clusters. Parallelization can dramatically improve the efficiency of training
deep and complicated models with large-scale data. A fundamental barrier against
the speedup of DNN training, however, is the trade-off between computation and
communication time. In other words, increasing the number of worker nodes
decreases the time consumed in computation while simultaneously increasing
communication overhead under constrained network bandwidth, especially in
commodity hardware environments. To alleviate this trade-off, we suggest the idea
of homomorphic parameter compression, which compresses parameters with the
least expense and trains the DNN with the compressed representation. Although
the specific method is yet to be discovered, we demonstrate that there is a high
probability that the homomorphism can reduce the communication overhead, thanks
to little compression and decompression times. We also provide theoretical speedup
of homomorphic compression.

1

Introduction

Deep learning (DL) derives structured information from raw data using deep neural networks (DNN).
DL finds hierarchical representations of data through several non-linear layers of a DNN. When
the problem to be solved by using DL is challenging, we need to grasp complicated representations
from the data. With the use of DNNs to solve an increasing number of high-abstraction problems
in various fields, the size of training models and the computational load to train the models have
continued to grow. Under current software and hardware constraints, DNN training demands a
massive amount of processing time [1], naturally leading to the need for distributed deep learning
naturally uprose [2, 3, 4, 5, 6]. Distributed DL divides the workload (training data or model) to
different machines and aims for faster learning while maintaining the original performance of the
model.
DNN iteratively optimizes weight parameters based on gradients computed from feedforward/backpropagation, which is highly sequential. Hence the implementation of distributed DNN
training requires specific design principles and strategies as they have been suggested for years [7]. To
give a brief illustration on how distributed DL is implementated in general, let’s take the synchronous
SGD update scenario as an example (Fig. 1). Synchronous SGD trains by iterating a set of processes
to update global parameters, described by a dotted box in Fig. 1. The set of global parameter update
consists of the following steps. First, all the worker nodes train until the designated number of
iterations (Local Parameter/Gradient Computation in Fig. 1). Then, all the worker nodes respectively
push their local parameters to the parameter server (Local Parameter Transfer in Fig. 1). Lastly, the
parameter server decides global parameters by aggregating all the pushed local parameters(Global
Parameter Update in Fig. 1), and pulls them to the worker nodes(Global Parameter Broadcast in
Fig. 1).
The worker nodes participating in the training frequently exchange their training status with other
nodes so that the model can reflect all the divided workloads. However, DNN models are large and so
the communication load. Hence they cause bottlenecks on transmission because of the contrained

Worker
3

Local
Parameter/
Gradient
Computation

Local
Parameter
Transfer

Local
Parameter/
Gradient
Computation

Local
Parameter
Transfer

Global Parameter
Broadcast

Worker
2

Exchange 2
Global Parameter
Update

Worker
1

Global Parameter
Broadcast

Exchange 1
Global Parameter
Update

Start

...

Computation time Parameter transfer time

Figure 1: Simplified process demonstration of distributed DL using synchronous SGD

communication bandwidth. Especially under commodity hardware environments, the weight-transfer
time overwhelms even the computing time. Along with the transmission time, the time technically
required for communication, that is, the time to prepare and sustain communication, is also included
as communication overhead. The communication overhead is one of the main factors that increase
the parallel training time. In order to alleviate the communication overhead, attempts have been made
to reduce the model size before communication [8, 9, 10].
The goal of our study is to demonstrate homomorphic parameter compression, which is a novel concept of compressed deep learning. As the term homomorphic suggests, it is a compression method that
reduces the size of parameters and allows key operations of DL to be executed without decompression.
Since parameters are transferred numerous times during distributed training settings, this method
can remarkably reduce the time consumed in communication, which is the main rate-limiting step
of distributed training. Furthermore, homomorphism prevents the generation of additional overhead
by repetitive compression and decompression. The main contributions of this paper include the
followings: 1) To our knowledge, this is the first attempt to demonstrate homomorphic parameter
compression. 2) We theoretically characterize the possible factors in the parameter compression,
e.g., the compression ratio, and provide thorough simulative analyses. 3) We provide the theoretical
reduction in training time of the homomorphically compressed distributed training in function of the
number of participating worker nodes for different values of the compression ratio.

2

Literature Survey on Compressed Deep Learning

Numerous studies have suggested compression in deep learning [11, 12, 13, 14]. Existing compression
methods aim at fitting very-large-scale models into a mobile device or single FPGA chip, at alleviating
the high communication overhead due to distributed training, and at improving computational
performance as well as storage and power efficiency.
Post-training compression for inference. A series of studies reduced the storage and energy
required to run inference on large DL models and deploy them on embedded systems or mobile
devices. Deep compression [11] used pruning, trained quantization, and Huffman encoding on weights
and demonstrated a high compression ratio to fit in on-chip memory. CNNpack [15] demonstrated
convolutional neural network (CNN) compression in the discrete cosine transform (DCT) frequency
domain. These methods can effectively reduce the size of networks while retaining pre-trained
information. On the other hand, since they are designed for compression after training is completed,
the time consumed in compression is a minor issue.
In-training compression for refficient deep learning. Employing compression in training enables
efficient computation and communication under limited resources. Especially when transferring
parameters in distributed DL, the constrained network bandwidth may consume a large amount of
time in communication and slow down the entire training process. The following approaches proposed
compression for both training and inference in order to improve computational performance as well as
energy and storage efficiency. We classified the approaches into two types: repetitive (de)compression
and one-time compression. The training process of the methods is illustrated in Fig. 2.
Repetitive (de)compression Some methods encode weights (or gradients) for every iteration We
call such methods repetitive (de)compression methods. Weight binarization methods [12, 13] binarize
weights in order to train from low-power devices or specialized hardware. The binarized weights are
used only during the forward and back propagations but not during parameter update. The authors of
[14, 16] used gradient binarization with distributed DL training in order to reduce the communication
overhead. They encoded gradients during global parameter update, and worker nodes have to decode
the gradients to update their local parameters. FreshNet [17] combined hashing [18] with DCT
2

Parameter Decompression

Global Parameter
Broadcast

...

Global Parameter
Broadcast

Exchange 2
Global Parameter
Update

Local
Parameter/
Gradient
Computation

Parameter Decompression
Global Parameter
Update
Parameter Compression

Parameter Compression

Local
Parameter/
Gradient
Computation

Local Parameter Transfer

Worker
3

Local
Parameter/
Gradient
Computation

Global Parameter
Update

Worker
2

Exchange 2

Exchange 1
Local Parameter Transfer

(b)

Parameter Compression

Start
Worker
1

Parameter Decompression

Worker
3

Local
Parameter
Transfer

Global Parameter
Broadcast

Local
Parameter/
Gradient
Computation

Worker
2

Global Parameter
Broadcast

(a)

Parameter Decompression
Global Parameter
Update
Parameter Compression

Exchange 1
Parameter Compression

Start
Worker
1

...

Figure 2: Simplified process demonstration of mid-training compression methods, based on synchronous SGD: (a) repetitive (de)compression, and (b) one-time compression. Note that we considered
parameter transfer time as the time in which a worker node is not training but waiting for new gradient
update. (Best viewed in color)
Table 1: Examples of Gzip compression
Size

AlexNet Caffemodel
ILSVR2012 Train Data
CIFAR-100 Train Data

Compression
ratio
233MB
1.079
240GB
1.269
147MB
1.097

Compression
time
8.079s
10 532.177s
660.876s

Decompression
time
1.898s
3886.498s
2.242s

to compress CNN models and train in the frequency domain. Good compression performance
and robustness in model accuracy were demonstrated . However, continuous compressions and
decompressions involve high risks for additional compression overhead, as shown in Fig. 2(b).
Although good compression performance was demonstrated, more careful considerations are needed
to utilize the aforementioned methods in distributed training, as is done with QSGD [16] by double
buffering. A quantitative analysis of compression overhead will be presented in Section 4.
One-time compression If DL models are compressed only once, the compression time will not significantly affect the overall training time. Compressed linear algebra (CLA) [10] exploits lightweight
database compression techniques to compress matrices and perform computations in the compressed
representation. Despite the compression ratio and the operation performance being close to that of
uncompressed operations, it is difficult to be applied directly to distributed DL training because more
nonlinear operations are required in DL training. Especially, operations that are frequently used in DL
training, such as normalization and pooling, are not yet conducted in the compressed representation 1 .

3

Algorithmic Design

Communication overhead is one of the major drawbacks of distributed DL, and compressing the
parameters can reduce the communication workload. On the other hand, if the parameters are compressed and decompressed at every update, there are high risks for additional compression overhead.
As indicated in Table 1, compressing and decompressing parameters can take considerable time.
Therefore, we need a compression approach that does not increase the computing time significantly
while reducing the size of parameters properly.
We propose homomorphic parameter compression, inspired from homomorphic encryption [19].
Homomorphism suggests an algebraic system that is encoded from another algebraic system and
performs operations equivalent to those of the encoded system. Our goal is to propose a compression
method that can be trained without decompression. By referring to the early formulation of homomorphic encryption [19], the definition of homomorphic compression is as follows. Suppose we have
a system S =< W ; f1 , f2 , ... > that consists of a set of parameters W and operations fi s concerned
with training. The possible fi s may vary depending on the model structure. As Fig. 3 shows, linear
operations take the majority of the operations and nonlinear operations such as pooling and relu are
1 The

confirmed version of September 2017 can be found at https://github.com/apache/systemml

3

62.83% gemm
17.78% img to col transformation
6.08% normalization
4.13% axpy
3.88% pooling
2.81% optimization (SGD)
2.38% relu
0.07% dropout
0.04% softmax+loss

Figure 3: GPU kernel analysis of AlexNet training (Caffe)

included as well. We propose finding the encoding function φ : S → S0 , where S0 =< W 0 ; f00 , f20 , ... >,
where is the compressed system.
1. An encoded version of a weight w0i = φ (wi ) should be smaller than original weight wi .
2. φ should be easy to compute. Conversion by φ should not take too much time. We point
out how compression overhead can slow down the total training time in Fig. 5. We can
continue training even without decompression if the compression time is long enough to
affect the total training time, as it is difficult to expect temporal gain through homomorphic
compression in such a case.
3. The operations fi0 should be efficiently computable. When training DL, varied operation
functions ( fi s) are required, such as matrix multiplication and activation functions, as
shown in Fig. 2 in Supplement. If we encode the functions to equivalent fi0 operations, the
computational efficiency of fi0 operations is also required to be high.

4
4.1

Experimental Study
Experimental Settings

Notations & formulated assumptions We assumed of a distributed training environment where
there are M worker nodes. It parallelizes a single-node training with minibatch size B on target dataset
of size D. The single node consumes C of time when computing a minibatch, and the total size of
weight parameters is measured as W.
Optimization Scheme When conducting distributed training, we can define various optimization
schemes according to when local parameters have been updated (worker nodes have been trained) with
global parameters (parameters that all worker nodes share). Communication overhead is inevitable
regardless of the strategy we shall choose. In order to emphasize the effect of communication
overhead with different numbers of worker nodes, we assumed that we optimize the DNN training by
synchronous stochastic gradient descent (synchronous SGD).
Synchronous SGD trains by iterating a set of processes to update global parameters, as described in
Fig. 1. In this paper, we defined the time required in the local parameter/gradient computation step
until the designated number of iterations i as computation time, Tcmt , and the time required in the
remaining steps as parameter transfer time, Ttn f which adds up to the time required for one set of
global parameter update, Tupdate = Tcmt + Ttn f .
Minibatch size, B → b We assumed data-parallelized training, which divides the training dataset D
into M worker nodes. As the dataset is divided, the ratio of the original minibatch size to training
data size becomes larger. If the batch size is too large, the training may be delayed [20]. Hence it is
assumed that the minibatch size is reduced by the reduction amount of the training set. Therefore,
B
b= M
.
Minibatch computation time, C → c As the minibatch size decreases by
C
one iteration is expected to decrease by the same ratio. Therefore, c = M
4

1
M,

the time consumed for

Time (sec)

Time (sec)

16
120
100
80
60
40
20
0

Computation
time
Parameter
transfer time

12
8
4
0

Single

2
4
8
16
32
Number of worker nodes
(a)

64

Single

2
4
8
16
32
Number of worker nodes
(b)

64

Figure 4: Simulated analysis of Tcmt vs. Ttn f on vanilla synchronous SGD training. We conducted
experiments for one set of global parameter update. (a) 200-iter AlexNet training (Caffe). (b) 20-iter
STREET training (TensorFlow).

Time (sec)

Time (sec)

240
160
80
0

Single

2
4
8
16
32
Number of worker nodes
(a)

64

30
25
20
15
10
5
0

Computation
time
Parameter
transfer time
Compression/
decompression time
Single

2
4
8
16
32
Number of worker nodes
(b)

64

Figure 5: Simulated analysis of Tcmt vs. Ttn f on vanilla synchronous SGD training with a common
compression method. We conducted experiments for one set of global parameter update. We assumed
Gzip compression/decompresion at every parameter update. (a) 200-iter AlexNet training (Caffe). (b)
20-iter STREET training (TensorFlow).
Computation time per update, Tcmt Since the data and minibatch size are decreased by M1 , training
iterations are required to be tje same as single-node training in order to train the same number of
epochs. Hence, Tcmt = i · c = i·C
M.
Parameter transfer time, Ttn f In synchronous SGD, every local parameter of the worker node is
collected when updating a global parameter. If the number of participating worker nodes increases, the
number of parameters to be exchanged linearly increases. That is, if we define the size of the weight
parameter as W and train with M worker nodes, the number of parameters required to communicate
is W · M. Letting the transmission rate of the cluster be χ, the parameter transfer time is denoted as
Ttn f = Wχ·M .
From the assumptions stated in Section 3, we simulated distributed training for CNN and RNN models.
We trained the ImageNet dataset [21] using AlexNet [22] to simulate distributed CNN training. For
simulative distributed RNN training, we conducted STREET model [23] training on the French Street
Name Signs (FSNS) dataset 2 . We used Caffe [24] for the CNN model and TensorFlow [6] for the
RNN model as computing engines. Synchronous SGD iterates the same processes of global parameter
update, where the worker nodes train up to the designated iterations and communicate parameters to
apply the global training trend. Hence, Figs. 4, 5, 6, and Fig. 7(b), (c) represent only one set of global
updates for the total training trend in terms of time.
The hardware we used in the simulation is a commodity cluster. We used a homogeneous cluster
consisting of 25 identical machines connected via Gigabit Ethernet. Each worker node has an Intel
Core i7-4790 processor with 16GB of main memory and an NVIDIA GTX970 GPU.
4.2

Simulated Analysis of Communication Overhead Effect and Naïve Parameter
Compression

Fig. 4 shows the simulated global parameter update of vanilla synchronous SGD. As the number
of nodes increases, the computation time decreases but the parameter transfer time increases. At
some points, parameter transfer takes more time than minibatch computation, which demonstrates the
serious inefficiency of resource utility due to communication overhead in distributed training. If we
keep increasing the number of nodes, the parameter transfer time even exceeds the single minibatch
computation time, becoming a hindrance rather than contributing to speedup.
If we can compress weight parameters by ρx, the parameter transfer requirement will be reduced
by 1r . This will also reduce the time required for parameter transfer. However, the time consumed in
2 TensorFlow

implementation at https://github.com/tensorflow/models/tree/master/street

5

(a)
ρ=0.2

(b) ρ=0.2
ρ=0.5

13

Computation time
50

Ideal

11

7
HPC

5
Vanilla
SSGD

3
1
1

2
4
8 16 32
Number of worker nodes

Time (sec)

40

9
Speedup

(c) ρ=0.5
Parameter transfer time

30

Vanilla
Sync SGD

20

Theoretical
lower bound
Ideal

10
0

1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0
h (Computational overhead)

Figure 6: (a) Theoretical speedup as a function of the number of worker nodes for different compression ratios when training the ImageNet with AlexNet. (b)&(c) Simulated analysis of the proposed
method for 200-iter AlexNet training on 16 worker nodes, when h = 1.0, 1.1, ...2.0 and (b) ρ = 0.2.
and (c) ρ = 0.5. We illustrated the expected Tcmt and Ttn f for one set of global parameter update.
Compression and decompression are not shown in these graphs because they are only performed
once throughout the training.(Best viewed in color)
compressing parameters is a problem. Suppose after a worker node finished its batch, it compresses
the trained parameter using Gzip. Then, the parameter server aggregates the compressed parameters
and since Gzip-compressed data are not computable, the parameter server has to decompress the
parameters to aggregate and average. After global parameters are set, the parameter server compresses
the parameters again and sends them back to the worker nodes. When a worker node receives the
compressed parameters, it has to decompress them again to keep training.
If compressed training is conducted as explained above, there is a high possibility of another overhead
called compression overhead. We simulated this in Fig. 4, and the results are shown in Fig. 5. If
we use Gzip compression, the parameter transfer time is decreased because of the reduced size of
parameters. On the other hand, the compression and decompression time can significantly exceed the
original training time, as shown in Fig. 5.
4.3

The Expected Gain of Homomorphic Compression

From the simulation results in Section 4, we can learn two things. First, compressing the parameter
size can reduce the communication workload, and second, we need a compression approach that takes
compression overhead into account. Therefore, we propose homomorphic parameter compression
here.
Fig. 6(a) shows the theoretical speedup of homomorphically compressed distributed training based
on the simulative analysis conducted in this section. It is represented as a function of the number
of participating worker nodes for different compression ratios. The orange dotted curve in Fig. 6(a)
shows the ideal distributed training case, where there is no communication at all so we can achieve
linear speedup. The yellow dashed curve is the speedup of vanilla SGD. The ideal speedup in
homomorphic compression occurs when there is no overhead due to the increased operation time.
Hence, the green solid and red double solid curves are the theoretical upper bounds in speedup when
the compression ratio is 0.2 and 0.5 respectively.
In actual homomorphic compression, the encoded operations are likely to require more time than the
original operations. The simulated results based on this assumption are shown in Fig. 7. Note that
the compression time in Fig. 7 is not included in every parameter update but only in the early stage.
We assumed that computing in the compressed representation may take more time than the original
computations. However, if we can manipulate the operation overhead and compression ratio, fast and
large-scale DL training is attainable, as shown in Fig. 7.

5

Discussion and Future Work

We analyzed the effect of homomorphic compression on distributed training in Section 4. In this section, we discuss the parameters required when designing a homomorphic compression method. Below,
we present an in-depth analysis on the computational efficiency for fi0 ’s based on the assumptions
made in Section 3 and Section 4:
Let ρ be a compression ratio, ρ =

φ (wi )
wi

(where 0 < ρ < 1), and T fi ) and T fi0 be the time consumed

in performing fi and fi0 respectively. And we define operation overhead h as h =
6

Tf 0
i

T fi

. Then, the

Computation time

With Gzip
compression
Vanilla
Sync SGD

(De)Compression time
24

50

20
Time (sec)

Time (sec)

HPC
(proposed)

Parameter transfer time

60
40
30
20
10
0

1×10 5 2×10 5
Time (sec)
(a)

3×105

16
12
8
4

0
Single 2
4
8
16
32
(uncompressed) Number of worker nodes
(b)

64

0
Single 2
4
8
16
32
(uncompressed) Number of worker nodes

64

(c)

Figure 7: Simulated analysis of the proposed method. (a) Comparison of total Alexnet training time
among the proposed method, synchronous SGD with Gzip compression, and vanilla synchronous
SGD. (b)&(c) Tcmt vs. Ttn f on synchronous SGD training with proposing method. We illustrated the
expected result of one set of global parameter update. Compression and decompression shown in the
graphs are performed only once throughout the training. (b) 200-iter AlexNet training (Caffe) when
h = 1.3, ρ = 0.2. (c) 20-iter STREET training (TensorFlow) when h = 1.5, ρ = 0.5.
0 and parameter transfer time T 0 are expressed as
compressed minibatch computation time Tcmt
tn f
C
MW
0
0
C · Tcmt = M · h, Ttn f = BW · ρ, respectively.
C
By setting the upper bound of the total training time as M
· r (where 1 ≤ r ≤ M), the relationship
between h and ρ can be obtained as expressed by Eq. ??. Therefore, we can achieve the desired
speedup if we can fit h and ρ under Eq. ??.

C
C
MW
·r ≥ ·h+
·ρ
M
M
BW
M 2W
∴h≤−
·ρ +r
CBW
Fig. 6(b) expected tendency of training time with respect to h and ρ. The most ideal training time for
C
M worker nodes to learn a model that takes batch computation time C is M
(orange dotted line in
Fig. 6(b)). However, even when we train in parallel, the learning status among nodes is interchanged.
Therefore, the realistic training time of synchronous SGD, considering communication time, is the
same as the red line in Fig. 6(b).
In addition to the upper bound the computation time for fi0 , S0 and φ are needed to be designed in
consideration of frequently used operations. Fig. 8 shows the GPU profiling result of AlexNet training
with Caffe, and it suggests that operations such as gemm take most of the computation time. It is
expected that, if we significantly reduce the time required for computing gemm, the operation overhead
effect will be much weaker. Our future work is to propose a detailed homomorphic compression
method.

References
[1] J. Keuper et al. Distributed training of deep neural networks: theoretical and practical limits of
parallel scalability. In Proceedings of the Workshop on Machine Learning in High Performance
Computing Environments. IEEE Press, 2016.
[2] R. Bekkerman, et al. Scaling up machine learning: Parallel and distributed approaches.
Cambridge University Press, 2011.
[3] J. Dean, et al. Large scale distributed deep networks. In Advances in neural information
processing systems, 2012.
[4] B. Recht, et al. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In
Advances in neural information processing systems, pages 693–701, 2011.
[5] H. Kim, et al. Deepspark: Spark-based deep learning supporting asynchronous updates and
caffe compatibility. CoRR, vol. abs/1602.08191, 2016.
[6] M. Abadi, et al. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015.
Software available from tensorflow.org.
[7] E. P. Xing, et al. Strategies and principles of distributed machine learning on big data. Engineering, 2(2):179–195, 2016.
7

[8] F. N. Iandola, et al. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5
mb model size. arXiv preprint arXiv:1602.07360, 2016.
[9] R. Spring et al. Scalable and sustainable deep learning via randomized hashing. arXiv preprint
arXiv:1602.08194, 2016.
[10] A. Elgohary, et al. Compressed linear algebra for large-scale machine learning. Proceedings of
the VLDB Endowment, 9(12):960–971, 2016.
[11] S. Han, et al. Deep compression: Compressing deep neural networks with pruning, trained
quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
[12] M. Courbariaux, et al. Binaryconnect: Training deep neural networks with binary weights
during propagations. In Advances in Neural Information Processing Systems, 2015.
[13] M. Courbariaux, et al. Binarized neural networks: Training deep neural networks with weights
and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.
[14] F. Seide, et al. 1-bit stochastic gradient descent and its application to data-parallel distributed
training of speech dnns. In Fifteenth Annual Conference of the International Speech Communication Association, 2014.
[15] Y. Wang, et al. Cnnpack: Packing convolutional neural networks in the frequency domain. In
Advances in Neural Information Processing Systems 29. 2016.
[16] D. Alistarh, et al. Qsgd: Randomized quantization for communication-optimal stochastic
gradient descent. arXiv preprint arXiv:1610.02132, 2016.
[17] W. Chen, et al. Compressing convolutional neural networks in the frequency domain. In
Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery
and Data Mining, KDD ’16, pages 1475–1484, New York, NY, USA, 2016. ACM.
[18] W. Chen, et al. Compressing neural networks with the hashing trick. In Proceedings of the
32Nd International Conference on International Conference on Machine Learning - Volume 37,
ICML’15, pages 2285–2294. JMLR.org, 2015.
[19] R. L. Rivest, et al. On data banks and privacy homomorphisms. Foundations of secure
computation, 4(11):169–180, 1978.
[20] Y. Bengio. Practical recommendations for gradient-based training of deep architectures. In
Neural networks: Tricks of the trade. Springer, 2012.
[21] J. Deng, et al. Imagenet: A large-scale hierarchical image database. In Computer Vision and
Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. IEEE, 2009.
[22] A. Krizhevsky, et al. Imagenet classification with deep convolutional neural networks. In
Advances in neural information processing systems, 2012.
[23] R. Smith, et al. End-to-End Interpretation of the French Street Name Signs Dataset. Springer
International Publishing, 2016.
[24] Y. Jia, et al. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint
arXiv:1408.5093, 2014.

8

