D IRAC N ETS : T RAINING V ERY D EEP N EURAL N ETWORKS W ITHOUT S KIP -C ONNECTIONS

arXiv:1706.00388v2 [cs.CV] 26 Jan 2018

Sergey Zagoruyko, Nikos Komodakis
Université Paris-Est, École des Ponts ParisTech
Paris, France
{sergey.zagoruyko,nikos.komodakis}@enpc.fr

A BSTRACT
Deep neural networks with skip-connections, such as ResNet, show excellent performance in various image classification benchmarks. It is though observed that
the initial motivation behind them - training deeper networks - does not actually
hold true, and the benefits come from increased capacity, rather than from depth.
Motivated by this, and inspired from ResNet, we propose a simple Dirac weight
parameterization, which allows us to train very deep plain networks without explicit skip-connections, and achieve nearly the same performance. This parameterization has a minor computational cost at training time and no cost at all at
inference, as both Dirac parameterization and batch normalization can be folded
into convolutional filters, so that network becomes a simple chain of convolutionReLU pairs. We are able to match ResNet-1001 accuracy on CIFAR-10 with
28-layer wider plain DiracNet, and closely match ResNets on ImageNet. Our parameterization also mostly eliminates the need of careful initialization in residual
and non-residual networks. The code and models for our experiments are available
at https://github.com/szagoruyko/diracnets

1

I NTRODUCTION

There were many attempts of training very deep networks. In image classification, after the success of Krizhevsky et al. (2012) with AlexNet (8 layers), the major improvement was brought by
Simonyan & Zisserman (2015) with VGG (16-19 layers) and later by Szegedy et al. (2015) with
Inception (22 layers). In recurrent neural networks, LSTM by Hochreiter & Schmidhuber (1997)
allowed training deeper networks by introducing gated memory cells, leading to major increase of
parameter capacity and network performance. Recently, similar idea was applied to image classification by Srivastava et al. (2015) who proposed Highway Networks, later improved by He et al.
(2015a) with Residual Networks, resulting in simple architecture with skip-connections, which was
shown to be generalizable to many other tasks. There were also proposed several other ways of
adding skip-connections, such as DenseNet by Huang et al. (2016), which passed all previous activations to each new layer.
Despite the success of ResNet, a number of recent works showed that the original motivation of
training deeper networks does not actually hold true, e.g. it might just be an ensemble of shallower
networks Veit et al. (2016), and ResNet widening is more effective that deepening Zagoruyko &
Komodakis (2016), meaning that there is no benefit from increasing depth to more than 50 layers.
It is also known that deeper networks can be more efficient than shallower and wider, so various
methods were proposed to train deeper networks, such as well-designed initialization strategies and
special nonlinearities Bengio & Glorot (2010); He et al. (2015b); Clevert et al. (2015), additional
mid-network losses Lee et al. (2014), better optimizers Sutskever et al. (2013), knowledge transfer Romero et al. (2014); Chen et al. (2016) and layer-wise training Schmidhuber (1992).
To summarize, deep networks with skip-connections have the following problems:
• Feature reuse problem: upper layers might not learn useful representations given previous
activations;
• Widening is more effective than deepening: there is no benefit from increasing depth;
1

• Actual depth is not clear: it might be determined by the shortest path.
However, the features learned by such networks are generic, and they are able to train with massive
number of parameters without negative effects of overfitting. We are thus interested in better understanding of networks with skip-connections, which would allow us to train very deep plain (without
skip-connections) networks and benefits they could bring, such as higher parameter efficiency, better
generalization, and improved computational efficiency.
Motivated by this, we propose a novel weight parameterization for neural networks, which we call
Dirac parameterization, applicable to a wide range of network architectures. Furthermore, by use of
the above parameterization, we propose novel plain VGG and ResNet-like architecture without explicit skip-connections, which we call DiracNet. These networks are able to train with hundreds of
layers, surpass 1001-layer ResNet while having only 28-layers, and approach Wide ResNet (WRN)
accuracy. We should note that we train DiracNets end-to-end, without any need of layer-wise pretraining. We believe that our work is an important step towards simpler and more efficient deep
neural networks.
Overall, our contributions are the following:
• We propose generic Dirac weight parameterization, applicable to a wide range of neural
network architectures;
• Our plain Dirac parameterized networks are able to train end-to-end with hundreds of layers. Furthermore, they are able to train with massive number of parameters and still generalize well without negative effects of overfitting;
• Dirac parameterization can be used in combination with explicit skip-connections like
ResNet, in which case it eliminates the need of careful initialization.
• In a trained network Dirac-parameterized filters can be folded into a single vector, resulting
in a simple and easily interpretable VGG-like network, a chain of convolution-ReLU pairs.

2

D IRAC PARAMETERIZATION

Inspired from ResNet, we parameterize weights as a residual of Dirac function, instead of adding
explicit skip connection. Because convolving any input with Dirac results in the same input, this
helps propagate information deeper in the network. Similarly, on backpropagation it helps alleviate
vanishing gradients problem.
Let I be the identity in algebra of discrete convolutional operators, i.e. convolving it with input x
results in the same output x ( denotes convolution):
I

x=x

(1)

In two-dimensional case convolution might be expressed as matrix multiplication, so I is simply an
identity matrix, or a Kronecker delta δ. We generalize this operator to the case of a convolutional
layer, where input x ∈ RM,N1 ,N2 ,...,NL (that consists of M channels of spatial dimensions (N1 , N2 ,
..., NL )) is convolved with weight Ŵ ∈ RM,M,K1 ,K2 ,...,KL (combining M filters1 ) to produce an
output y of M channels, i.e. y = Ŵ x. In this case we define Dirac delta I ∈ RM,M,K1 ,K2 ,...,KL ,
preserving eq. (1), as the following:

1 if i = j and lm ≤ Km for m = 1..L,
I(i, j, l1 , l2 , . . . , lL ) =
(2)
0 otherwise;
Given the above definition, for a convolutional layer y = Ŵ x we propose the following parameterization for the weight Ŵ (hereafter we omit bias for simplicity):
y = Ŵ

x,

Ŵ = diag(a)I + W,
1

outputs are over the first dimension of Ŵ, inputs are over the second dimension of Ŵ

2

(3)
(4)

where a ∈ RM is scaling vector learned during training, and W is a weight vector. Each i-th
element of a corresponds to scaling of i-th filter of W. When all elements of a are close to zero,
it reduces to a simple linear layer W x. When they are higher than 1 and W is small, Dirac
dominates, and the output is close to be the same as input.
We also use weight normalization Salimans & Kingma (2016) for W, which we find useful for
stabilizing training of very deep networks with more than 30 layers:
Ŵ = diag(a)I + diag(b)Wnorm ,

(5)

where b ∈ RM is another scaling vector (to be learned during training), and Wnorm is a normalized
weight vector where each filter is normalized by it’s Euclidean norm. We initialize a to 1.0 and b
to 0.1, and do not l2 -regularize them during training, as it would lead to degenerate solutions when
their values are close to zero. We initialize W from normal distribution N (0, 1). Gradients of (5)
can be easily calculated via chain-rule. We rely on automatic differentiation, available in all major
modern deep learning frameworks (PyTorch, Tensorflow, Theano), to implement it.
Overall, this adds a negligible number of parameters to the network (just two scaling multipliers per
channel) during training, which can be folded into filters at test time.

2.1

C ONNECTION TO R ES N ET

Let us discuss the connection of Dirac parameterization to ResNet. Due to distributivity of convolution, eq. (3) can be rewritten to show that the skip-connection in Dirac parameterization is implicit
(we omit a for simplicity):


y = σ (I + W) x = σ x + W x ,
(6)
where σ(x) is a function combining nonlinearity and batch normalization. The skip connection in
ResNet is explicit:
y = x + σ(W x)
(7)
This means that Dirac parameterization and ResNet differ only by the order of nonlinearities. Each
delta parameterized layer adds complexity by having unavoidable nonlinearity, which is not the
case for ResNet. Additionally, Dirac parameterization can be folded into a single weight vector on
inference.

3

E XPERIMENTAL RESULTS

We adopt architecture similar to ResNet and VGG, and instead of explicit skip-connections use
Dirac parameterization (see table 1). The architecture consists of three groups, where each group
has 2N convolutional layers (2N is used for easier comparison with basic-block ResNet and WRN,
which have N blocks of pairs of convolutional layers per group). For simplicity we use max-pooling
between groups to reduce spatial resolution. We also define width k as in WRN to control number
of parameters.
We chose CIFAR and ImageNet for our experiments. As for baselines, we chose Wide ResNet with
identity mapping in residual block He et al. (2016) and basic block (two 3 × 3 convolutions per
block). We used the same training hyperparameters as WRN for both CIFAR and ImageNet.
The experimental section is composed as follows. First, we provide a detailed experimental comparison between plain and plain-Dirac networks, and compare them with ResNet and WRN on CIFAR.
Also, we analyze evolution of scaling coefficients during training and their final values. Then, we
present ImageNet results. Lastly, we apply Dirac parameterization to ResNet and show that it eliminates the need of careful initialization.
3.1

P LAIN NETWORKS WITH D IRAC PARAMETERIZATION

In this section we compare plain networks with plain DiracNets. To do that, we trained both with
10-52 layers and the same number of parameters at the same depth (fig. 1). As expected, at 10 and
3

name

output size

layer type

conv1
group1
max-pool
group2
max-pool
group3
avg-pool

32 × 32
32×32
16×16
16×16
8×8
8×8
1×1

[3×3, 16]
[3×3, 16 × 16k]×2N
[3×3, 32k × 32k]×2N
[3×3, 64k × 64k]×2N
[8 × 8]

Table 1: Structure of DiracNets. Network width is determined by factor k. Groups of convolutions are shown
in brackets as [kernel shape, number of input channels, number of output channels] where 2N is a number of
layers in a group. Final classification layer and dimensionality changing layers are omitted for clearance.

16 layers there is no difficulty in training plain networks, and both plain and plain-Dirac networks
achieve the same accuracy. After that, accuracy of plain networks quickly drops, and with 52 layers
only achieves 88%, whereas for Dirac parameterized networks it keeps growing. DiracNet with 34
layers achieves 92.8% validation accuracy, whereas simple plain only 91.2%. Plain 100-layer network does not converge and only achieves 40% train/validation accuracy, whereas DiracNet achieves
92.4% validation accuracy.
96
24.47M
6.12M

95

7.40M
1.86M
2.74M

94

0.69M

93

0.69M
1.19M

1.54M
6.13M

1.85M
0.47M

1.54M

92
91

0.30M
0.30M0.18M

0.47M

0.17M
0.17M

DiracNet, width=1
DiracNet, width=2
DiracNet, width=4
ResNet, width=1
ResNet, width=2
plain, width=1

90
89
88
20

40

60

80

100

Figure 1: DiracNet and ResNet with different depth/width, each circle area is proportional to number of parameters. DiracNet needs more width (i.e. parameters) to match ResNet accuracy. Accuracy is calculated as
median of 5 runs.

3.2

P LAIN D IRAC NETWORKS AND RESIDUAL NETWORKS

To compare plain Dirac parameterized networks with WRN we trained them with different width k
from 1 to 4 and depth from 10 to 100 (fig. 1). As observed by WRN authors, accuracy of ResNet
is mainly determined by the number of parameters, and we even notice that wider networks achieve
better performance than deeper. DiracNets, however, benefit from depth, and deeper networks with
the same accuracy as wider have less parameters. In general, DiracNets need more parameters than
ResNet to achieve top accuracy, and we were able to achieve 95.25% accuracy with DiracNet-2810 with 36.5M parameters, which is close to WRN-28-10 with 96.0% and 36.5M parameters as
well. We do not observe validation accuracy degradation when increasing width, the networks still
perform well despite the massive number of parameters, just like WRN. Interestingly, plain DiracNet
with only 28 layers is able to closely match ResNet with 1001 layers (table 2)
4

depth-width
NIN, Lin et al. (2013)
ELU, Clevert et al. (2015)
VGG
DiracNet (ours)
ResNet
Wide ResNet

# params

CIFAR-10

CIFAR-100

16
28-5
28-10

20M
9.1M
36.5M

8.81
6.55
6.09±0.11
5.16±0.14
4.75±0.16

35.67
24.28
25.92±0.09
23.44±0.14
21.54±0.18

1001-1
28-10

10.2M
36.5M

4.92
4.00

22.71
19.25

Table 2: CIFAR performance of plain (top part) and residual (bottom part) networks on with horizontal flips
and crops data augmentation. DiracNets outperform all other plain networks by a large margin, and approach
residual architectures. No dropout it used. For VGG and DiracNets we report mean±std of 5 runs.

a¯j

group0

group1
1.8

1.8

1.6

1.6

1.6

1.4

1.4

1.4

1.2

1.2

1.2

1.0

1.0
0

b¯j

group2

1.8

50

100

150

200

1.0
0

50

100

150

200

0.5

0.5

0.5

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0.0

0.0
0

50

100

150

200

0

50

0

50

100

150

200

100

150

200

0.0
0

50

epoch

100

150

200

epoch

epoch

Figure 2: Average values of a and b during training for different layers of DiracNet-34. Deeper color means
deeper layer in a group of blocks.

3.3

A NALYSIS OF SCALING COEFFICIENTS

As we leave a and b free of l2 -regularization, we can visualize significance of various layers and
how it changes during training by plotting their averages ā and b̄, which we did for DiracNet-34
trained on CIFAR-10 on fig. 2. Interestingly, the behaviour changes from lower to higher groups
of the network with increasing dimensionality. We also note that no layers exhibit degraded a to b
ratio, meaning that all layers are involved in training. We also investigate these ratios in individual
feature planes, and find that the number of degraded planes is low too.
3.4

D IRAC PARAMETERIZATION FOR R ES N ET WEIGHT INITIALIZATION

As expected, Dirac parameterization does not bring accuracy improvements to ResNet on CIFAR,
but eliminates the need of careful initialization. To test that, instead of usually used MSRA init He
et al. (2015b), we parameterize weights as:
Ŵ = I + W,

(8)

omitting other terms of eq. (5) for simplicity, and initialize all weights from a normal distribution
N (0, σ 2 ), ignoring filter shapes. Then, we vary σ and observe that ResNet-28 converges to the same
validation accuracy with statistically insignificant deviations, even for very small values of σ such
as 10−8 , and only gives slightly worse results when σ is around 1. It does not converge when all
5

Network
plain

residual

# parameters

top-1 error

top-5 error

VGG-CNN-S Chatfield et al. (2014)
VGG-16 Simonyan & Zisserman (2015)
DiracNet-18
DiracNet-34

102.9M
138.4M
11.7M
21.8M

36.94
29.38
30.37
27.79

15.40
10.88
9.34

ResNet-18 [our baseline]
ResNet-34 [our baseline]

11.7M
21.8M

29.62
27.17

10.62
8.91

Table 3: Single crop top-1 and top-5 error on ILSVRC2012 validation set for plain (top) and residual (bottom)
networks.
50

50
ResNet-18, 11.69 parameters
DiracNet-18, 11.52 parameters

ResNet-34, 21.80 parameters
DiracNet-34, 21.64 parameters

45

40

40

35

35

top-5 error,

top-5 error,

45

30
25

30
25

20

20

15

15

10

10
0

20

40

60

80

100

0

epoch

20

40

60

80

100

epoch

Figure 3: Convergence of DiracNet and ResNet on ImageNet. Training top-5 error is shown with dashed lines,
validation - with solid. All networks are trained using the same optimization hyperparameters. DiracNet closely
matches ResNet accuracy with the same number of parameters.

weights are zeros, as expected. Additionally, we tried to use the same orthogonal initialization as
for DiracNet and vary it’s scaling, in which case the range of the scaling gain is even wider.
3.5

I MAGE N ET RESULTS

We trained DiracNets with 18 and 34 layers and their ResNet equivalents on ILSVRC2012 image
classification dataset. We used the same setup as for ResNet training, and kept the same number of
blocks per groups. Unlike on CIFAR, DiracNet almost matches ResNet in accuracy (table 3), with
very similar convergence curves (fig. 3) and the same number of parameters. As for simple plain
VGG networks, DiracNets achieve same accuracy with 10 times less parameters, similar to ResNet.
Our ImageNet pretrained models and their simpler folded convolution-ReLU chain variants are
available at https://github.com/szagoruyko/diracnets.

4

D ISCUSSION

We presented Dirac-parameterized networks, a simple and efficient way to train very deep networks
with nearly state-of-the-art accuracy. Even though they are able to successfully train with hundreds
of layers, after a certain number of layers there seems to be very small or no benefit in terms of
accuracy for both ResNets and DiracNets. This is likely caused by underuse of parameters in deeper
layers, and both architectures are prone to this issue to a different extent.
Even though on large ImageNet dataset DiracNets are able to closely match ResNet in accuracy with
the same number of parameters and a simpler architecture, they are significantly behind on smaller
CIFAR datasets, which we think is due to lack of regularization, more important on small amounts
of data. Due to use of weight normalization and free scaling parameters DiracNet is less regularized
than ResNet, which we plan to investigate in future.
6

We also observe that DiracNets share the same property as WRN to train with massive number
of parameters and still generalize well without negative effects of overfitting, which was initially
thought was due to residual connections. We now hypothesize that it is due to a combination of
SGD with momentum at high learning rate, which has a lot of noise, and stabilizing factors, such as
residual or Dirac parameterization, batch normalization, etc.

R EFERENCES
Yoshua Bengio and Xavier Glorot. Understanding the difficulty of training deep feedforward neural
networks. In Proceedings of AISTATS 2010, volume 9, pp. 249–256, May 2010.
K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving
deep into convolutional nets. In British Machine Vision Conference, 2014.
T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accelerating learning via knowledge transfer. In
International Conference on Learning Representation, 2016.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network
learning by exponential linear units (elus). CoRR, abs/1511.07289, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing
human-level performance on imagenet classification. CoRR, abs/1502.01852, 2015b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual
networks. CoRR, abs/1603.05027, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory, 1997.
Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected
convolutional networks. arXiv preprint arXiv:1608.06993, 2016.
A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural
networks. In NIPS, 2012.
C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-Supervised Nets. 2014.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. CoRR, abs/1312.4400, 2013.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. FitNets: Hints for thin deep nets. Technical Report Arxiv report 1412.6550,
arXiv, 2014.
Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to
accelerate training of deep neural networks. In Neural Information Processing Systems 2016,
2016.
J. Schmidhuber. Learning complex, extended sequences using the principle of history compression.
Neural Computation, 4(2):234–242, 1992.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR,
abs/1505.00387, 2015.
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of
initialization and momentum in deep learning. In Sanjoy Dasgupta and David Mcallester (eds.),
Proceedings of the 30th International Conference on Machine Learning (ICML-13), volume 28,
pp. 1139–1147. JMLR Workshop and Conference Proceedings, May 2013.
7

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks are exponential ensembles of relatively shallow networks. CoRR, abs/1605.06431, 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.

8

