arXiv:1801.05895v1 [] 18 Jan 2018

Sparsely Connected Convolutional Networks
Ligeng Zhu*
Simon Fraser University

Ruizhi Deng*
Simon Fraser University

Zhiwei Deng
Simon Fraser University

lykenz@sfu.ca

secondauthor@i2.org

zhiweid@sfu.ca

Greg Mori
Simon Fraser University

Ping Tan
Simon Fraser University

mori@sfu.ca

pingtan@sfu.ca

Abstract

more than 100 layers. Inspired by the success of ResNets,
DenseNets [7] further propose a densely connected structure that has a skip connection between any two layers and
the input to each layer is the concatenation of all the features from previous layers. Such a densely connected structure ensures maximal information flow and facilitates feature reuse as each layer can have direct access to the features from previous layers. DenseNets further push the
boundaries of image recognition accuracy by deep neural
networks.

Residual learning [6] with skip connections permits
training ultra-deep neural networks and obtains superb performance. Building in this direction, DenseNets [7] proposed a dense connection structure where each layer is directly connected to all of its predecessors. The densely connected structure leads to better information flow and feature reuse. However, the overly dense skip connections
also bring about the problems of potential risk of overfitting, parameter redundancy and large memory consumption. In this work, we analyze the feature aggregation patterns of ResNets and DenseNets under a uniform aggregation view framework. We show that both structures densely
gather features from previous layers in the network but combine them in their respective ways: summation (ResNets)
or concatenation (DenseNets). We compare the strengths
and drawbacks of these two aggregation methods and analyze their potential effects on the networks’ performance.
Based on our analysis, we propose a new structure named
SparseNets which achieves better performance with fewer
parameters than DenseNets and ResNets.

Work [18] analyzing the behavior of ResNets see this
connection structure as a collection of paths that choose to
take each shortcut connection or not and attribute the success of ResNets to the existence of abundant short end-toend paths. We apply the same method to DenseNets and
find that the number and distribution of end-to-end paths in
a densely connected structure [7] are the same as a residual connection structure [6] given models with equal depth.
Therefore, short paths seems not to be the answer to why
DenseNets perform better than ResNets.
To better understand the superior performance of
DenseNets, we propose the aggregation view. This reveals that both DenseNets and ResNets densely aggregate
features from prior layers and their essential difference is
how features are aggregated: ResNets aggregate features by
summation and DenseNets aggregate them by concatenation. Many strengths of dense feature aggregation are inherited from those of skip connections including facilitating information and gradient flows as analyzed by previous
work [6, 7, 18]. Aggregation by concatenation brings the
extra benefits of preserving the original feature maps and
giving the model more flexibility to combine the aggregated
features. However, the drawbacks of concatenation in dense
aggregation structures are also more obvious as the network
gets deeper. They include parameter explosion, redundancy
and large memory consumption.

1. Introduction
Recognizing objects in images is one of the fundamental problems in computer vision. Convolutional neural networks (CNNs) have become the standard approach for this
task since the seminal work of AlexNet [11] on large-scale
datasets like ImageNet [2].
Recent work [14, 6, 7, 21, 12] address the problem of
training deeper neural networks by introducing skip connections between layers. Highway Networks [14] and
ResNets [6] create shortcut connections between layers
with an identity mapping and are among the first works
that successfully trained convolutional neural networks with

To take full advantage of feature aggregation via con4321

Figure 1: Connection Structure of SparseNets. Each layer is connected to layers having offsets of power of two from it.
catenation, we propose a new structure that sparsely aggregates features from previous layers: each layer only
takes features from layers that have exponential offsets
away from it. We name this structure sparsely aggregated
convolutional networks, or simply, SparseNets. Experimental results on the CIFAR-10 and CIFAR-100 datasets
demonstrate that SparseNets are able to achieve comparable performance to current state-of-the art models with significantly fewer parameters. It also takes fewer trainable
weights to build deeper and wider networks with our sparse
aggregation structure. As recent work [15] demonstrate that
larger capacity is vital for a model to fully exploit an ultra large dataset (300 times larger than ImageNet [2]) and
the performance of deep learning models can be further improved, we believe SparseNets can permit further scaling in
depth of deep networks.
We summarize our contributions in this paper as follows:

SparseNets in Section 7 and conclude the paper with this
section.

2. Related works
The difficulties of training deep neural networks are
widely acknowledged [1, 3, 5, 9, 14, 19]. Recent work addresses the problems of training deep neural networks with
skip connections. Highway networks [14] and ResNets[6]
are among the first works that successfully train network
structures with more than 100 layers. Both works highlight shortcut connections across nonlinear mappings in network structures. Along each shortcut connection, highway
networks introduce two additional gates: a transform gate
and a carry gate. They assign weights to the original input
and the output of the nonlinear transformation and the final output is the weighted sum. In contrast, ResNets simply
take the plain sum of the original input signal and the output of the nonlinear mapping in each of its building block.
ResNets are a special case of Highway Networks when both
the transform gate and the carry gate always evaluate to one.
ResNets achieved both record-breaking network depth and
image recognition accuracy on ImageNet [2]. The success
of ResNets and Highway Networks reveals the importance
of shortcut connections in training ultra deep CNNs. Variants of ResNets [20, 16, 17, 19] emerged following its impressive success. These variants build wider networks with
residual connections without making significant changes to
the connection pattern of ResNets [6].
DenseNets [7] use a clear and effective skip connection
pattern: every pair of two layers is connected by a shortcut
path. Such densely connected structure ensures the maximal possible gradient and information flow, promotes feature reuse as it provides the largest numbers of skip connections available in a feed-forward neural network. These
benefits come at a heavy cost due to the quadratic asymptotic growth of the number of parameters with respect to the
depth of the network. FractalNet [12] defines fractal archi-

• We proposed the aggregation view that shows the essential similarity and difference between ResNets and
DenseNets and use it to explain the performance improvements made by DenseNets over ResNets.
• We propose SparseNets, a simple feature aggregation
structure with shortcut paths bypassing exponentially
growing number of layers; we also show its empirical
effectiveness on standard datasets for image recognition.
The rest of the paper is organized as follows: in Section 2, we review past work related to training deep neural
networks and recent network structures with skip connections. The formal definition of the aggregation view and
analysis on ResNets and DenseNets are in Section 3. We
present SparseNets based on this analysis and elaborate on
its components in Section 4. Experiments and evaluation
results on the CIFAR-10 and CIFAR-100 datasets are presented in Section 6. We briefly discuss the problems of
short paths and parameter redundancy in DenseNets and
4322

• It consists of convolution, rectified linear unit (ReLU),
batch normalization and pooling with no restrictions
on the number of each operation.

tectures recursively and creates parallel paths of different
length in hierarchy without relying on residual connections.
Yu et al. [21] propose different structures for aggregating the activations across multiple layers in feed-forward
neural networks to get a final output that explicitly combines the representation of different hierarchies. Our aggregation view is inspired by their iterative deep aggregation
structure. Both their work and ours address the importance
of organizing features across layers. However, their work
studies new iterative and hierarchical structures of organizing features. By default, their method of combining features is concatenation followed by convolution. In contrast
our work studies the effects of densely aggregating features
with different aggregation methods in deep neural networks
and our studies are based on the effective, state-of-the-art
ResNets [6] and DenseNets [7].
Many works study the impacts of end-to-end short
paths [18, 19, 8] in ResNets [6]. Deep networks with
stochastic depth [18] randomly bypass certain nonlinear
transformations in ResNets during training so the backpropagated gradients only go through the identity mappings
in some residual blocks. This training algorithm shortens
training time and improves test results compared with training the entire ResNet at each step. The unraveled view [18]
introduced by Veit et al. treats a ResNet as a collection of
short paths. A path either goes through the identity mapping
or not at each skip connection. Therefore the total number
of end-to-end paths in a residual connection structure with
n skip connections is 2n ; the distribution of path lengths is
binomial. Their study also shows that relatively short paths
make most of the contribution in terms of predictions and
backpropagated gradient flows, so they attribute the success
of ResNets to the existence of relatively short paths with
depths in an effective range. Wu et al. [19] further point
out that the long paths in ResNets [6] can not be trained in
an end-to-end manner. The unraveled view and short paths
are problems complementary to our work on increasing network depths and analyses of aggregation methods.

• It takes input from an aggregation function and its output is part of the input to another aggregation function.
• No intermediate operations, except the first and last
one, takes input from or feeds output to an aggregation function.
A nonlinear unit is sometimes also referred to as a layer in
this paper. We use F` (·) to denote the `th basic unit. x`
denotes its input and y` denotes the output. The length of a
path from x`1 to y`2 is the number of basic units along the
path.

3.1. Counting Paths in DenseNets
ResNets. The residual block in ResNets [6] consists of a
nonlinear unit and a shortcut identity mapping. It can be
expressed with the following recursion formula:
y` = F` (x` )
x`+1 = ReLU (y` + x` ).

(1)

DenseNets. The connection pattern in the densely connected structure [7] can be described by the following recursion formula:
y` = F` (x` )
x`+1 = y` ⊗ y`−1 ⊗ ... ⊗ y1

(2)

i.e., x`+1 is the concatenation of all previous layers’ features from 1 to `. ⊗ stands for concatenating two tensors
along the channel dimension. Since concatenation is not
exchangeable, we require consecutive concatenations to be
evaluated from right to left.
We adopt the unraveled view [18] to scrutinize the performance improvements of DenseNets over ResNets from
the perspective of short paths. However, we find that under
the unraveled view the number and distribution of end-toend paths in a densely connected structure is the same as
the residual connection structure given they have the same
number of nonlinear units. We observe that there exists a
simple bijective mapping that maps a path in DenseNets to a
path with the same length in ResNets. It is worth noting that
no path in a densely connected structure can take consecutive skip connections. When a feature map is concatenated
with others after going through a skip connection, it must
be forwarded into the immediately following basic nonlinear unit instead of taking another skip connection again as
is highlighted by the bold red arrows in Fig. 2. Therefore,
we can map a path in a residual connection structure to a
unique path in a densely connected structure with the same
length by mapping every sequence of consecutive identity

3. Aggregation View
To motivate this section, we make the following definitions:
An aggregation function A(·, ·, ·, ..., ) is a function defined recursively by combining the input tensors in the
parenthesis in the order they appear to compute its output,
i.e., A(T1 , T2 , ..., Tn ) = f (T1 , A(T2 , ..., Tn )) for some operation f . Examples of aggregation functions include summation and concatenation.
We define a nonlinear unit to be a nonlinear transformation consisting of a consecutive series of common operations in deep neural networks that satisfies the following
properties:
4323

mappings to a skip connection with the same beginning and
end and vice versa. An example of such a mapping is shown
in Fig. 3.

stands for a function that sums two input tensors and only
takes the positive parts. The recursion formula for residual
connection can be simplified as the following equation:
x`+1 = F` (x` ) ⊕ x`

(3)

The same recursion formula can be applied to x` as well
and we get
x`+1 = F` (x` ) ⊕ x`
= F` (x` ) ⊕ (F`−1 (x`−1 ) ⊕ x`−1 )

Figure 2: A path cannot take consecutive skip connections.

= F` (x` ) ⊕ F`−1 (x`−1 ) ⊕ ... ⊕ F1 (x1 )

(4)

= y`−1 ⊕ y`−2 ⊕ ... ⊕ y1 .
Similar to Eq. 2, we also require consecutive ⊕ operations
to be evaluated from right to left.
By replacing the ⊕ with concatenation ⊗, we get exactly
Eq. 2, the basic formula of DenseNet structure. This result
draws a second similarity between DenseNets and ResNets:
The output of each basic unit is the result of a nonlinear
transformation applied to the aggregation of all the features
from the previous units. Using the notation of aggregation
function, the recursion formula for DenseNets and ResNets
can both be written as:
Figure 3: Mapping from a path in ResNets to DenseNets.
Bold green arrows indicate the connections taken by the
path. ⊕ stands for the composite of summation and ReLU
and ⊗ stands for concatenation.

y` = F` (x` )
x`+1 = A(y` , y`−1 , . . . , y1 )

(5)

or simply
y`+1 = F`+1 (A(F` , F`−1 , . . . , F1 ))

An alternative visualization of DenseNets structure with
the same path is presented in Fig. 4 to help better illustrate
the mapping. In this visualization, the output of a basic
unit is directly concatenated with its input via a shortcut
connection and the concatenated tensor becomes the input
to the next unit.

(6)

We call this way of computing the feature map of one layer
by aggregating the feature maps of previous layers the aggregation view. Since both DenseNets and ResNets aggregate all the features from the previous basic units, we call
both of them dense aggregation structures. However, the
most significant difference under the aggregation view is
the way features are aggregated. ResNets aggregate the features through summation while DenseNets aggregate them
via concatenation. Our analysis of DenseNets’ excellent
performance is largely based on this insight.

Figure 4: An alternative visualization of DenseNets.

3.2. ResNets & DenseNets Under Aggregation View

3.3. Dense Feature Aggregations

While DenseNets have many more shortcut connections
and parameters than ResNets, their numbers and distributions of end-to-end paths are strikingly similar. This similarity calls for both a revisit of ResNets and other explanations for the effectiveness of DenseNets. Inspired by the
work of deep layer aggregation [21], we propose an aggregation view that encompasses both ResNets and DenseNets
in the same framework and reveals an essential difference
that can explain the superior performance of ResNets over
DenseNets.
Recall the definition of residual connection in Eq. 1. We
define a new operator ⊕ to simplify the notation. ⊕ sign

The success of training networks with more than 100 layers using DenseNet and ResNet structures can be partially
attributed to dense feature aggregation. Since dense feature aggregation can create lots of shortcut connections, its
strengths have been studied in many past works [7, 6, 18,
19]. In summary, the dense feature aggregation allows each
nonlinear layer to be directly supervised by the output layer
during training; it relieves the gradient vanishing and exploding problems. From the forward propagation perspective, dense feature aggregation makes it possible for the feature of each layer to be accessed by latter layers without any
intermediate nonlinear transformation units.
4324

(1) ResNet and DenseNet

(2) SparseNet

Figure 5: ResNet, DenseNet and SparseNet under aggregation view. The key difference between ResNet and DenseNet is
the aggregation method. SparseNet follows the aggregation method of DenseNet, but only layers with exponential offsets are
gathered.
However, dense feature aggregation also comes with
many drawbacks. These drawbacks appear in different
forms in the ResNet-styled aggregation by summation and
the DenseNet-styled aggregation by concatenation. Theoretically speaking, it is impossible to disentangle each one
of a set of features after taking their sum. The aggregation by recursive summation and ReLU in ResNets prevents
individual features from previous layers to be directly accessed by subsequent layers in their original forms. Corruption and washing-out of the information carried by feature
maps accumulate as the number of features maps summed
grows linearly. The way of combining features is also hardcoded in the design of ResNets, giving the model seemingly
little flexibility to learn to combine the features by itself.

tion, concatenation, than the summation used by ResNets.
However, DenseNets can easily become overly dense from
the aggregation view perspective.

In contrast, the aggregation style of DenseNets combines
the features through direct concatenation which preserves
the original form of the features from previous layers no
matter how far they are. The model also has the potential to learn optimal linear combinations of the input feature
maps. The better performance and parameter efficiency of
DenseNets than ReseNets can be explained by the advantage of concatenation over summation. However, aggregation by concatenation has its own problems: The numbers
of skip connections and required parameters grows at the
asymptotic rate of O(N 2 ) where N is the number of basic
units in a dense aggregation structure. This asymptotically
quadratic growth rate brings about higher risk of overfitting
and practical problems in settings with limited computing
resources when we design deeper networks. Experiments
also show that it is hard for the model to make full use of all
the parameters and dense skip connections. In the original
work of DenseNets [7], a large fraction of the skip connections have average absolute weights of convolution filters
close to zero. This result implies that the feature maps aggregated by these skip connections are not fully exploited.

where k is the largest non-negative integer such that 2k < `.
We name this aggregation structure Sparsely Aggregated
Convolutional Networks, or SparseNets. It is worth noting
that the number of features aggregated by the `th nonlinear unit is asymptotically O(log `). Considering the slow
growth of logarithm functions, this number can be bounded
by a relatively small constant in most modern deep neural
network models.

4. SparseNets
Based on the analysis of dense aggregation structures in
ResNets and DenseNets, we propose simple feature aggregation schemes that retain the strengths of concatenation but
avoid its drawbacks by gathering previous layers’ features
sparsely: Each basic unit only takes features from layers
that are behind it by offsets of powers of two.
y` = Fl (F`−1 ⊗ F`−2 ⊗ F`−4 ⊗ F`−8 ⊗ ... ⊗ F`−2k ) (7)

4.1. Network Details
Non-linear Unit in SparseNets is the composition of
batch normalization, ReLU and a convolution layer with
kernel size 3 and stride 1.
Transition Unit. Feature downsampling is an essential component of convolutional neural networks to learn
hierarchical representations from data. Downsampling is
usually implemented by strided convolution or pooling. In
SparseNet, we adopt the same average pooling method as
DenseNets [7]. Since feature aggregation or skip connections cannot be made across downsampling operations, they
divide a convolutional neural network into several large
blocks of basic nonlinear units and feature aggregation

In summary, DenseNet uses a better aggregation func4325

structures, namely ResNets, DenseNets and SparseNets, are
only applied within each block. We call the sequence of operations, including downsampling, between two blocks of
nonlinear units a transition unit. Similar to DenseNets, we
divide our SparseNets models into three blocks with equal
depth.
Output channel number is the number of channels of
the feature maps produced by each nonlinear transforming
unit. A fixed and relatively small number, e.g. 12 or 24,
of output channels is used across all blocks of DensNets;
the number of output feature map channels doubles across
each block of ResNets, e.g. (32,64,128). Since the number of feature maps aggregated by each layer in SparseNets
is relatively small, it can adopt the output channel number
schemes of both ResNets and DenseNets with controllable
numbers of input channels to each layer in a deep model.
The number of output channels is denoted by k.
Bottleneck structure and compressing transition are
structures adopted by ResNets and DenseNets to reduce the
number of parameters and improve computation efficiency.
DenseNets and ResNets both introduce a similar bottleneck
structure. The bottleneck structures in both of them first
use a 1 × 1 filter to reduce the number of channels of the
input tensor before convolving it with filters of a larger size.
To make the size of output compatible with the input for
element-wise summation, ResNets applies another 1 × 1
filter to increase the number of feature map channels. To
make the model even more compact, DenseNets also explicitly compress the number of feature map channels in
each transition unit by introducing a compression factor of
θ. Suppose the number of output channels from the previous DenseNets block is m. The number of output channels of the the transition unit is bθmc. While we also conducted experiments on SparseNets with the same bottleneck
structures and compressing transition units as DenseNets,
the better parameter efficiency of our model is mainly due
to the replacement of overly dense aggregation structures
with sparse aggregation. SparseNets models with bottleneck and compression structures with θ = 0.5 are denoted
by SparseN ets − BC in Tab. 1.

Intel i7-6700k CPU and NVIDIA GTX 1080 Ti GPUs.
In DenseNet’s official implementation, they achieve
dense connections by simply concatenating the outputs of
the current layer with the input. Our implementation of
DenseNets and SparseNets aggregate features by directly
concatenating them from the outputs of layers.
CIFAR Dataset. A convolution with the number of filters equal to the number of output channels is performed
on the input images. Experiments on the CIFAR dataset set
the first convolution kernel size to be 3 × 3, and has three
main blocks of the same number of layers after first convolution layer. There are two types of convolutional kernels
in each main blocks: 3 × 3 and 1 × 1. The 3 × 3 convolution layers have one pixel padded on both sides to keep
the size of the feature maps fixed. The 1 × 1 convolutional
layers are used before 2 × 2 average pooling in a transition unit between two contiguous main blocks. The size of
the feature maps are 32 × 32, 16 × 16 and 8 × 8 respectively. To make a fair comparison, all experiment settings
on plain DenseNet and DenseNet-BC structures in the original work [7] are preserved. Further, we also evaluate models with other configurations including {L = 40, k = 36},
{L = 100, k = 36} and {k = 16, 32, 64} to compare
DenseNets and SparseNets where L is the depth of the network and k is the number of output channels.

6. Experiments
We demonstrate the effectiveness of SparseNets through
image classification task on the CIFAR datasets [10] and
compare it with the current state-of-the-art models, namely
ResNets [6], DenseNets [7]. The image classification accuracy appears to saturate on some large SparseNets and
DenseNets models when we continue increasing model
depth or output channel number. However, we think the
saturation is not caused by the limit of model capacity but
the dataset and task complexity based on our experimental
results.

6.1. Datasets
CIFAR: Both the CIFAR-10 and CIFAR-100 datasets
[10] have 50,000 training images and 10,000 testing images
with size of 32 × 32 pixels. We hold out 5,000 images from
the training set for validation. CIFAR-10 (C10) and CIFAR100 (C100) have 10 and 100 classes respectively. The mark
+ indicates standard data augmentations including mirroring and shifting which are used in [7]. For preprocessing,
we normalize the data by the channel means and standard
deviation. We train the model for 300 epochs with batch
size 64. All SparseNets models are trained on a single GPU
without any memory optimization. The training of most
DenseNets models are also run on a single GPU, except the
ones exceeding single GPU memory limit, e.g. DenseNetBC-250-24 on 2 GPUs, DenseNet-BC-190-40 on 2 GPUs.

5. Implementation Details
We implemented our models in the PyTorch framework
[13]. SGD is used as the optimizer with mini-batch 64 on
single GPU (unless GPU memory limit is exceeded). We
set weight decay to 0.0001 with Nesterov momentum 0.9
and train the model for 300 epochs. Following the schedule
from the Torch implementation of ResNet [4]1 , the learning rate starts from 0.1 and is divided by 10 two times at
epoch 150 and 300. We adopt the standard weight initialization [5]. All experiments are trained on a server with an
1 https://github.com/facebook/fb.resnet.torch

4326

Method
ResNet [6]
ResNet(pre-activation)[6]
ResNet(pre-activation)[6]
Wide ResNet [22]
Fractal [12]
DenseNet (k=12)[7]
DenseNet (k=12)[7]
DenseNet (k=24)[7]
DenseNet-BC (k=12)[7]
DenseNet-BC (k=24)[7]
DenseNet-BC (k=40)[7]
SparseNet (k=24)
SparseNet (k=24)
SparseNet (k=36)
SparseNet (k=16, 32, 64)
SparseNet (k=32, 64, 128)
SparseNet-BC (k=24)
SparseNet-BC (k=36)
SparseNet-BC (k=16, 32, 64)
SparseNet-BC (k=32, 64, 128)

Depth
110
164
1001
16
21
40
100
100
100
250
190
40
100
100
100
100
100
100
100
100

Params
1.7M
1.7M
10.2M
11.0M
38.6M
1.1M
7.2M
28.28M
0.8M
15.3M
25.6M
0.76M
2.52M
5.65M
7.22M
27.72M
1.46M
3.26M
4.38M
16.72M

C10+
6.61
5.46
4.62
4.81
5.52
5.39*
4.28*
4.04*
4.68*
3.65
3.75*
5.13
4.64
4.34
4.11
3.88
4.03
3.91
-

C100+
24.33
22.71
22.07
23.30
24.79*
20.97*
19.61*
22.62*
17.6
17.53*
24.65
22.41
20.50
19.49
18.80
22.12
20.31
19.71
17.71

Table 1: Experiments and comparison of SparseNets, DenseNets, ResNets and their variants. Results marked ∗ are from our
implementation.
quires 1.1 million parameters to achieve error rate 24.79%
on CIFAR100+, while SparseNet can achieve a similar error rate 24.65% in setting L = 40, K = 24 with only
0.76 million parameters. On the same dataset, the smallest number of parameters required by DenseNets to get
an error rate smaller than 20% is more than 15 million
but SparseNet only requires less than 7 million. The second is when having comparable numbers of parameters,
SparseNet have higher accuracy. When both networks have
less than 15 million parameters, SparseNet always outperforms DenseNet with similar number of parameters. For
example, DenseNet-100-12 has 7.2M parameters with error
rate 20.97, while SparseNet-100-{16, 32, 64} has 6.78M
parameters with error rate 19.71. Counter examples do
exist, such as the comparison between SparseNet-BC-100{32, 64, 128} and DenseNet-BC-250-24. The latter model
performs slightly better than the previous one with less parameters (17.71 vs 17.53). We argue this is an example of
performance saturation considering DenseNet-BC-190-40
only has slightly higher accuracy than DenseNet-BC-25024 with much more parameters (25.6 million vs 15.3 million) and it should result from the nature of the task of image classification on the CIFAR-100 dataset (17.53 vs 17.6).
These two general trends are summarized in the parameterperformance plot in Fig. 6.

All the models are evaluated on the test sets of CIFAR-10
or CIFAR-100.

6.2. Results on CIFAR
Accuracy & Parameter Efficiency. We observe that
the performance gain of DenseNets is mainly due to deeper
structure and especially more output channels. The densely
connected structure brings efficient reuse of the features but
also limits the network to go deeper as the large paramater
count heavily taxes computational resources. As shown in
Tab. 1, the number of parameters increases by 6 times when
depth grows from L = 40 to L = 100, 3 times when the
number of output channels doubles. We can imagine that
if we simply go deeper with the DenseNet structure, the
number parameters can easily exceed the memory limit of
most GPU servers and brings risk of overfitting. While the
ResNets structure does not have the problem of parameter
explosion, it might suffer from greater information loss. In
contrast, SparseNets has neither issue. The original feature maps are kept after aggregation by concatenation while
the parameter number growth rate is logarithmic. We can
safely build deeper models and increase the number of output channels.
The experimental results on CIFAR [10] are reported in
Tab. 1. There are two obvious general trends in the results. First, SparseNet usually requires fewer parameters
than DenseNet when they have close performance. Most
notably, DenseNet setting L = 40, k = {16, 32, 64} re-

7. Discussion and Conclusions
4327

24

SparseNet-40-24

23

SparseNet-40-36
DenseNet-40-24

22

20

DenseNet-100-24

SparseNet-100-{16,32,64}
0

5

10

15

#parameters x107

20

25

2

2

4

4

6

6

6

8

8

8

10

10

10

2

4

6

8

10

Dense Block 1

0

0

Target Layer( )

2

4

6

8

10

2

2

4

4

4

6

6

6

8

8

8

10

10

10

4

6

8

10

Target Layer( )

0

2

4

6

8

0.8

10

Target Layer( )

2

4

6

8

10

Target Layer( )
Dense Block 3

0

2

2

1.0

0

Target Layer( )
Dense Block 2

0

Sparse Block 3

0

4

0

Figure 6: Comparison between DenseNets and SparseNets
(without bottleneck and compression) on top 1% error and
number of parameters with different configurations.

Sparse Block 2

0

2

0

DenseNet-100-12
SparseNet-100-36

21

Sparse Block 1

0

Source Layer( )

DenseNet-40-12

Source Layer( )

Validation Error (%)

25

0.6

0.4

0.2

0.0
0

2

4

6

8

10

Target Layer( )

Figure 7: The average absolute filter weights of convolutional layer in trained DenseNet and SparseNet. The
color pixel (s, `) indicates the weights average of connection from layer s to ` within a block. The first row encodes
the weights attached to the first input layer of Dense/Sparse
Block.

7.1. Feature Reuse and Parameter Redundancy
The original DenseNets work [7] conducts a simple
experiment to investigate how well a trained network
reuses features across layers. In short, for layer ` in each
densely connected block, they compute the average absolute
weights of the part of the filters that convolves with each
previous layer’s feature. The averaged absolute weights are
rescaled between 0 and 1 for each layer `. The k t h normalized value implies the relative dependency of the feature of
layer ` on the feature of layer k compared to other layers.
The experiments are performed on a DenseNet consisting
of 3 blocks with l = 40 and k = 12. We perform a similar
experiment on a SparseNets model with the same configuration. The results are plotted as heat maps in Fig. 7. We also
include the heat maps of the experiment on DenseNets [7]
for comparison.
In this heap map, a red pixel at location (`, k) indicates
layer ` makes heavy use of the features of layer k and a blue
pixel indicate relatively little usage. A white pixel indicates
there is no direct connection between layer ` and layer k.
We made the following observations from the heat maps:

7.2. Short Path Revisited
The work of Veit et al. [18] reveals the vitality of short
end-to-end paths in ResNets for backpropagated gradient
flows to train the model. DenseNets [7] casts light on
the importance of direct skip connections between layers
for information flow and feature reuse. Compared with
DenseNets, there is no direct access or path of 0 length between some pairs of layers in SparseNets. However, the
shortest path that connects y` and x` +n is of length at most
dlog2 ne. Intuitively, the longer the path between two layers
is, the more likely information is corrupted and washed out
when propagated along this path. In most network depth
settings, dlog2 ne is bounded above by a small number.
Therefore, even a layer in a SparseNet does not have direct
access to a previous feature map, the shortest path between
two layers could still propagate these features without too
much cost. Similar arguments can be applied to supervision
by backward gradients.
Veit et al. [18] also proposes the concept of effective depth, the range of end-to-end short path depths that
contributes most of the gradients during backpropagation.
However, many end-to-end paths in ResNets still have
depths smaller than this effective range. While whether
network training can benefit from directly supervising low
level features with the final loss remains an open problem,
SparseNets significantly reduce the number of overly short
paths with length below the proposed effective depth range.
In fact, in most cases of SparseNets, there’s no short path
that can directly backpropagate the gradients from the output to the input without going through any non-linear transformation. Further explorations on these related problems
are left as future work.

• Most of the non-white pixels in the heat map of
SparseNets are close to red, indicating that each layer
takes full advantage of all the features it directly aggregates. It also indicates almost all the parameters are fully exploited, leaving little parameter redundancy. This result is not surprising considering the
high parameter-performance efficiency of our model.
• In general, the pixel value at position (`, k) in
DenseNets decreases as the offset between ` and k gets
larger. However, such a decaying trend does not appear
in the heat map of SparseNets, implying that layers in
SpareNets have better ability to extract useful features
from preceding layers of long distance.

4328

7.3. Conclusion

[14] R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway
networks. arXiv preprint arXiv:1505.00387, 2015.
[15] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting
unreasonable effectiveness of data in deep learning era. arXiv
preprint arXiv:1707.02968, 1, 2017.
[16] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi.
Inception-v4, inception-resnet and the impact of residual
connections on learning. In AAAI, pages 4278–4284, 2017.
[17] S. Targ, D. Almeida, and K. Lyman.
Resnet in
resnet: generalizing residual architectures. arXiv preprint
arXiv:1603.08029, 2016.
[18] A. Veit, M. J. Wilber, and S. Belongie. Residual networks
behave like ensembles of relatively shallow networks. In
Advances in Neural Information Processing Systems, pages
550–558, 2016.
[19] Z. Wu, C. Shen, and A. v. d. Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. arXiv preprint
arXiv:1611.10080, 2016.
[20] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated
residual transformations for deep neural networks. arXiv
preprint arXiv:1611.05431, 2016.
[21] F. Yu, D. Wang, and T. Darrell. Deep layer aggregation.
arXiv preprint arXiv:1707.06484, 2017.
[22] S. Zagoruyko and N. Komodakis. Wide residual networks.
CoRR, abs/1605.07146, 2016.

We propose the aggregation view that relates the features
of each layer in a deep neural network to the aggregated features of previous layers. The aggregation view provides us
with a uniform framework to scrutinize the superb performance of ResNets and especially DenseNets. Based on our
analysis, we proposed a new and simple feature aggregation scheme called Sparsely Aggregated Convolutional Networks: Each layer only gathers features from layers that are
powers of two layers from it. Our model can outperform the
current state-of-the-art networks with fewer parameters.

References
[1] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term
dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157–166, 1994.
[2] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei. Imagenet: A large-scale hierarchical image database.
In Computer Vision and Pattern Recognition, 2009. CVPR
2009. IEEE Conference on, pages 248–255. IEEE, 2009.
[3] X. Glorot and Y. Bengio. Understanding the difficulty of
training deep feedforward neural networks. In Proceedings
of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010.
[4] S. Gross and M. Wilber. Training and investigating residual
nets. 2016.
[5] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into
rectifiers: Surpassing human-level performance on imagenet
classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
[6] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
770–778, 2016.
[7] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten.
Densely connected convolutional networks. arXiv preprint
arXiv:1608.06993, 2016.
[8] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger.
Deep networks with stochastic depth. In European Conference on Computer Vision, pages 646–661. Springer, 2016.
[9] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift. In
International Conference on Machine Learning, pages 448–
456, 2015.
[10] A. Krizhevsky and G. Hinton. Learning multiple layers of
features from tiny images. 2009.
[11] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Advances in neural information processing systems, pages
1097–1105, 2012.
[12] G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. CoRR,
abs/1605.07648, 2016.
[13] A. Paszke, S. Chintala, R. Collobert, K. Kavukcuoglu,
C. Farabet, S. Bengio, I. Melvin, J. Weston, and J. Mariethoz. Pytorch: Tensors and dynamic neural networks in
python with strong gpu acceleration, may 2017.

4329

