Is each layer non-trivial in CNN?

Many convolutional neural network (CNN) models have achieved great success in many fields. The networks get deeper and deeper. However, is each layer non-trivial in networks? To answer these questions, we propose to replace the convolution kernels with zeros. We compare these results with baseline and show that we can reach similar or even same performances. Although convolution kernels are the cores of networks,we demonstrate that some are trivial and that these layers are regular.


Introduction
The structures of neural networks is get more and more complex. There ara tow basis form: short-connection and noconnection. Long-connection: UNet (Ronneberger, Fischer, and Brox 2015), SegNet (Badrinarayanan, Kendall, and Cipolla 2017). Short-connection:ResNet (He et al. 2015). No-connection: VGG (Simonyan and Zisserman 2014). Long-connection can be seen as a special no-connection in local area. First, we define the non-triviality by the changes of result after convolution kernels replacing by 0. It is obvious that each layer in no-connection form is important. However, we believe many layers in ResNet are trivial.
The main contributions of this paper can be summarized as :(1). We demonstrate some layers are trivial in ResNet.
(2). We demonstrate the feature decomposition layers are non-trivial, while the rest are trivial.
Analysis the convolution kernels of ResNet replaced by 0 ResNet residual unit can be formulated as: x l+1 =σ(BN (x l * w 1 * 1 l ) + BN (σ(BN (x l * w l )) * w l )) Replacing one of the convolution kernels in residual unit with 0 can be written as: Copyright c 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
The input and output feature maps of the lth residual unit. x l+1 , x l+1 , x l+1 : The output feature maps of the lth residual unit. w l , w l : The first convolution kernel and the second convolution kernel of the lth residual unit. * : Convolution operation. BN : Batch normalization operation. β , β , β : The bias in BN layers.

Experiment
We choose ResNet34 and PSPNet-ResNet34 (Zhao et al. 2016) conduct classification task and image segmentation task on Caifar-10 (Krizhevsky 2009) and T1 (Fahmy et al. 2019). The baseline are 84% and 87%. We conduct 3 groups of experiments. First, we replace each layer's convolution kernel with 0. Second, we replace all convolution kernels with 0 in the same layer block (A continuous layer with the same feature maps is a layer block) except for the featur e decomposition layers and adjacent layers. Third, we replace feature decomposition layers of short-connection with 0.

Results
The classification results of Cifar-10: Figure 1 and Table  1,2.The segmentation results of T1: Figure 2 and Table 3,4.       Table 3 also prove our conjecture. Table 2 and Table 4 demonstrate the feature decomposition layers of shortconnection are non-trivial.

Discussion
We argues that ResNet is a continuous process of feature decomposition and information storage. ResNet shows different changes in non-trivialness at the front and back of the network for different tasks. Because the classification task needs to learn enough information about the global abstract feature, if enough information is learned in the front, the back is no longer non-trivial. Segmentation requires information for each pixel, so the back layers are non-trivial.

Conclusion
There are redundant parameters in some networks, not all layers of the network are non-trivial, or some layers may not be needed when the network parameters have learned enough information.The feature decomposition layers and identity mapping are important. The feature decomposition layers are responsible for the feature decomposition, the identity mapping is responsible for the information storage, and the residual layers are responsible for the adjustment of the feature to make it fit the final target.According to the above conclusion, we can eliminate unnecessary layers in the resnet and improve the training efficiency on the premise of ensuring the performance.