Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
Zheng Qin Zhaoning Zhang Dongsheng Li Yiming Zhang Yuxing Peng




#1#1



networks methods para-meters appears op-tical net-works semi-conduc-tor store stor-age snap-shot space reads among Ursa exits scal-able uses sys-tems IOPS dif-ferent GFS HDFS FS gen-erally SCSI NFS HBase SSD slightly stored two anony-mized speak-ing lineariza-bility server elim-inate states pro-cesses signifi-cantly sin-gle OCFS blocks par-allelism parallel-ism returns pre-senting present-ing stand-ard stores IOPS servers appends scala-bility pri-mary mode respec-tively re-spectively sat-isfy UPSs CPUs avail-ability availa-bility availabil-ity clouds usu-ally against re-quests inde-pendently namely storing Meituan evolv-ing since ob-ject tests BS mounted under-loaded consis-tency status jour-nals place-ment statis-tics exclu-sively meta-data mainly collabora-tively collabo-ratively FDS focus nodes remote conven-tional strata single enough daily require small sequen-tial Theore-tically coroutine acce-leration accele-ration effi-cient matrix vectors imple-mentations implemen-tations implementa-tions larger Mobile-Nets GPUs com-puted cuDNN hyper-parameters hyperpara-meters Con-sequently Conse-quently app-lications appli-cations applica-tions adopted GPU accele-rating acce-lerating Tensor-Flow results frame-works convo-lutions con-volutions convolu-tions convo-lution con-volution convolu-tion threads sig-nificant signi-ficant signifi-cant ii iii GEMM aca-demic acade-mic its weights algo-rithms uti-lize also rela-tively redun-dancy imple-mentation implemen-tation implementa-tion Mobile-Net multi-plier fewer res-pectively respec-tively kernel table uti-lizes every in-creases diag-onalwise Multiplications

=1000






Diagonalwise Refactorization: An Efficient
 Training Method for Depthwise Convolutions

    
Zheng Qin, Zhaoning Zhang^*^*Corresponding author., Dongsheng Li, Yiming Zhang, Yuxing Peng
Science and Technology on Parallel and Distributed Laboratory 

National University of Defense Technology

Changsha, China

qinzheng12@nudt.edu.cn; zzningxp@gmail.com; lds1201@163.com; sdiris@gmail.com; pengyuxing@nudt.edu.cn

    
===================================================================================================================================================================================================================================================================================================================================




Depthwise convolutions
provide significant performance benefits
owing to the reduction in both parameters and mult-adds.
However, training depthwise convolution layers with GPUs is slow in current deep learning frameworks
because their implementations cannot fully utilize the GPU capacity.
To address this problem,
in this paper we present an efficient method (called diagonalwise refactorization)
for accelerating the training of depthwise convolution layers.
Our key idea is to
rearrange the weight vectors of a depthwise convolution into a large diagonal weight matrix
so as to convert the depthwise convolution into one single standard convolution,
which is well supported by the cuDNN library that is highly-optimized for GPU computations.
We have implemented our training method in five popular deep learning frameworks.
Evaluation results show that
our proposed method gains 15.4× training speedup on Darknet, 8.4× on Caffe, 5.4× on PyTorch, 3.5× on MXNet, and 1.4× on TensorFlow,
compared to their original implementations of depthwise convolutions.



Acceleration,
convolutional neural network,
depthwise convolution




§ INTRODUCTION


Deep convolutional neural networks (ConvNets) <cit.> have recently become increasingly important for computer vision applications.
However,
standard deep ConvNets suffer from high computational cost
due to their high numbers of parameters and mult-add operations.

MobileNets <cit.> uses depthwise separable convolutions,
which factorize a standard convolution into a depthwise convolution and a pointwise (1 × 1) convolution,
to effectively reduce the numbers of both parameters and mult-add operations.
Xception <cit.> leverages depthwise separable convolutions
to improve its classification performance.
However,
as reported in <cit.> and <cit.>,
depthwise convolutions have a low computation vs. memory access rate,
which means memory access takes more execution time than computation and it is more difficult to implement depthwise convolutions as efficient as computation-intensive layers like standard convolutions.
This makes training depthwise convolution layers with GPUs very slow in current deep learning frameworks such as Caffe <cit.>, PyTorch <cit.>, MXNet <cit.> and TensorFlow <cit.>,
mainly because their implementations of depthwise convolutions cannot fully utilize the GPU capacity.

Caffe, PyTorch and MXNet
implement depthwise convolutions
by performing the standard convolution channel-by-channel.
This method simply launches a CUDA kernel or cuDNN function for each of the input channels,
and applies no inter-channel optimizations such as filter combination.
Consequently,
the number of threads launched for each standard convolution is small
and the utilization of GPU cores is very low.
For example,
although depthwise convolutions have only about 3% of the mult-adds and 1% of the parameters
when training a MobileNet <cit.>,
they spend over 82% of the overall training time  on Caffe which is much higher than any other layer types,
as shown in our evaluation (Table <ref>).

Different from the channel-by-channel method,
TensorFlow
adopts the specialized kernel method
which
implements depthwise convolutions
by designing a specialized CUDA kernel
and computing all the input channels in the single kernel.
This method is more efficient in training depthwise convolution layers
because it exploits the inter-channel parallelism.
However,
the specialized kernel method prevents TensorFlow from leveraging the cuDNN library <cit.>
with the algorithm-level and microarchitecture-level optimizations,
which are vital for high-performance GPU computations.



This paper presents diagonalwise refactorization,
an efficient method for accelerating the training of depthwise convolution layers.
First,
the weight vectors (filters) of the input channels are rearranged into a diagonal matrix
to construct one single large filter.
Then,
the depthwise convolution is computed as a standard convolution with the large filter,
which supports to leverage the cuDNN library to accelerate the computation.
We further adopt a grouping mechanism for convolutions with large numbers of input channels,
where
the channels are divided into several groups
and the diagonalwise refactorization is performed for each group.
By combining all filters into a large one,
our method could exploit the inter-channel parallelism
to utilize the GPU capability more efficiently.
By supporting the cuDNN library,
our method could directly enjoy its algorithm-level and microarchitecture-level optimizations.

We have implemented our method
on five popular frameworks
including Darknet <cit.>, Caffe, PyTorch, MXNet, and TensorFlow.
Evaluation results show that
our method gains 15.4× speedup on Darknet, 8.4× on Caffe, 5.4× on PyTorch, 3.5× on MXNet, and 1.4× on TensorFlow, when training a standard MobileNet,
compared to their original implementations of depthwise convolutions.
We conduct extensive experiments on different MobileNet hyper-parameters
including shallow models, width multiplier and resolution multiplier,
and perform detailed analysis on the layer-by-layer training time.
Code has been made publicly available at <https://github.com/clavichord93>[Code in Caffe, PyTorch and TensorFlow has been made publicly available at <https://github.com/clavichord93/diagonalwise-refactorization-caffe>, <https://github.com/clavichord93/diagonalwise-refactorization-pytorch> and <https://github.com/clavichord93/diagonalwise-refactorization-tensorflow>.].

The contribution of this paper is summarized as follows.

  
  * We propose a novel method (diagonalwise refactorization) that effectively accelerates the training of depthwise convolutions.
  
  * We implement our method on five popular frameworks
  and provide detailed performance comparison and .
  
  * We discuss the extensibility of our method
  and show that it could be adopted in the training of many acceleration techniques
  such as pruning and group convolutions.






§ RELATED WORK


Many techniques have been proposed to compress existing ConvNets.
Network pruning <cit.> accelerates the inference of networks by reducing spatial, connection and channel redundancy.
Parameter quantization <cit.> trains deep ConvNets directly with binary weights and gains significant acceleration.
Tensor decomposition <cit.> adopts a low-rank approximation to original convolution filters to reduce parameters.

Layer factorization
has been proposed to build lightweight networks.
SqueezeNet <cit.> proposes fire modules which mix 1 × 1 and 3 × 3 convolutions and achieves AlexNet-level accuracy with 50 × fewer parameters.
MobileNet <cit.> replaces standard convolutions with depthwise separable convolutions <cit.> and provides competitive accuracy with state-of-the-art networks.
ShuffleNet <cit.> further applies group convolutions in depthwise covolutions with channel shuffle operations.



A depthwise separable convolution is a combination of a depthwise convolution and a pointwise convolution.
As shown in Figure <ref>,
a depthwise convolution filter (kernel) is applied to one input channel with its own set of weights.
For an M-channel input feature map, a depthwise convolution creates an M-channel output feature map.
Depthwise separable convolutions achieve a significant reduction in parameters and mult-add operations,
and thus effectively accelerate the computation of ConvNets.
For example,
MobileNets perform very fast inference on mobile devices <cit.>.
We also achieve real-time single-category detection with a 0.375-MobileNet-416 model on i7 CPU and a 0.375-MobileNet-128 model[Only the first 11 layers of the standard MobileNet is used. 0.375 indicates the width multiplier while 416 and 128 indicate the resolution multiplier.] on iMx6 ARM with NEON acceleration, in the YOLOv2 <cit.> detection framework.



Currently there are two methods (channel-by-channel and specialized kernel)
for implementing depthwise convolutions,
both of which are based on the standard convolutions.

Standard convolutions.
Assume that the numbers of input and output channels are respectively M and N,
the size of the feature map is F × F, and the size of the convolutional kernels is K × K.
A standard convolution is implemented in two steps,
as shown in Figure <ref>.
The first step is an im2col operation,
where the input feature map 𝐗_M × F × F is rearranged
into an intermediate matrix 𝐂_(M · K · K) × (F · F),
and every block to be convolved in 𝐗 is rearranged into a column of 𝐂.
The second step is a matrix multiplication between the convolution weight matrix 𝐖_N × (M · K · K) and the intermediate matrix 𝐂.
The result is a matrix 𝐙_N × (F · F),
each row of which represents a single channel of the output feature map.



Channel-by-channel method.
As shown in Figure <ref>,
this method simply performs the standard convolution for each of the input channels.
M intermediate matrices 𝐂^(i)_(K · K) × (F · F) are first generated using an im2col operation.
Then M multiplications between the weight vectors 𝐰^(i)_K × K and the intermediate matrices 𝐂^(i) are performed.
Finally, tiling the M result vectors 𝐳^(i)_F × F generates the result matrix 𝐙.
The channel-by-channel method is adopted in Caffe, PyTorch and MXNet.
Caffe only supports to leverage the general matrix multiply (GEMM) operations,
but not cuDNN,
for channel-by-channel convolutions,
while PyTorch and MXNet support to use cuDNN
which offers a remarkable performance improvement compared to Caffe (as shown in Table <ref>).

Specialized kernel method.
This method does not explicitly generate the intermediate matrix 𝐂
and designs its own specialized CUDA kernel for implementing depthwise convolutions
instead of performing standard convolutions channel-by-channel with GEMM or cuDNN.
In a CUDA thread,
a K × K block from the input feature map 𝐗 is convolved with the weights 𝐰^(i) of the same channel to compute one pixel in the output feature map 𝐙.
The specialized kernel method is adopted in TensorFlow,
which performs its own hand optimizations for GPU's shared memory and caches.



§ DESIGN


In this section,
we first introduce the diagonalwise refactorization method,
and then describe the grouping mechanism for large numbers of input channels.
At last we analyze the advantage of our method over previous proposals.



 §.§ Diagonalwise Refactorization


Consider a depthwise convolution with M input channels.
In normal depthwise convolutions, a single filter is a vector 𝐰^(i) of length K × K.
The convolution operation is the multiplication of a weight vector 𝐰^(i) and the intermediate matrix 𝐂^(i) of the same input channel.
The depthwise convolution is composed of M vector-matrix multiplications.
In diagonalwise refactorization, we convert a depthwise convolution into a standard convolution.
The M weight vectors 𝐰^(i) are rearranged into a large weight matrix 𝐖_M × (M · K · K) on the diagonal positions and all other positions are set to 0.
The im2col matrices 𝐂^(i) are tiled from top to bottom and form a large intermediate matrix 𝐂_(M · K · K) × (F · F), which is the same as in a standard convolution.
Figure <ref> shows how we rearrange the depthwise weight vectors into a large weight matrix.
During backward propagation,
the gradients from the top layer are passed only to the diagonal weights and all other positions remain 0.

This conversion can be expressed as an M-channel standard convolution preceded by a multiplication of the weight matrix 𝐖 and a constant mask matrix 𝐀_M × (M · K · K) where


    𝐖=[
      [ 𝐰^(1)                  ;       𝐰^(2)            ;                 ⋱      ;                   𝐰^(M);       ]],



    𝐀=[
      [ 1_1 × (K · K)                                          ;               1_1 × (K · K)                            ;                                         ⋱              ;                                           1_1 × (K · K);               ]].


The depthwise convolution could be written as

    𝐖̂    = 𝐖⊙𝐀
    𝐙    = 𝐖̂⊗𝐗


where 𝐗 is the input feature map, 𝐙 is the output feature map, ⊙ represents elementwise multiplication and ⊗ represents convolution.
With the mask matrix 𝐀,
redundant weights are filtered out and only depthwise weights are used for convolution.
During backward propagation, the gradients of the weight matrix are

    ∂𝐙/∂𝐖 = ∂𝐙/∂𝐖̂·∂𝐖̂/∂𝐖 = ∂𝐙/∂𝐖̂⊙𝐀,


where redundant gradients are also filtered out and only depthwise weights are activated.





 §.§ Grouping Mechanism



Compared to previous methods for implementing depthwise convolutions,
our method introduces extra computational cost due to the refactorization of weight vectors,
making our method inefficient when the number of input channels is very large.
We propose a grouping mechanism to address this problem,
where
depthwise convolutions are divided into diagonalwise groups and the diagonalwise refactorization is performed for each group.

For a depthwise convolution with M input channels,
the grouping mechanism has the following three steps.
First,
we divide the input channels into G groups, each of which contains M/G channels.
Second,
every group of weight vectors are refactorized into a diagonalwise matrix for that group.
Third,
each group is computed as a standard convolution which supports to leverage the cuDNN library.
By this means
an M-channel depthwise convolution is converted into G standard convolutions each having M/G input channels,
instead of one large standard convolution having M input channels.



 §.§ Analysis


Compared to our diagonalwise refactorization method,
the channel-by-channel method (adopted in Caffe, PyTorch and MXNet)
launches smaller CUDA kernels with fewer threads,
because the workload of a single input channel is small.
The one-channel-at-a-time computation leads to low utilization of GPU resources.
In contrast,
our method rearranges the input channels into diagonalwise groups,
and the computation within a group could be done by a single CUDA kernel or cuDNN function.
This enables the computation of input channels in the same group to be executed in parallel
and thus more GPU threads could be launched.
Consequently,
our method obtains higher computation parallelism and has a higher utilization of GPU resources.

The specialized kernel method (adopted in TensorFlow) launches one specialized CUDA kernel for all the input channels of a depthwise convolution layer,
and achieves considerable improvement in GPU resource utilization compared to the channel-by-channel method.
However, it directly computes the convolutions
and cannot leverage the algorithm-/microarchitecture-level optimizations of the cuDNN library.
In contrast,
our method convert a depthwise convolution into a standard convolution
which supports to leverage the cuDNN library to compute convolutions.
At the algorithm level,
cuDNN provides fast algorithms (such as Fast Fourier Transform and Winograd Transform) for standard convolutions.
At the microarchitecture level,
cuDNN uses kernels specifically optimized for NVIDIA GPUs with full support of shared memory and caches.
Consequently,
our method is more efficient in leveraging the GPU capacity
compared to the specialized kernel method of TensorFlow.





§ EXPERIMENTS


In this section, we introduce the experimental results
when adopting our diagonalwise refactorization method
in training MobileNets <cit.> on state-of-the-art frameworks.
The results show that our method gains significant acceleration compared to the original implementations in these frameworks.
We further investigate the influence of different grouping strategies and network architectures on our method.
We also evaluate the memory efficiency of our method.
All experiments are conducted on an NVIDIA GTX 1080Ti GPU,
and each result is an average of 1000 batches (with batch size of 64) in the training procedure.



 §.§ Acceleration on MobileNets


We have implemented our method on five popular deep learning frameworks
including Darknet, Caffe, PyTorch, MXNet and TensorFlow.
The results are shown in Table <ref>,
where “*” indicates the original implementation in the corresponding framework
(and all other cells in the table are implemented by us).

To separate the effect of our design,
we evaluate three versions of our implementation.
(i)  represents our implementation that does not utilize the cuDNN library;
(ii)  represents the implementation that utilizes cuDNN without the grouping mechanism;
and (iii)  represents the full implementation that utilizes cuDNN with grouping.
We also implement and evaluate the training performance of four typical methods for depthwise convolutions on Caffe and Darknet,
including three channel-by-channel (C-by-C) methods
(,  and )
as well as the specialized kernel method,
where
 represents the channel-by-channel implementation using only GEMM API,
 represents all GEMM operations are pipelined with CUDA streams to gain acceleration,
and  represents the channel-by-channel implementation using the cuDNN library.

First,
the specialized kernel methods provide considerably higher efficiency than the channel-by-channel methods,
but have slightly worse or similar performance compared to the standard convolutions.
The  implementation of our method performs better than all channel-by-channel methods
without using cuDNN or the grouping mechanism,
but worse than the specialized kernel methods.
When cuDNN is utilized (),
our method offers better training performance than most specialized kernel methods except on TensorFlow.
Finally, with a carefully selected grouping strategy
(which will be discussed later),
our method () surpasses all other methods.



Second,
our method achieves 15.4× speedup on Darknet, 8.4× on Caffe, 5.4× on PyTorch and 3.5× on MXNet compared to their channel-by-channel methods,
and achieves 1.7× speedup on Darknet and 1.4× speedup on Caffe and TensorFlow
over their specialized kernel methods.
The overall speedup could be decomposed to different parts of our method.
On Caffe,
for example,
 offers 3.84 × speedup over the original ,
while  and  incrementally offer 1.72 × and 1.27 × speedups,
resulting in an overall speedup of 8.4 (≈ 3.84 × 1.72 × 1.27) times.
The ablation analysis shows that in our method the refactorization contributes the most in the speedup.
Note that the implementations of our method on PyTorch, MXNet and TensorFlow use Python API provided by the frameworks,
meaning that they do not benefit from operator fusion or simplification
and thus could be further accelerated when being implemented in C++.



 §.§ Layer-by-layer Experiments


To make a detailed comparison of these methods, we further conduct layer-by-layer experiments evaluating the time of the three different training methods on Caffe.

From Table <ref>,
the channel-by-channel method () has very poor performance in the last few layers.
These layers all have wide-and-small feature maps
and have at least 256 input channels.
None of them has feature maps larger than 28 × 28.
For ,
only a small number of threads are launched for small feature maps
and the utilization of GPU resources is low.
For instance,
the last depthwise convolution layer (No. 26) of 
has only 1/8 threads compared to the first row of layer (No. 2).
This amplifies the performance gap between the channel-by-channel method and other methods.

During forward propagation,
 outperforms all other methods in the total training time.
 is better than  in the first layers,
but 
surpasses  in the last few layers.
This is because
small feature maps usually lead to more efficient usage of shared memory and higher cache-hit rate,
which improve the performance of the specialized kernel method.
For ,
it is relatively difficult to find a balance between computational redundancy and GPU utilization
when the number of channels is large,
resulting in slightly lower performance compared to the specialized kernel method
in the last layers.



During backward propagation,
 significantly outperforms other methods.
To understand the poor performance of ,
we also evaluate it without computing the gradients of the weights ().
The result shows that  spends most of its training time in computing the gradients.
This is because there are memory write hazards when computing the gradients,
and the atomic operations slow down the whole training procedure of the specialized kernel method.
This problem is more serious in the first few layers,
where the feature maps are larger and memory write conflicts are more frequent.
Optimized kernels and algorithms in the cuDNN library give significant acceleration to our diagonalwise refactorization method.



Table <ref> shows the training time of each layer type using three methods to train a standard MobileNet.
Compared to  and ,
the ratio of time spent on training depthwise convolution layers
dramatically decreases (to 45.41%)
using the diagonalwise refactorization method ().





 §.§ Grouping Strategies


We propose two strategies for grouping in our method
and investigate their influence on the training performance.

Grouping by the number.
The first strategy is to group by the number of groups.
Assuming that the number of groups is G, every depthwise convolution layer is divided into G diagonalwise groups and each group contains M/G channels.
We compare the efficiency of our method with G from 1 to 16 and the results are demonstrated in Figure <ref>.

Grouping by the size.
The second strategy is to group by the group size.
Assuming that the group size is S channels, there will be M/S groups in each depthwise convolution layer after grouping.
We compare the efficiency of our method with S from 16 to 128.
Figure <ref> shows the results under different configurations.

From Figure <ref>,
the utilization of a group strategy considerably improves the performance of our method.
Compared to the methods without grouping,
grouping by the number offers 1.6 × speedup
while grouping by the size achieves 1.76 × speedup.
Grouping by the size outperforms grouping by the number,
because the first few thin layers does not suffer from the redundant computation.



 §.§ Different MobileNets Hyperparameters


To evaluate the generalization ability of our method,
we conduct extensive experiments on MobileNet variants with different hypermeters.
The variants include shallow MobileNet, thinner MobileNet with width multiplier of 0.75 and 0.5, and low-resolution MobileNet with resolution multiplier of 128,
respectively representing networks with fewer layers, fewer channels and smaller feature maps.
Table <ref> shows the results for these variants
where
 significantly outperforms all other methods,
demonstrating our method is adaptive to different networks built with depthwise convolutions.



Compared to the results on standard MobileNets (Table <ref>),
our method offers lower speedup over channel-by-channel methods
but higher speedup over specialized kernel methods on shallow MobileNet and thinner MobileNet:
our method offers 6.47 × speedup on Caffe and 1.8 × on TensorFlow over their original implementations.
But on low-resolution MobileNets, the results are opposite:
our method achieves 14.6 × speedup on Caffe and only 1.2 × on TensorFlow.
These differences are attributed to the change in the number of wide-and-small depthwise convolution layers.
Shallow MobileNet removes 5 wide-and-small depthwise convolution layers,
while thinner MobileNet reduce the number of channels in every layer.
This reduces the number of wide-and-small layers, which improves the average utilization of GPU resources in the channel-by-channel methods and provides better performance.
But for the specialized kernel methods, the average shared memory and cache hit-rate is lower.
Low-resolution MobileNet shrink every feature map in the networks.
Small features maps further reduce the utilization of GPU resources in the channel-by-channel methods.
The specialized kernel methods benefit more from the shrinking of feature maps because of higher shared memory and caches hit rate.



 §.§ Memory Efficiency



One potential side effect of our method is that the increase of the parameters may lead to large GPU memory usage.
We present extensive experiments with GPU memory usage on Darknet, Caffe and PyTorch to evaluate the memory efficiency of our method.
We compare our method with two channel-by-channel methods ( and ) and the specialized kernel method.
From Table <ref>,
the increase in memory usage of our method is negligible compared to  and the specialized kernel method,
and our method consumes less GPU memory than ,
proving our method is memory-efficient.



§ DISCUSSION




 §.§ Extra Computational Cost


As demonstrated in Section <ref>,
by converting a depthwise convolution into a standard convolution using diagonalwise refactorization,
extra computational cost is introduced.
But this extra computational cost can be ignored compared with the speedup achieved by our method for three reasons.

Firstly, our method is focused on the training performance of ConvNets
built with depthwise convolutions.
During training procedure, execution speed and memory consumption are the two key points
which affect the training time and the batch size,
while the number of floating point operations is less concerned.
From these two aspects, 
our method achieves significant speedup on five popular deep learning frameworks
with little impact on memory utilization (as demonstrated in Section <ref>).
These advantages make our method more efficient for training depthwise convolutions.

Secondly, in practice, the reduction in computational cost by depthwise convolutions does not provide high training speed
on hardware with high parallelism like GPUs.
This is because the performance is significantly affected by the hit rates of shared memory and cache,
but not rigorously related to the number of floating point operations.
From Table <ref>,
channel-by-channel method and specialized kernel method spend most of the training time (83% for channel-by-channel method and 72% for specialized kernel method) on depthwise convolutions,
which have only 3% of the FLOPs,
due to the inefficient shared memory utilization (in channel-by-channel method)
and the lack of algorithm-level and microarchitecture-level optimization (in specialized kernel method).
Instead, our method increases the inter-channel parallelism and leverages the highly efficient cuDNN optimization.
By this means, the extra cost is hidden by the parallel computation and has little impact on training speed,
which provides significant speedup compared with the other two methods.

Thirdly, our method is much more implementation-friendly than the specialized kernel method of TensorFlow,
and provides better compatibility by using cuDNN.
Instead of manually designing kernels to handle various cases,
our method only needs to add a simple mask filtering on regular convolutions and provide high efficiency.




 §.§ Extensibility


In this subsection
we briefly discuss the extensibility of the proposed method.
The utilization of mask matrices makes diagonalwise refactorization
easy to be adopted in networks with sparse connections.

One example is group convolution.
Group convolutions are adopted in the state-of-the-art networks such as <cit.> and <cit.>.
Depthwise convolution could be viewed as a special case of group convolution
where the number of groups equals the number of channels.
In a more general case,
several groups in a group convolution can be rearranged into a larger diagonalwise group
using our method
and the computation can be accelerated by increasing the parallelism between groups.
Moreover, our method can be utilized to more flexible networks architectures which adopt convolutional kernels of different sizes to each group.

Another example is network pruning,
a technique that cuts off redundant weights/connections and accelerates the inference procedure.
The flexibility of network pruning strategies introduces extra cost into the training procedure,
because the connections between neurons in a network become sparser and harder to control.
Our method can accelerate the training procedure and eliminate the extra cost 
by setting pruned positions in mask matrices to 0 and other positions to 1,
and offers significant speedup when pruning networks built with depthwise convolutions and group convolutions.



§ CONCLUSION


In this paper,
we analyze the problems of typical methods for implementing depthwise convolutions in popular deep learning frameworks,
and propose a novel method (called diagonalwise refactorization)
to accelerate the training of depthwise convolution layers.
By rearranging the depthwise filters in a large diagonal weight matrix,
our method increases the computation parallelism in depthwise convolutions.
Experiments on five popular frameworks show that
the diagonalwise refactorization method offers significant acceleration.



§ ACKNOWLEDGMENT


This work is supported by the Major State Basic Research Development Program of China (973) under Grant no. 2014CB340303.

IEEEtran


