MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution
Taojiannan Yang Sijie Zhu Chen Chen Shen Yan Mi Zhang Andrew Willis




headings
1196  

 













Abbreviated paper title






F. Author et al.



University of North Carolina at Charlotte 
 {tyang30,szhu3,chen.chen,arwillis}@uncc.edu Michigan State University

 {yanshen6,mizhang}@msu.edu


MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution
    Taojiannan Yang1 Sijie Zhu1
 Chen Chen1 Shen Yan2 Mi Zhang2 Andrew Willis1

    December 30, 2023
=================================================================================





 We propose the width-resolution mutual learning method (MutualNet) to train a network that is executable at dynamic resource constraints to achieve adaptive accuracy-efficiency trade-offs at runtime. 
 
 
 Our method trains a cohort of sub-networks with different widths[Number of channels in a layer.] using different input resolutions to mutually learn multi-scale representations for each sub-network.
 It achieves consistently better ImageNet top-1 accuracy over the state-of-the-art adaptive network US-Net <cit.> under different computation constraints, and outperforms the best compound scaled MobileNet in EfficientNet <cit.> by 1.5%. The superiority of our method is also validated on COCO object detection and instance segmentation as well as transfer learning. Surprisingly, the training strategy of MutualNet can also boost the performance of a single network, which substantially outperforms the powerful AutoAugmentation <cit.> in both efficiency (GPU search hours: 15000 vs. 0) and accuracy (ImageNet: 77.6% vs. 78.6%). Code is available at <https://github.com/taoyang1122/MutualNet>









§ INTRODUCTION


r0.43

    

    
    < g r a p h i c s >

    

    Accuracy-FLOPs curves of US-Net+ and US-Net. 
    




Deep neural networks have triumphed over various perception tasks.
However, deep networks usually require large computational resources, making them hard to deploy on mobile devices and embedded systems. This motivates research in reducing the redundancy in deep neural networks by designing efficient convolutional blocks <cit.> or pruning unimportant network connections <cit.>. However, these works ignore the fact that the computational cost is determined by both the network scale and input scale. Only focusing on reducing network scale cannot achieve the optimal accuracy-efficiency trade-off. EfficientNet <cit.> has acknowledged the importance of balancing among network depth, width and resolution. 

But it considers network scale and input scale separately.
The authors conduct grid search over different configurations and choose the best-performed one, while we argue that network scale and input scale should be considered jointly in learning to take full advantage of the information embedded in different configurations. 

Another issue that prevents deep networks from practical deployment is that the resource budget (e.g., battery condition) varies in real-world applications, while traditional networks are only able to run at a specific constraint (e.g., FLOP). To address this issue, SlimNets <cit.> are proposed to train a single model to meet the varying resource budget at runtime. They only reduce the network width to meet lower resource budgets. As a result, the model performance drops dramatically as computational resource goes down. Here, we provide a concrete example to show the importance of balancing between input resolution and network width for achieving better accuracy-efficiency trade-offs. Specifically, to meet a dynamic resource constraint from 13 to 569 MFLOPs on MobileNet v1 backbone, US-Net <cit.> needs a network width range of [0.05,1.0]× given a 224×224 input, while this constraint can also be met via a network width of [0.25,1.0]× by adjusting the input resolution from {224, 192, 160, 128} during test time. We denote the latter model as US-Net+. As shown in Fig. <ref>, simply combining different resolutions with network widths during inference can achieve a better accuracy-efficiency trade-off than US-Net without additional efforts.























Inspired by the observations above, we propose a mutual learning scheme which incorporates both network width and input resolution into a unified learning framework. 

As depicted in Fig. <ref>, our framework feeds different sub-networks with different input resolutions. Since sub-networks share weights with each other, each sub-network can learn the knowledge shared by other sub-networks, which enables them to capture multi-scale representations from both network scale and input resolution.  
Table <ref> provides a comparison between our framework and previous works. In
summary, we make the following contributions:


   
  * We highlight the importance of input resolution for efficient network design. Previous works either ignore it or treat it independently from network structure. In contrast, we embed network width and input resolution in a unified mutual learning framework to learn a deep neural network (MutualNet) that can achieve adaptive accuracy-efficiency trade-offs at runtime. 
   
  * We carry out extensive experiments to demonstrate the effectiveness of our MutualNet. It significantly outperforms independently-trained networks and US-Net on various network structures, datasets and tasks under different constraints. To the best of our knowledge, we are the first to benchmark arbitrary-constraint adaptive networks on object detection and instance segmentation.
   
  * We conduct comprehensive ablation studies to analyze the proposed mutual learning scheme. We further demonstrate that our framework is promising to serve as a plug-and-play strategy to boost the performance of a single network, which substantially outperforms the popular performance-boosting methods, e.g., data augmentations <cit.>, SENet <cit.> and knowledge distillation <cit.>.
   
  * The proposed framework is a general training scheme and model-agnostic. It can be applied to any networks without making any adjustments to the structure. This makes it compatible with other state-of-the-art techniques (e.g., Neural Architecture Search (NAS) <cit.>, AutoAugmentation <cit.>).




§ RELATED WORK


Light-weight Network. 
There has recently been a flurry of interest in designing light-weight networks. MobileNet <cit.> factorizes the standard 3×3 convolution into a 3×3 depthwise convolution and a 1×1 pointwise convolution which reduce computation cost by several times. ShuffleNet <cit.> separates the 1×1 convolution into group convolutions to further boost computation efficiency.

MobileNet v2 <cit.> proposes the inverted residual and linear block for low-complexity networks. ShiftNet <cit.> introduces a zero-flop shift operation to reduce computation cost. Most recent works <cit.> also apply neural architecture search methods to search efficient networks. However, none of them considers the varying resource constraint during runtime in real-world applications. To meet different resource budgets, these methods need to deploy several models and switch among them, which is not scalable.

Adaptive Neural Networks. To meet the dynamic constraints in real-world applications, MSDNet <cit.> proposes a multi-scale and coarse to fine densenet framework. It has multiple classifiers and can make early predictions to meet varying resource demands. NestedNet <cit.> uses a nested sparse network which 

consists of multiple levels

to enable nested learning. S-Net <cit.> introduces a 4-width framework to incorporate different complexities into one network, and proposes the switchable batch normalization for slimmable training. <cit.> leverages knowledge distillation to train a multi-exit network. However, these approaches can only execute at a limited number of constraints. US-Net <cit.> can instantly adjust the runtime network width for arbitrary accuracy-efficiency trade-offs. 

But its performance degrades significantly as the budget lower bound goes down. <cit.> proposes progressive shrinking to finetune sub-networks from a well-trained large network, but the training process is complex and expensive.





Multi-scale Representation Learning. The effectiveness of multi-scale representation has been explored in various tasks. FPN <cit.> fuses pyramid features for object detection and segmentation. <cit.> proposes a multi-grid convolution to pass message across the scale space. HRNet <cit.> designs a multi-branch structure to exchange information across different resolutions. However, these works resort to the multi-branch fusion structure which is unfriendly to parallelization <cit.>. Our method does not modify the network structure, and the learned multi-scale representation is not only from image scale but also from network scale.





§ METHODOLOGY



 §.§ Preliminary




Sandwich Rule. US-Net <cit.> trains a network that is executable at any resource constraint. The solution is to randomly sample several network widths for training and accumulate their gradients for optimization. However, the performance of the sub-networks is bounded by the smallest width (e.g., 0.25×) and the largest width (e.g., 1.0×). Thus, the sandwich rule is introduced to sample the smallest and largest widths plus two random ones for each training iteration.

Inplace Distillation. Knowledge distillation <cit.> is an effective method to transfer knowledge from a teacher network to a student network. Following the sandwich rule, since the largest network is sampled in each iteration, it is natural to use the largest network, which is supervised by the ground truth labels, as the teacher to guide smaller sub-networks in learning. This gives a better performance than training all sub-networks with ground truth labels.

Post-statistics of Batch Normalization (BN). US-Net proposes that each sub-network needs their own BN statistics (mean and variance), but it is insufficient to store the statistics of all the sub-networks. Therefore, US-Net collects BN statistics for the desired sub-network after training. Experimental results show that 2,000 samples are sufficient to get accurate BN statistics.



 §.§ Rethinking Efficient Network Design


The computation cost of a vanilla convolution is C_1 × C_2 × K × K × H × W, where C_1 and C_2 are the number of input and output channels, K is the kernel size, H and W are output feature map sizes. Most previous works only focus on reducing C_1 × C_2. The most widely used group convolution decomposes standard convolution into groups to reduce the computation to C_1 × (C_2/g) × K × K × H × W, where g is the number of groups. A larger g gives a lower computation but leads to higher memory access cost (MAC) <cit.>, making the network inefficient in practical applications. Pruning methods <cit.> also only consider reducing structure redundancies.


In our approach, we shift the attention to reducing H × W, i.e., lowering input resolution for the following reasons. First, as demonstrated in Fig. <ref>, balancing between width and resolution achieves better accuracy-efficiency trade-offs. Second, downsampling input resolution does not necessarily hurt the performance. It even sometimes benefits the performance. <cit.> points out that lower image resolution may produce better detection accuracy by reducing focus on redundant details. Third, different resolutions contain different information <cit.>. Lower resolution images may contain more global structures while higher resolution ones may encapsulate more fine-grained patterns. Learning multi-scale representations from different scaled images and features has been proven effective in previous works <cit.>. But these methods resort to a multi-branch structure which is unfriendly to parallelization <cit.>. Motivated by these observations, we propose a mutual learning framework to consider network scale and input resolution simultaneously for effective network accuracy-efficiency trade-offs.


















 §.§ Mutual Learning Framework


Sandwich Rule and Mutual Learning.
As discussed in Section <ref>, different resolutions contain different information. We want to take advantage of this attribute to learn robust representations and better width-resolution trade-offs. 

The sandwich rule in US-Net can be viewed as a scheme of mutual learning <cit.> where an ensemble of networks are learned collaboratively. 
Since the sub-networks share weights with each other and are optimized jointly, they can transfer their knowledge to each other. Larger networks can take advantage of the features captured by smaller networks. Also, smaller networks can benefit from the stronger representation ability of larger networks. In light of this, we feed each sub-network with different input resolutions. By sharing knowledge, each sub-network is able to capture multi-scale representations.

Model Training. We present an example to illustrate our framework in Fig. <ref>. We train a network where its width ranges from 0.25× to 1.0×. We first follow the sandwich rule to sample four sub-networks, i.e., the smallest (0.25×), the largest (1.0×) and two random width ratios α_1, α_2 ∈ (0.25, 1). Then, unlike traditional ImageNet training with 224× 224 input, we resize the input image to four resolutions {224, 196, 160, 128} and feed them into different sub-networks. We denote the weights of a sub-network as W_0:w, where w ∈ (0, 1] is the width of the sub-network and 0:w means the sub-network adopts the first w × 100% weights of each layer of the full network. I_R=r represents a r× r input image. Then N(W_0:w,I_R=r) represents the output of a sub-network with width w and input resolution r× r. For the largest sub-network (i.e., the full-network in Fig. <ref>), we always train it with the highest resolution (224× 224) and ground truth label y. The loss for the full network is 

    loss_full = CrossEntropy(N(W_0:1,I_R=224),  y).

For the other sub-networks, we randomly pick an input resolution from {224, 196, 160, 128} and train it with the output of the full-network. The loss for the i-th sub-network is

    loss_sub_i = KLDiv(N(W_0:w_i,I_R=r_i),  N(W_0:1,I_R=224)),

where KLDiv is the Kullback-Leibler divergence. The total loss is the summation of the full-network and sub-networks, i.e.,

    loss = loss_full+∑_i=1^3loss_sub_i.

The reason for training the full-network with the highest resolution is that the highest resolution contains more details. Also, the full-network has the strongest learning ability to capture the  discriminatory information from the image data.

Mutual Learning from Width and Resolution. In this part, we explain why the proposed framework can mutually learn from different widths and resolutions. For ease of demonstration, we only consider two network widths 0.4× and 0.8×, and two resolutions 128 and 192 in this example. As shown in Fig. <ref>, sub-network 0.4× selects input resolution 128, sub-network 0.8× selects input resolution 192. Then we can define the gradients for sub-network 0.4× and 0.8× as ∂ l_W_0:0.4,I_R=128/∂ W_0:0.4 and ∂ l_W_0:0.8,I_R=192/∂ W_0:0.8, respectively. Since sub-network 0.8× shares weights with 0.4×, we can decompose its gradient as 

    ∂ l_W_0:0.8,I_R=192/∂ W_0:0.8=∂ l_W_0:0.8,I_R=192/∂ W_0:0.4⊕∂ l_W_0:0.8,I_R=192/∂ W_0.4:0.8

where ⊕ is vector concatenation. Since the gradients of the two sub-networks are accumulated during training, the total gradients are computed as

    ∂ L/∂ W   =∂ l_W_0:0.4,I_R=128/∂ W_0:0.4 + ∂ l_W_0:0.8,I_R=192/∂ W_0:0.8
       = ∂ l_W_0:0.4,I_R=128/∂ W_0:0.4 + (∂ l_W_0:0.8,I_R=192/∂ W_0:0.4⊕∂ l_W_0:0.8,I_R=192/∂ W_0.4:0.8) 
       = ∂ l_W_0:0.4,I_R=128 + ∂ l_W_0:0.8,I_R=192/∂ W_0:0.4⊕∂ l_W_0:0.8,I_R=192/∂ W_0.4:0.8








Therefore, the gradient for sub-network 0.4× is ∂ l_W_0:0.4,I_R=128 + ∂ l_W_0:0.8,I_R=192/∂ W_0:0.4, and it consists of two parts. The first part is derived from itself (0 : 0.4×) with 128 input resolution. The second part is derived from sub-network 0.8× (i.e., 0:0.4× portion) with 192 input resolution. Thus the sub-network is able to capture multi-scale representations from different input resolutions and network scales. Due to the random sampling of network width, every sub-network is able to learn multi-scale representations in our framework.




Model Inference. The trained model is executable at various width-resolution configurations. The goal is to find the best configuration under a particular resource constraint. A simple way to achieve this is via a query table. Specifically, we sample network width from 0.25× to 1.0× with a step-size of 0.05×, and sample network resolution from {224, 192, 160, 128}. We test all these width-resolution configurations on a validation set and choose the best one under a given constraint (FLOPs or latency). Since there is no re-training, the whole process is once for all. 



§ EXPERIMENTS


In this section, we first present our results on ImageNet <cit.> classification to illustrate the effectiveness of MutualNet. Next, we conduct extensive ablation studies to analyze the mutual learning scheme. Finally, we apply MutualNet to transfer learning datasets and COCO <cit.> object detection and instance segmentation to demonstrate its robustness and generalization ability. 



 §.§ Evaluation on ImageNet Classification


We compare our MutualNet with US-Net and independently-trained networks on the ImageNet dataset. 
We evaluate our framework on two popular light-weight structures, MobileNet v1 <cit.> and MobileNet v2 <cit.>. 
These two networks also represent non-residual and residual structures respectively.

Implementation Details. We compare with US-Net under the same dynamic FLOPs constraints ([13, 569] MFLOPs on MobileNet v1 and [57, 300] MFLOPs on MobileNet v2). US-Net uses width scale [0.05, 1.0]× on MobileNet v1 and [0.35, 1.0]× on MobileNet v2 based on the 224×224 input resolution. To meet the same dynamic constraints, our method uses width scale [0.25, 1.0]× on MobileNet v1 and [0.7, 1.0]× on MobileNet v2 with downsampled input resolutions {224, 192, 160, 128}. Due to the lower input resolutions, our method is able to use higher width lower bounds (i.e., 0.25× and 0.7×) than US-Net. The other training settings are the same as US-Net.
    
Comparison with US-Net. We first compare our framework with US-Net on MobileNet v1 and MobileNet v2 backbones. The Accuracy-FLOPs curves are shown in Fig. <ref>. We can see that our framework consistently outperforms US-Net on both MobileNet v1 and MobileNet v2 backbones. Specifically, we achieve significant improvements under small computation costs. This is because our framework considers both network width and input resolution, and can find a better balance between them. For example, if the resource constraint is 150 MFLOPs, US-Net has to reduce the width to 0.5× given its constant input resolution 224, while our MutualNet can meet this budget by a balanced configuration of (0.7× - 160), leading to a better accuracy (65.6% (Ours) vs. 62.9% (US-Net) as listed in the table of Fig. <ref>(a)). On the other hand, our framework is able to learn multi-scale representations which further boost the performance of each sub-network. We can see that even for the same configuration (e.g., 1.0×-224) our approach clearly outperforms US-Net, i.e., 72.4% (Ours) vs. 71.7% (US-Net) on MobileNet v1, and 72.9% (Ours) vs. 71.5% (US-Net) on MobileNet v2 (Fig. <ref>).




Comparison with Independently Trained Networks. Different scaled MobileNets are trained separately in <cit.>. The authors consider width and resolution as independent factors, thus cannot leverage the information contained in different configurations. We compare the performance of MutualNet with independently-trained MobileNets under different width-resolution configurations in Fig. <ref>. For MobileNet v1, widths are selected from {1.0×, 0.75×, 0.5×, 0.25×}, and resolutions are selected from {224, 192, 160, 128}, leading to 16 configurations in total. Similarly, MobileNet v2 selects configurations from {1.0×, 0.75×, 0.5×, 0.35×} and {224, 192, 160, 128}. From Fig. <ref>, our framework consistently outperforms MobileNets. Even for the same width-resolution configuration (although it may not be the best configuration MutualNet finds at that specific constraint), MutualNet can achieve much better performance. This demonstrates that MutualNet not only finds the better width-resolution balance but also learns stronger representations by the mutual learning scheme.







 §.§ Ablation Study



  §.§.§ Balanced Width-Resolution Configuration via Mutual Learning.
 

As evident in Fig. <ref>, we can apply different resolutions to US-Net during inference to yield improvement over the original US-Net. However, this cannot achieve the optimal width-resolution balance due to lack of width-resolution mutual learning. In the experiment, we test US-Net at width scale [0.25, 1.0]× with input resolutions {224, 192, 160, 128} and denote this improved model as US-Net+. In Fig. <ref>, we plot the Accuracy-FLOPs curves of our method and US-Net+ based on MobileNet v1 backbone, and highlight the selected input resolutions with different colors. As we decrease the FLOPs (569→468 MFLOPs), our MutualNet first reduces network width to meet the constraint while keeping the 224×224 resolution (red line in Fig. <ref>). After 468 MFLOPs, MutualNet selects lower input resolution (192) and continues reducing the width to meet the constraint. On the other hand, US-Net+ cannot find such balance. It always slims the network width and uses the same (224) resolution as the FLOPs decreasing until it goes to really low. This is because US-Net+ does not incorporate input resolution into the learning framework. Simply applying different resolutions during inference cannot achieve the optimal width-resolution balance.



  §.§.§ Difference with EfficientNet.


EfficientNet acknowledges the importance of balancing between network width, depth and resolution. But they are considered as independent factors. The authors use grid search over these three dimensions and train each configuration independently to find the optimal one under certain constraint, while our MutualNet incorporates width and resolution in a unified framework. We compare with the best model scaling EfficientNet finds for MobileNet v1 at 2.3 BFLOPs (scale up baseline by 4.0×).  Similar to this scale setting, we train our framework with a width range of [1.0×,2.0×] (scaled by [1.0×,4.0×]), and select resolutions from {224, 256, 288, 320}. This makes MutualNet executable in the range of [0.57, 4.5] BFLOPs. We pick the best performed width-resolution configuration at 2.3 BFLOPs. The results are compared in Table <ref>. MutualNet achieves significantly better performance than EfficientNet because it can capture multi-scale representations for each model scaling due to the width-resolution mutual learning.




















  §.§.§ Difference with Multi-scale Data Augmentation.

Multi-scale data augmentation is popular in detection and segmentation. We show that, first, our method is different from multi-scale data augmentation in principle. Second, our method significantly outperforms multi-scale data augmentation.

In multi-scale data augmentation, 

the network may take images of different resolutions in different iterations. 
But within each iteration, the network weights are optimized in the same resolution direction. While our method randomly samples four sub-networks which share weights with each other. Since sub-networks can select different image resolutions, the weights are optimized in a mixed resolution direction in each iteration as illustrated in Fig. <ref>. This enables each sub-network to effectively learn multi-scale representations from width and resolution. To validate the superiority of our mutual learning scheme, we apply multi-scale data augmentation to MobileNet and US-Net.

MobileNet + Multi-scale data augmentation. We train MobileNet v2 (1.0× width) with multi-scale images. To have a fair comparison, input images are randomly sampled from scales {224, 192, 160, 128} and the other settings are the same as MutualNet. As shown in Table <ref>, multi-scale data augmentation only marginally improves the baseline (MobileNet v2) while our MutualNet (MobileNet v2 backbone) clearly outperforms both of them by considerable margins.









US-Net + Multi-scale data augmentation. Different from our framework which feeds different scaled images to different sub-networks, in this experiment, we randomly choose a scale from {224, 192, 160, 128} and feed the same scaled image to all sub-networks in each iteration. That is each sub-network takes the same image resolution. In this way, the weights are still optimized towards a single resolution direction in each iteration, but the direction varies among different iterations. The experiment is based on MobileNet v2. Width and resolution settings are the same as those in Sec. <ref>. As shown in Fig. <ref>, our method clearly performs better than US-Net + multi-scale data augmentation over the entire FLOPs spectrum. These experiments demonstrate that the improvement comes from our mutual learning scheme rather than the multi-scale data augmentation.



  §.§.§ Effects of Width Lower Bound.
 The executable constraint range and model performance are affected by the width lower bound. To study its effects, we conduct experiments with three different lower bounds (0.7×, 0.8×, 0.9×) on MobileNet v2. The results in Fig. <ref> show that a higher lower bound gives better overall performance, but the executable range is narrower. 

One interesting observation is that the performance of the full-network (1.0×-224) is also largely improved as the width lower bound increases from 0.7× to 0.9×. This property is not observed in US-Net. We attribute this to the robust and well-generalized multi-scale representations which can be effectively re-used by the full-network, while in US-Net, the full-network cannot effectively benefit from 
sub-networks.












  §.§.§ Boosting Single Network Performance.
 As discussed above, the performance of the full-network is greatly improved as we increase the width lower bound. Therefore, we apply our training framework to improve the performance of a single full network. We compare our method with the popular performance-boosting techniques (e.g., AutoAugmentation (AA) <cit.>, SENet <cit.> and Knowledge Distillation (KD) <cit.>) to show its superiority. We conduct experiments using Wide-ResNet <cit.> on Cifar-10 and Cifar-100 and ResNet-50 on ImageNet. MutualNet adopts the width range [0.9, 1.0]× as it achieves the best-performed full-network in Fig. <ref>. The resolution is sampled from {32, 28, 24, 20} on Cifar-10 and Cifar-100 and {224, 192, 160, 128} on ImageNet. Wide-ResNet is trained for 200 epochs following <cit.>. ResNet is trained for 100 epochs. The results are compared in Table <ref> and Table <ref>. MutualNet achieves substantial improvements over other techniques. It is important to note that MutualNet is a general training scheme which does not need the expensive searching procedure or additional network blocks or stronger teacher networks. Moreover, MutualNet training is as easy as the regular training process, and is orthogonal to other performance-boosting techniques, e.g., AutoAugmentation <cit.>. Therefore, it can be easily combined with those methods.





 §.§ Transfer Learning

To evaluate the representations learned by our method, we further conduct experiments on three popular transfer learning datasets, Cifar-100 <cit.>, Food-101 <cit.> and MIT-Indoor67 <cit.>. Cifar-100 is superordinate-level object classification, Food-101 is fine-grained classification and MIT-Indoor67 is scene classification. Such large variety is suitable to evaluate the robustness of the learned representations. We compare our approach with US-Net and MobileNet v1. We fine-tune ImageNet pre-trained models
[For US-Net, we use the officially released model at width [0.25, 1.0]×.] 
with a batch size of 256, initial learning rate of 0.1 with cosine decay schedule and a total of 100 epochs. Both MutualNet and US-Net are trained with width range [0.25, 1.0]× and tested with resolutions from {224, 192, 160, 128}.
The results are shown in Fig. <ref>. Again, our MutualNet achieves consistently better performance than US-Net and MobileNet. This verifies that MutualNet is able to learn well-generalized representations.






 §.§ Object Detection and Instance Segmentation



We also evaluate our method on COCO object detection and instance segmentation <cit.>. The experiments are based on Mask-RCNN-FPN <cit.> and MMDetection <cit.> toolbox on VGG-16 <cit.> backbone. We first pre-train VGG-16 on ImageNet following US-Net and MutualNet respectively. Both methods are trained with width range [0.25, 1.0]×. Then we fine-tune the pre-trained models on COCO. The FPN neck and detection head are shared among different sub-networks. For simplicity, we don't use inplace distillation. Rather, each sub-network is trained with the ground truth. The other training procedures are the same as training ImageNet classification. Following common settings in object detection, US-Net is trained with image resolution 1000×600. Our method randomly selects resolutions from 1000 ×{600, 480, 360, 240}. All models are trained with 2× schedule for better convergence and tested with different image resolutions. The mean Average Precision (AP at IoU=0.50:0.05:0.95) are presented in Fig. <ref>. These results reveal that our MutualNet significantly outperforms US-Net under all resource constraints. Specifically, for the full network (1.0×-600), MutualNet significantly outperforms both US-Net and independent network. This again validates the effectiveness of our width-resolution mutual learning scheme. Fig. <ref> provides some visual examples which reveal that MutualNet is more robust to small-scale and large-scale objects than US-Net.




§ CONCLUSION AND FUTURE WORK

This paper highlights the importance of simultaneously considering network width and input resolution for efficient network design. A new framework namely MutualNet is proposed to mutually learn from network width and input resolution for adaptive accuracy-efficiency trade-offs. Extensive experiments have shown that it significantly improves inference performance per FLOP on various datasets and tasks. The mutual learning scheme is also demonstrated to be an effective training strategy for boosting single network performance. The generality of the proposed framework allows it to translate well to generic problem domains. This also makes logical extensions readily available by adding other network dimensions, e.g., network depth and bit-width, to the framework. The framework can also be extended to video input and 3D neural networks, where both spatial and temporal information can be leveraged.




























































































































































































































splncs04

