Promoting CNNs with Cross-Architecture Knowledge Distillation for Efficient Monocular Depth Estimation
Zhimeng Zheng Tao Huang Gongsheng Li Zuyi Wang





DisDepth]Promoting CNNs with Cross-Architecture Knowledge Distillation
for Efficient Monocular Depth Estimation


























  Zhejiang University
  Hangzhou
  China
22110145@zju.edu.cn





Corresponding author.

  The University of Sydney
  Sydney
  Australia
thua7590@uni.sydney.edu.au



  Zhejiang University
  Hangzhou
  China
22210102@zju.edu.cn



  Zhejiang University
  Hangzhou
  China
zuyiwang@zju.edu.cn
























Recently, the performance of monocular depth estimation (MDE) has been significantly boosted with the integration of transformer models. However, the transformer models are usually computationally-expensive, and their effectiveness in light-weight models are limited compared to convolutions. This limitation hinders their deployment on resource-limited devices. In this paper, we propose a cross-architecture knowledge distillation method for MDE, dubbed DisDepth, to enhance efficient CNN models with the supervision of state-of-the-art transformer models. Concretely, we first build a simple framework of convolution-based MDE, which is then enhanced with a novel local-global convolution module to capture both local and global information in the image. To effectively distill valuable information from the transformer teacher and bridge the gap between convolution and transformer features, we introduce a method to acclimate the teacher with a ghost decoder. The ghost decoder is a copy of the student's decoder, and adapting the teacher with the ghost decoder aligns the features to be student-friendly while preserving their original performance. Furthermore, we propose an attentive knowledge distillation loss that adaptively identifies features valuable for depth estimation. This loss guides the student to focus more on attentive regions, improving its performance. Extensive experiments on KITTI and NYU Depth V2 datasets demonstrate the effectiveness of DisDepth. Our method achieves significant improvements on various efficient backbones, showcasing its potential for efficient monocular depth estimation.







<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224</concept_id>
       <concept_desc>Computing methodologies Computer vision</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010257.10010293</concept_id>
       <concept_desc>Computing methodologies Machine learning approaches</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Computer vision
[300]Computing methodologies Machine learning approaches

























[
    Zuyi Wang
    April 29, 2024
==================



§ INTRODUCTION

Monocular depth estimation(MDE)  <cit.>  has achieved remarkable success by the adventure of automatic feature engineering in deep neural networks. With the advent of better global feature representation in vision transformers <cit.>, recent state-of-the-art MDE methods resort to complicated transformer-based backbones <cit.> and decoder structures <cit.>, which are computational-expensive and hard to be deployed on resource-constrained devices. For example, PixelFormer <cit.> with Swin-L backbone has ∼704B FLOPs and ∼271M parameters, which is too heavy for edge devices to inference.



In this paper, we aim to obtain competitive performance while maintaining an efficient inference of MDE model by exploring a CNN-based framework without transformers. This is achieved by proposing (1) an efficient CNN-based MDE framework and (2) a novel cross-architecture knowledge distillation (KD) method that adjusts and distills the transformer features into CNNs. Concretely, as shown in Figure <ref>, we first construct a simple and efficient CNN-based MDE framework that contains a CNN backbone, a simple decoder with convolutions and upsamplings, and two heads for predicting bin centers and bin probabilities <cit.>. Then, a local-global convolution (LG-Conv) module is proposed to enhance the capability of global representations in CNNs. Our LG-Conv is inspired by our analysis that the role of self-attention in transformers is to aggregate and broadcast the information globally among tokens, while this global information exchange can also be done with simple convolution and pooling operations. The resulting global branch in LG-Conv is a supplement of the original local convolution, and can be directly utilized on a pretrained backbone without retraining on pretraining dataset. Our LG-Conv is efficient and friendly to deployment, and experiments show that it effectively extracts the global information and boosts the performance.



Besides, we propose a new KD method for cross-architecture distillation from transformers to CNNs. Our basic motivation is to leverage the state-of-the-art transformer-based MDE models to improve our CNN models with KD. However, we empirically find that due to capacity and architecture gaps, the student is difficult to imitate teacher's features, and result in poor distillation performance, and even worse than the CNN teacher with significant worse performance. To understand what restricts the distillation performance, we conduct experiments and find that the devil of cross-architecture KD is in the architecture-intrinsic information[For a brief, we use a fixed and pretrained student decoder to adapt the teacher features, and the teacher still obtains similar performance, while a significant gain on distillation performance has been obtained.]. Therefore, we intuitively decouple the information in teacher feature into two parts, namely intrinsic information and extrinsic information. The intrinsic information is architecture-independent, and small students with different architectures are arduous in learning this information. While the extrinsic information is crucial for generating the task predictions, and is common and invariant to different architectures; it can be easily adapted and distilled to students.

To this end, we propose to acclimate the transformer teacher with a ghost decoder, which is a copy of the student's decoder, so that we can obtain adapted teacher features that are more appropriate for distillation. Specifically, we inject a feature acclimation module (FAM, a transformer block) into the teacher backbone after each output layer, which is used to acclimate the teacher features. The resulting adapted features are fed into the ghost decoder to get the predictions, which are then passed to the task loss function. This error between prediction and ground-truth is used to optimize the FAM, and other modules including original backbone and ghost decoder are fixed. Through this adaptation, the intrinsic information in teacher features is repressed and task-relevant extrinsic information is acclimated, as the student-trained decoder prefers student-like features. Furthermore, instead of directly optimizing the distillation loss between adapted teacher feature and student feature, we introduce a attentive KD loss, which learns valuable regions in the teacher features, and then use the learned region importances to guide the student to focus more on valuable features.

In sum, our contribution is threefold.

    
  * We propose a simple CNN framework for MDE, with a supplementary local-global convolution module to enhance the global representations. Our framework achieves competitive performance and is more efficient than current state-of-the-art transformer-based frameworks.
    
  * We propose a novel cross-architecture knowledge distillation method, which effectively represses the architecture-intrinsic information and adapts task-relevant information in the transformer teacher features for better distillation to CNN student. Additionally, an attentive KD loss is introduced to learn and distill valuable regions in teacher features.
    
  * Extensive experiments on KITTI and NYU Depth V2 datasets demonstrate our effectiveness. As shown in Figure <ref>, our DisDepth achieves significant efficiency and performance superiorities on various model scales.






§ RELATED WORK




 §.§ Monocular Depth Estimation

Monocular depth estimation (MDE) is an active area of research that aims to regress a dense depth map from a single RGB image. The pioneer, <cit.>, which initiated the trend with its coarse-to-fine network and innovative scale-invariant log loss (SIlog) for optimisation. Building on this foundation, the field embraced architectural advances such as residual networks <cit.>, multi-scale fusion <cit.>, and transformers <cit.>. On the other hand, some works are proposed to improve the optimization techniques, including reverse Huber loss <cit.>, classification-based schemes <cit.>, and ordinal regression <cit.>. The field has also seen a move towards ordinal regression methods, which are similar to mask-based segmentation in their approach to depth discretisation. By associating each depth value with a confidence mask, the culmination in depth prediction becomes a weighted interplay of these discretised values.




 §.§ Knowledge Distillation


Knowledge distillation (KD) <cit.> is an idea seeded in the realm of image recognition by <cit.>, it serves as a conduit to transfer knowledge from a high-capacity teacher model to a more compact student model. Its versatility has been affirmed with its myriad applications across various computer vision tasks, including domain adaptation <cit.>, object detection <cit.>, and learning from noisy labels <cit.>. In the MDE context, <cit.> and <cit.> spearheaded efforts to harness the power of distillation, setting the stage for subsequent innovations. The application of distillation in depth estimation for resource-constrained environments, particularly mobile systems, is of great significance <cit.>, <cit.>, <cit.>. The challenge is balancing depth estimation accuracy with computational efficiency, crucial for real-time applications like autonomous driving. This balance becomes even more pronounced in bespoke hardware setups, like dual-pixel sensors <cit.>. While the journey has begun, the landscape of knowledge distillation in MDE, especially under constraints, remains a dynamic and evolving field.




§ OUR APPROACH




In this section, we present our method DisDepth. We begin by discussing the design of our CNN-only MDE framework. We then elaborate on our local-global convolution, which is designed to enhance the global representations of the model. Additionally, we introduce our cross-architecture knowledge distillation approach, where we adapt the transformer teacher with a ghost decoder. Finally, we propose an attentive knowledge distillation loss that helps identify valuable regions for the student model to focus on.



 §.§ Frustratingly Simple CNN Framework


To improve the performance of MDE, recent state-of-the-art approaches have often relied on complex and heavy decoder structures. For instance, AdaBins <cit.> proposes a mViT module with multiple transformer layers to enhance the decoded feature pyramid. DepthFormer <cit.> introduces an HAHI module that combines deformable cross-attention and self-attention to bridge the encoder and decoder. PixelFormer <cit.> utilizes a UNet structure with advanced feature fusion modules, including window cross-attention. However, these sophisticated decoder designs come at the cost of reduced efficiency and deployment challenges. The heavy computational requirements and complexity make it difficult to deploy these models in real-world scenarios.

In our research, we have discovered that by using a powerful backbone encoder, a simple CNN-based decoder can still achieve competitive performance in MDE. This finding has led us to develop a surprisingly simple CNN framework. For instance, in Table <ref>, our framework achieves an RMSE of 2.058 on the KITTI dataset when using the Swin-Large backbone. In comparison, the PixelFormer model, which employs an advanced decoder, achieves an RMSE of 2.081. This observation suggests that we can effectively replace complex and advanced decoder approaches with a simpler CNN-based decoder, allowing us to focus primarily on improving the encoder. This insight opens up new possibilities for developing efficient and effective MDE models that strike a balance between performance and simplicity.

Implementation details. Our framework is illustrated in Figure <ref> (see the bottom-half student model). With an input image, it first encodes its features using an encoder (backbone), then passes the features of the last four stages to get the feature pyramid. A simple convolutional and upsampling decoder <cit.> is employed to fuse and extract features from the feature pyramid. The decoder combines information from different scales to generate a final feature map. Following the approach introduced in AdaBins <cit.>, the feature map is passed through two heads. One head predicts the bin centers, while the other head predicts the bin probabilities. The predicted bin centers and probabilities are further processed to obtain the final predicted depth map.






 §.§ Enhanced CNNs with Local-Global Convolutions


Compared to CNNs, transformers <cit.> are acknowledged to have better global representation capability due to the global self-attention. Some recent methods <cit.> have successfully shown that transformers can significantly boost the MDE performance due to better representation power and relationship modeling. However, the computation of self-attention requires has a large peak memory footprint and has not been widely optimized and supported by edge computation frameworks, which have achieved remarkable success by deploying CNNs into real-life productions (see supplementary material for details).

To enhance the CNNs with global dependency, we propose a convolution-based module named local-global convolution (LG-Conv), which works as a supplement of the original local convolution layers. Different from recent approaches in bringing long-term dependency into CNNs, which focus on mixing transformers and convolutions into one block or design complicated computation scheme that cannot be efficiently deployed, our LG-Conv is composed with only convolutions and poolings, and has the following superiorities. (1) Our LG-Conv does not require retraining the backbone. (2) Our LG-Conv is efficient. (3) Our LG-Conv is deployment-friendly.

Implementation details. As shown in Figure <ref>, which a pretrained CNN backbone composed of multiple convolution layers, we expand each 3× 3 convolutions with an additional global branch in LG-Conv. The global branch takes the same input feature as the original local convolution. The feature is first transformed using a Conv-ReLU structure. Then, the hidden feature is obtained by concatenating the local feature with the global feature, which is obtained through average pooling. We utilize the hidden feature to predict multi-head attentions over the spatial dimension, which are then used to aggregate the pixel features into one feature. Additionally, we introduce a broadcast branch to predict probabilities that determine whether the aggregated global information should be applied to each pixel.
To ensure that the pretrained semantic information is not disturbed by directly supplementing the global branch's feature to the original feature, we propose initializing the scale parameters γ in the last batch normalization layer with a small value (e.g., 0.001). This initialization ensures that the global branch has minimal impact on the original features during the initial stages of training, allowing the backbone to evolve smoothly.




 §.§ Acclimating Teacher with A Ghost Decoder


Among existing methods, the transformer-based models outperform CNN-based models by a large margin. Therefore, to enhance our efficient model, an intuitive idea is to use the state-of-the-art transformer models as the teacher to distill our model (student). However, we empirically find that directly imitating the features of transformer teacher does not gains significant improvements as expected. For instance, as summarized in Table <ref>, we train our EfficientNet-B0 student with EfficientNet-B5 and Swin-L teachers, and show that, although the Swin-L has an obvious superiority in performance (2.06 RMSE vs. 2.60 RMSE), its distillation performance (2.95 RMSE) is largely behind the EfficientNet-B5 teacher (2.78 RMSE), which has a similar architecture to the student.

To better understand the distinction between CNN features and transformer features, we conduct experiments and make an interesting observation. By using a fixed student-trained decoder to replace the original decoder in the pretrained teacher model, and optimizing the teacher with task loss, we find that the new teacher model achieves similar performance compared to the original teacher. This trial successfully acclimate the teacher features to be similar to the student's features, while still preserving the task-relevant information necessary for good performance.

Based on this observation, we propose a hypothesis that the teacher features can be decoupled into two types of information: architecture intrinsic information and extrinsic information. The architecture intrinsic information is architecture-independent, and the student model finds it challenging to mimic this information due to its smaller capacity and architectural differences. On the other hand, the extrinsic information is task-relevant, and it is common and consistent across different architectures. These features can be easily adapted and imitated by the student model.

Consequently, to achieve effective and student-friendly cross-architecture distillation, we propose acclimating the transformer teacher model with the student's decoder and distilling knowledge from the acclimated features. This process is illustrated in Figure <ref>. To facilitate this acclimation, we introduce a ghost decoder, which is a continuous copy of the student's decoder throughout the entire training period. The original teacher features are adjusted using additional trainable feature acclimation modules (FAMs), the ghost decoder, and task loss, which is a transformer block in the teacher model. Subsequently, we perform distillation between the acclimated features of the teacher model and the features of the student model. The iterative procedure of our DisDepth is summarized in Algorithm <ref>. Note that the FAMs are trained simultaneously with the student model, at the early of training, the student decoder is not converged and the acclimated features are with low performance, so we add a warmup that starts distillation after a few epochs trained.










 §.§ Attentive Knowledge Distillation


We now have successfully built our method of acclimating teacher features. In feature distillation, how to select valuable features from the dense feature map for effective distillation is also a crucial problem  <cit.>. Considering that the teacher features are acclimated with the task loss, we can naturally get the importance of features by injecting a attention module between acclimated teacher feature and decoder. The attention module is also optimized with task loss. If a region is important to the performance, its attention weights should be large; in contrast, if a feature is useless to the prediction, we just multiply a zero weight to remove it. Therefore, our proposed loss attention module (LAM), is designed with a cross-attention layer. As illustrated in Figure <ref>, LAM uses the average of input feature as the query to capture the importance of each pixel, then the output feature is weighted by the importance scores. Besides, the scores, which represents the importances of each pixel, is also returned to compute the distillation loss.



Attentive distillation loss. Given the same image as input, we have the output features F_l^(t)∈ℝ^C× H× W and F_l^(s)∈ℝ^C× H× W of the teacher and student backbones, where C, H, W denote the numbers of channels, feature height, feature width, respectively, and l=1,...,L is the index of last L=4 stages of backbone. Our attentive KD first generates the spatial attentions A_l using the teacher features, then these attentions representing the importance of each pixel to the depth predictions, are multiplied onto the teacher and student features to guide the student focus on imitating the important regions of teacher features. The attentive KD loss is formulated as

    ℒ_kd := ∑_l=1^L||A_l(F^(s)_l - F^(t)_l)||_2^2.




Task loss. Following previous works <cit.>, we use a scaled version of the Scale-Invariant loss (SILog) <cit.> as our task loss. With the ground-truth depth d_i^* and the predicted depth d̂_i at each pixel index of n pixels, we first calculate the logarithmic distance g_i = log(d̂_i) - log(d^*_i), then the SILog is computed as

    ℒ_SILog = α√(1/n∑_i=1^n g_i^2 - β/n^2(∑_i=1^n g_i)^2),

we set α=10 and λ=0.85 in all our experiments.

Overall loss. As a result, our overall loss is the summation of task loss ℒ_SILog, teacher's task loss ℒ_SILog^(t), and attentive KD loss ℒ_kd, i.e.,

    ℒ = ℒ_SILog + ℒ_SILog^(t) + λℒ_kd,

where λ is the factor for balancing loss terms.





§ EXPERIMENTS




Datasets. Following previous works, we use two popular benchmark datasets KITTI <cit.> and NYU Depth V2 <cit.> to validate our efficacy on outdoor and indoor depth estimations, respectively.






Models. The evaluate our method sufficiently, we conduct experiments with various backbone variants, including EfficientNet <cit.>, GhostNet <cit.>, ResNet <cit.>, and Swin Transformer <cit.>.

Evaluation Metrics. We use standard metrics following previous studies <cit.>, such as average relative error (Abs Rel), root mean squared error (RMSE), and average log error (log10). In addition, we use threshold accuracy (δ_i) at thresholds δ= 1.25, 1.25^2, 1.25^3 to compare our method to state-of-the-art methods used in earlier works. For KITTI evaluation, we additionally use square relative error (Sq Rel).

Training strategies. The DisDepth is implemented in Pytorch <cit.>.We use Adam optimizer <cit.> with β_1=0.9, β_2=0.999, with a batch size of 8 and weight decay of 10^-2. After a warmup of 7 epochs, distillation begins.We use 25 epoch for both KITTI and NYU-V2 dataset with an initial learning rate of 4 × 10^-5, which is decreased linearly to 4 × 10^-6 across the training iterations. We use a similar test protocol as in <cit.>. We set KD weight λ=0.05 in all experiments.

Compared methods. We compare our DisDepth with recent state-of-the-art methods, including NeWCRFs <cit.>, BTS <cit.>, P3Depth <cit.>, AdaBins <cit.>, DepthFormer <cit.>, BinsFormer <cit.>, PixelFormer <cit.>, and LightDepth <cit.>.



 §.§ Results on KITTI

On the Eigen split of the KITTI dataset, we compare several state-of-the-art depth estimation methods with our proposed family of DisDepth models, as shown in Table <ref>. We can see that, the methods with transformer backbones have a large FLOPs, parameters, and latency, while our models enjoy significant efficiency and performance superiority compared to existing methods. For example, our DisDepth obtains 2.546 RMSE with only 35.7B, surpassing the 42.8B FLOPs LightDepth by a large margin of 0.377. With ResNet-50 backbone, DisDepth-R50 obtains similar performance compared to state-of-the-art approaches PixelFormer, while saving ∼30% FLOPs.




 §.§ Results on NYU Depth v2


We summarize the results on NYU Depth v2 dataset in Table <ref>. Similar to the results on KITTI, our models achieve competitive performance with obvious reductions on FLOPs, parameters, and latency. For example, our DisDepth-B0 achieves similar performance with only 7.5% of FLOPs compared to BTS.




 §.§ Ablation Study


Comparisons of student distilled by teacher with or without FA. In Table <ref>, we compare our method with the traditional KD method without feature acclimation (FA). The results show that, without feature acclimation, the cross-architecture distillation with Swin-L teacher obtains poor performance compared to EfficientNet-B5 teacher. While our FA can improve the distillation performance of both teachers, and the Swin-L teacher obtains better results since it is more advanced. This implies that the FA is crucial and effective for cross-architecture distillation.



Effects of the proposed components. We validate the effects of each of our proposed components in Table <ref>. (1) LG. The student model with our local-global convolution achieves a 0.18 improvement on RMSE. (2) KD. Compared to the independent training, the classical KD method further brings ∼0.20 decrease.  (3) FAM. Our feature acclimation significantly improves the performance by 0.27. (4) LAM. Our complete method with loss attention module obtains the optimal performance.






Performance of ghost teacher. To show how the teacher's performance is affected by the ghost decoder, we report the performance of the original teacher and ghost teacher in Table <ref>. We can see that, the ghost decoder only has a minor degradation of the performance. This demonstrate our hypothesis that the task-relevant extrinsic feature can be easily adapted and removing the architecture intrinsic feature would not affect the performance.



Warmup epochs. Table <ref> examines the performance of different warmup epochs. We obtain the optimal performance when the number of warmup epochs is set to 7.



Loss Weight.
In Table <ref>, we conduct experiments to tune the KD loss weight λ. We can see that, the λ=0.05 obtains the optimal performance.



Efficiency comparisons of student with or without local-global convolution. By incorporating Local-Global Convolutions (LG-Conv) into CNNs, there is a small rise in FLOPs, parameters, and latency, as shown in Table <ref>. Nevertheless, the benefit on performance is obvious. This indicates that LG-Conv is an effective tool for enhancing the capabilities of CNNs without imposing significant computational overloads.














§ CONCLUSION

This work presents DisDepth, an advanced monocular depth estimation (MDE) method that combines the robustness of transformer models with the efficiency of CNN architectures. By incorporating a specially designed local-global convolution module, DisDepth effectively captures both fine-grained details and broader scene context. The introduction of a ghost decoder mechanism streamlines the knowledge transfer process from transformers to CNNs, ensuring digestible knowledge alignment. Additionally, the attentive KD loss focuses on the most valuable features, resulting in more accurate depth predictions. Experimental results on the KITTI and NYU Depth V2 datasets demonstrate DisDepth's superior performance, achieving a harmonious balance between depth accuracy and computational efficiency. DisDepth represents a significant advancement in MDE and holds promise for various applications in computer vision.














ACM-Reference-Format




§ COMPARISONS OF DEPLOYING CNNS AND TRANSFORMERS IN EDGE COMPUTATION




 §.§ Challenges and Main Difficulties of Deploying Transformers


Transformers, with their large number of parameters in architectures such as Vision Transformers (ViT) <cit.> and BERT-like <cit.> Transformers, present a challenge in edge deployments due to their large model sizes. 
This size, combined with the inherent computational complexity, especially with the self-attention mechanism, demands more power and resources. Such requirements can lead to rapid battery drain, especially in real-time applications such as speech recognition or translation, where latency can be significant on devices with limited processing power. The sequence parallelism inherent in the Transformer architecture aids in highly parallel training. However, efficient deployment of these models is problematic in practice because generative inference progresses one token at a time, and the computation for each token sequentially depends on previously generated tokens <cit.>.
Moreover, while edge deployment frameworks aim to support a broad spectrum of deep learning operations, compatibility issues emerge with specific operations or implementations unique to Transformers. These architectures, which process variable-length sequences, often lead to dynamic computations <cit.>. Such computations might not align well with frameworks or devices optimized for static operations.  

Deploying Transformer models on devices such as mobile phones, IoT devices, robots, and especially the Edge TPU presents additional challenges due to their resource constraints. The Edge TPU, in particular, has a limited amount of on-chip memory and computational resources, making it difficult to deploy large Transformer models <cit.>.
Another obstacle is the mandatory conversion of Transformer models to the TensorFlow Lite (TFLite) format to make them compatible with the Edge TPU. This conversion, although crucial, can be tedious and may require additional hardware resources. Furthermore, the TFLite format itself, as the only format supported by the Edge TPU, has its own set of operational limitations, adding complexity to the deployment process.




 §.§ Advantages of CNNs over Transformers


Convolutional neural networks (CNNs), including architectures such as VGG<cit.> and ResNet<cit.>, are characterised by their computational simplicity, using operations such as  convolutions and pooling. In general, these models are smaller, with inventions such as MobileNet <cit.> and EfficientNet <cit.> that have specifically been optimised for edge devices. The long history of CNNs in this areas shows that tools including TensorRT, ONNX, and TensorFlow Lite have been made available with extensive optimisation techniques. Importantly, modern Graphical Processing Units (GPUs) and Application-Specific Integrated Circuits (ASICs) (such as Tensor Processing Units (TPUs) and Neural Processing Units (NPUs)) provide dedicated acceleration for convolution operations, thus making CNNs particularly efficient for deployment in real-world scenarios.

