Deep RAW Image Super-Resolution. A NTIRE 2024 Challenge Survey
Marcos V. Conde Florin-Alexandru Vasluianu Radu Timofte Jianxing Zhang Jia Li Fan Wang Xiaopeng Li Zikun Liu Hyunhee Park Sejun Song Changho Kim Zhijuan Huang Hongyuan Yu Cheng Wan Wending Xiang Jiamin Lin Hang Zhong Qiaosong Zhang Yue Sun Xuanwu Yin Kunlong Zuo Senyan Xu Siyuan Jiang Zhijing Sun Jiaying Zhu Liangyan Li Ke Chen Yunzhe Li Yimo Ning Guanhua Zhao Jun Chen Jinyang Yu Kele Xu Qisheng Xu Yong Dou


Synthesis of layered gold tellurides AuSbTe and Au_2Te_3 and their semiconducting and metallic behavior
    Daniel P. Shoemaker
    April 29, 2024
=======================================================================================================



* Marcos V. Conde († corresponding author, project lead), Florin-Alexandru Vasluianu, amd Radu Timofte are the challenge organizers, while the other authors participated in the challenge and survey. 

* University of Würzburg, CAIDAS & IFI, Computer Vision Lab.
 
NTIRE 2024 webpage: <https://cvlai.net/ntire/2024>.

Code: <https://github.com/mv-lab/AISP> 


This paper reviews the NTIRE 2024 RAW Image Super-Resolution Challenge, highlighting the proposed solutions and results. New methods for RAW Super-Resolution could be essential in modern Image Signal Processing (ISP) pipelines, however, this problem is not as explored as in the RGB domain. Th goal of this challenge is to upscale RAW Bayer images by 2x, considering unknown degradations such as noise and blur. In the challenge, a total of 230 participants registered, and 45 submitted results during thee challenge period. The performance of the top-5 submissions is reviewed and provided here as a gauge for the current state-of-the-art in RAW Image Super-Resolution.









§ INTRODUCTION


RAW Image Super-Resolution represents an active research direction, aiming at upscaling hardware-specific RAW image representations, while dealing with hardware characteristic properties, often depending on the technical  implementation of the camera product. The lack of standardization in camera Image Processing Signal (ISP) implementation induces plenty variety into the camera market segment, with plenty corrections made through image processing algorithms <cit.>, overcoming the hardware limitations of various devices.

The RAW image represents the discretized and quantized representation of the image signal. Of course, both the aforementioned operation depend on the sensor nature, with different sensor types, different spatial resolutions or color resolutions (the number of bits used for quantization). Naturally, portable camera devices, that are usually subject to extreme limitations in terms of size, power supply, and used optics are constricted in terms of achieved image quality, usually employing low color resolutions and spatial resolutions. However, these specific hardware implementations are currently leading the market in terms of new acquired devices, with cameras becoming ubiquitous and universal accessible. Therefore, the image information space is also characterized by a high availability of images corresponding to these devices, with complex ISP systems   <cit.> mapping the RAW image representation to a perceptually meaningful RGB counterpart. 

RAW image upscaling remains also relevant in the professional photography field. Given the strong development of Internet-based image transfer services, the drive for high resolution images is now higher than ever before. However, building a professional photography or videography setup is a difficult challenge, since the market is divided by various application-specific parameters, with the acquisition costs remaining the main limiting factor for a new investment. Thus, it is important to develop algorithms which are robust to the various hardware limitations affecting the low-cost amateur photography systems, matching the characteristics of the high resolution sensors used in highly professional applications. 






RAW information is extremely important, since the RAW data directly correlates with scene radiance, with the discretization and quantization being the only non linear operations affecting the naturally continuous radiance signal measure in a photography. This property of the RAW data is extremely beneficial in analyzing typical image degradations as acquisition noise or image blur.
Futher away from the sensor representation, ISP represents a sequence of highly non-linear operations, happening with irrevocable loss of information  <cit.>. This further complicates the image restoration task, with complex algorithms needing every achievable bit of signal based variance  <cit.>.

Considering all the aforementioned factors, RAW image processing poses significant advantages over the standard sRGB representation, with superior performance in a multitude of low-level imagery applications like image denoising <cit.>, deblurring <cit.>, exposure adjustment <cit.>, and image super-resolution <cit.>.





Due to the lack of standardization at the hardware implementation level, RAW images are characterized by vendor or even product specific properties, which are explainable only with private, with the photography products shielded by implementation patents or trade secrets. Coupled to the abundance of the standard sRGB representation, most of the existing high complexity image restoration algorithms <cit.> are specifically designed for compressed or uncompressed RGB image or video. 

As a sub-task of Image Restoration, cutting-edge Single Image Super-Resolution (SISR) algorithms <cit.> follow the same data specific, even if they are relying on deep convolutional networks or Transformer <cit.> architectures. 
One of the largest drawbacks characterizing these algorithms is represented by them being limited  by the quality of the data used for optimization. Various image restoration applications are characterized by extreme difficulties in acquiring real domain paired data <cit.>, driving the need for realistic and relevant data synthesis <cit.>.  Accurately modeling application specific degradations in the sRGB representation proves extremely difficult, given the fact that the highly nonlinear ISP characteristic shifts also the physical characteristics of the degradation appearance. This represents the main factor limiting the performance of these algorithms in real applications deployment, with the gap observable at the data level being difficult to overcome with model-free algorithms.

Therefore, studying RAW data becomes a crucial step in driving the general image restoration performance improvement trend. Consequently, RAW Image Super-Resolution applications can benefit from the increased variance signal, with algorithms robust to fine architectural properties of the involved devices. Developing highly effective algorithms like the ones well-established in the sRGB domain <cit.> can prove a step ahead in the development of high performance imagery applications, with cost effective devices. 

Thus, in this work we are presenting the solutions submitted for the NTIRE 2024 RAW Image Super Resolution Challenge. We are providing information regarding the challenge setup, with the task description and the challenge data properties characterizing the challenge dataset splits.  We are also listing information regarding the challenge participants, with their teams and affiliations. 

In <ref> we describe the challenge dataset and evaluation, and we discuss the overall results. 
In <ref> we provide detailed descriptions of the best solutions.








   









  
Related Computer Vision Challenges
Our challenge is one of the NTIRE 2024 Workshop [https://cvlai.net/ntire/2024/] associated challenges on: dense and non-homogeneous dehazing <cit.>, night photography rendering <cit.>, blind compressed image enhancement <cit.>, shadow removal <cit.>, efficient super resolution <cit.>, image super resolution (×4) <cit.>, light field image super-resolution <cit.>, stereo image super-resolution <cit.>, HR depth from images of specular and transparent surfaces <cit.>, bracketing image restoration and enhancement <cit.>, portrait quality assessment <cit.>, quality assessment for AI-generated content <cit.>, restore any image model (RAIM) in the wild <cit.>, RAW image super-resolution <cit.>, short-form UGC video quality assessment <cit.>, low light enhancement <cit.>.




§ NTIRE 2024 RAWSR CHALLENGE





 §.§ Dataset


The challenge dataset is based on BSRAW <cit.>. Following previous work <cit.>, we use images from the Adobe MIT5K dataset <cit.>, which includes images from multple Canon and Nikon DSLR cameras. 

The DSLR images are manually filtered to ensure diversity and natural properties (remove extremely dark or overexposed images), we also remove the blurry images (we only consider all-in-focus images).

The pre-processing is as follows: (i) we normalize all RAW images depending on their black level and bit-depth. (ii) we convert (“pack") the images into the well-known RGGB Bayer pattern (4-channels), which allows to apply the transformations and degradations without damaging the original color pattern information <cit.>.

Training: We provide the participants 1064 1024 × 1024 × 4 clean high-resolution (HR) RAW images. The LR degraded images can be generated on-line during training using the degradation pipeline proposed in BSRAW <cit.>. 

Such degradation pipeline considers different noise profiles, multiple blur kernels (PSFs) and a simple downsampling strategy to synthesize low-resolution (LR) RAW images. The participants can apply other augmentation techniques or expand the degradation pipeline to generate more realistic training data.



 §.§ Baselines


We use BSRAW <cit.> as the main baseline. The top performing challenge solutions improve the baseline performance, however, the neural networks are notably more complex in terms of design and computation.



 §.§ Results


We use three testing splits: (i) Validation, 40 1024px images using during the model development phase. (ii) Test 1MP, 200 images of 1024px resolution. (iii) The same 200 test images at full-resolution ≈12MP. The participants process the corresponding LR RAW images (512 × 512 × 4), and submit their results. Thus, the participants do not have access to the ground-truth images. 

We provide samples of the testing set in <ref>.

In <ref> we provide the challenge benchmark. Besides fidelity metrics such as PSNR and SSIM, we also provide relevant implementation details of each method. The methods can greatly improve the RAW images quality and resolution, even in the case of full-resolution 12MP images as output. 

We provide detailed visual comparisons in <ref>, <ref> and <ref>. All the proposed methods are able to increase the resolution and details of the RAW images while reducing blurriness and noise. Moreover, there are not detectable color artifacts.

We can conclude that (synthetic) RAW image super-resolution can be solved similarity to RAW denoising. However, more realistic downsampling remains an open challenge.






  
Acknowledgements
This work was partially supported by the Humboldt Foundation. We thank the NTIRE 2024 sponsors: Meta Reality Labs, OPPO, KuaiShou, Huawei and University of Würzburg (Computer Vision Lab).



§ CHALLENGE METHODS AND TEAMS








 §.§ Dual Stage RAW SR with Focal Pixel Loss










Team Samsung MX,SRC-B



Jianxing Zhang ^1,
Jia Li ^1,
Fan Wang ^1,
Xiaopeng Li ^1,
Zikun Liu ^1,
Hyunhee Park ^2,
Sejun Song ^2,
Changho Kim ^2





^1 Samsung Research China - Beijing (SRC-B)

^2 Samsung MX(Mobile eXperience) Business












Team Samsung MX,SRC-B is introducing a two-stage network for RAW Image Super Resolution. The solution is using the divide-and-conquer strategy, with the first stage tasked with recovering the image structure from the low resolution degraded RAW image, and the second stage aiming at recovering the maximum amount of details, offering a refined reconstruction. 

Moreover, the team is further extending existing methods for synthetic data generation, studying further into hardware specific RAW image degradations, proposing new definitions for the relevant device-specific noise profiles and new blur kernels aligning with typical real-world scenarios. They propose a randomized degradation model, simulating different interactions between the observed simulated defects. 

Finally, Team Samsung MX,SRC-B is proposing a novel Focal Pixel Loss, which was proven through the performance improvements during the model fine-tuning stage.


As shown in <ref>, the network structure mainly includes two stages. The first stage mainly draws on Restormer <cit.>, whose main role is to restore the main content of raw images. The second stage mainly uses a NAFNet <cit.> based design, whose main role is to restore more details on the basis of the first stage of recovery. 

The training procedure accounts for the dual stage design of the proposed method. The first stage of the training procedure follows the optimization of the parameters corresponding to the first stage of the model. In the second stage of the training procedure, the optimized parameters are frozen, with the parameters of the second stage starting being refined, for optimum specialization. The final estimation is then performed using both sets of optimized parameters (Restormer and NAFNet models).


one of the main details of the solution is the proposal of a novel training objective, based on the characteristics of the input data. The proposition of the Focal Loss (see <ref>) is a solution for the observed imbalance in terms of highly affected pixels ratio, given the non-unifor effect of the signal degradation function . Therefore, the Focal Pixel Loss (FPL) is introducing exponential penalties to those pixels characterized by a large signal shift.


    FPL(Î, I)=-D(Î, I)^γlog_10 D(Î, I)


In <ref>, D(., .) is a standard L-norm distance between the restored image Î and the reference image I,  and γ is an adjustable factor, controlling the strength of the penalty. 






  
Implementation details

The model is trained solely on the data provided by the challenge organizers. It only contains more than 1000 RAW images from various DSLR camera sensors. The dataset was augmented using standard image augmentation techniques and simulations of the image degradation pipeline. The training procedure is a dual stage operation, optimizing the model stages sequentially. 
The optimization technique used is  the AdamW <cit.> optimizer (β_1=0.5, β_2=0.999, weight decay 0.0001) with the cosine annealing strategy, where the learning rate gradually decreases from the initial learning rate 5×10^-5 to 1×10^-7 for 5×10^5 iterations. 
The model goes through a pre-train optimization phase, base don the L_1 loss. During the fune-tune phase, the objective is set to the  Focal Pixel Loss, with the initial learning rate being set to 5×10^-6. The training batch size is set to 4 and patch size is 384. Horizontal/vertical flipping and rotation are used for data augmentation. All experiments are conducted on A100 GPUs.





 §.§ EffectiveSR for RAW Images










Team XiaomiMMAI



Zhijuan Huang^1,
Hongyuan Yu^1,
Cheng Wan^2,
Wending Xiang,
Jiamin Lin^1,
Hang Zhong^1, 
Qiaosong Zhang^1, 
Yue Sun^1, 
Xuanwu Yin^1, 
Kunlong Zuo^1





^1Xiaomi Inc.
^2Georgia Institute of Technology







The solution proposed by Team XiaomiMMAI is a dual branch network based on HAT <cit.>, adopting re-parameterization <cit.> during training, using the additional parameters to fully exploit the potential of the method. They are introducing, a task-by-task and step-by-step training method for RAW Image Super-Resolution to simultaneously address three tasks: denoising, deblurring, and 2× Super Resolution. 



To address the limitation given by the low number of samples offered for training, Team XiaomiMMAI converts the RAW images into an RGB images and performs combined data enhancement on the set of produced RGB image, using random rotations, flips, color changes, brightness changes, random blur, etc.. Then, the set of enhanced RGB images are translated back to the RGGB RAW domain, with the processed data used to train <cit.> the proposed solution. 

The model proposed by Team XiaomiMMAI is inspired by HAT <cit.>, with the architecture being optimized for the RAW Image Super Resolution task. The optimized dual-branch network structure (DB-HAT) is shown in <ref>. The introduced Step-by-step and task-by-task training method for RAWISR further enhances the performance level achieved by their solution.

Step-by-step: To accelerate training and achieve good performance,  Team XiaomiMMAI adopted a strategy where each sub-task, including the final joint optimization, is trained based on a pyramid image representation. Initially, the model is trained on small scale images (64×64), gradually increasing the resolution of the image patches to 128×128 and 256×256.

Task-by-task: Team XiaomiMMAI divided RAWISR into three sub-tasks: denoising, deblurring, and 2x SR. Initially, they start by training for RAW image denoising, followed by the connected tasks of deblurring and 2× Super Resolution. Finally, a joint optimization procedure is applied for entire network to produce the final estimator.

In the process of training denosing and deblurring, Team XiaomiMMAI used the RepConv re-parameterization technique on the final stage of the proposed  DB-HAT, to improve the visual image quality of the task. The reparameterizable convolution block (RepConv) is shown in Fig <ref>. 







  
Implementation details

The dataset used for three sub-task training consists of 1000+ RAWs, and the data augmentation methods can refer to the previous section for details. In the final stage of joint optimization of the entire network, Team XiaomiMMAI used the provided 1000+ dataset instead of the augmented dataset. The learning rate is initialized at 4 × 10^-4 and decays according to the CosineAnnealing strategy during the training of three sub-tasks. The network undergoes training for a total of 2x10^5 iterations, with the L2 loss function being minimized as the trainign objective of the Adam optimizer.

Subsequently, finetuning is executed for two iterations, using the L2 loss and SSIM loss functions, with an initial learning rate of 5 × 10^-5 for 2x10^5 iterations. All experiments are conducted with the PyTorch 2.0 framework on 8 A100 GPUs. 







 §.§ RBSFormer: Enhanced Transformer Network for Raw Image Super-Resolution










Team USTC604



Senyan Xu,
Siyuan Jiang,
Zhijing Sun,
Jiaying Zhu




University of Science and Technology of China




Contact: <a804235820@gmail.com>








Team USTC604 proposed a transformer framework for raw image super-resolution, with a design based on the transformer block proposed in Restormer<cit.> (see <ref>), solution that excels in capturing long-range pixel interactions by applying self-attention across channels. 

The solution used the data provided in the NTIRE 2024 RAW Image Super Resolution challenges, with the degradation pipeline described in <cit.>. 

For a 4-channel RGGB RAW image patch of size 224×224, the computational cost of the model proposed by Team USTC604 amounts to 14.6 GFLOPS, being characterized by a number of 3.31 trainable parameters. On a consumer-grade gaming GPU, the NVIDIA RTX4090Ti, the forward pass needed of a full resolution image estimation needs 650 ms, following the limitations of the used backbone, as one of the computationally expensive solutions proposed in the image restoration field.  

The software characteristic to the performed experiments is based on the PyTorch 1.8 framework, with the experiments being performed on NVIDIA RTX4090Ti devices.  The trainign procedure is based on the Adam optimizer with the decay parameters parameters β_1 = 0.9 and β_2 = 0.99. The initial learning rate is 3 × 10^-4 and changes with Cosine Annealing scheme to 1 × 10^-7, with the training procedure covering 120K iterations in a time-frame of around 10 hours. We refer the reader to the author's paper RBSFormer <cit.> for more details.








 §.§ SwinFSR Raw Image Super Resolution










Team McMaster



Liangyan Li,
Ke Chen,
Yunzhe Li,
Yimo Ning, 
Guanhua Zhao,
Jun Chen





McMaster University




Contact: <lil61@mcmaster.ca>












Team McMaster proposed an algorithm that considers multiple acquisition sensors, accounting for various image signal degradations induced by hardware limitations. The model is trained and learned directly from the 4-channel RAW data with an enhanced degradation pipeline. With a broader variance of noise during the degradation process, the solution demonstrates increased robustness, efficiently producing high-quality images from degraded inputs, thereby enhancing overall performance on the official datasets. 

Their approach works directly with 4-channel RGGB RAW images after a designed degradation process. The architecture is a hybrid model which integrates SwinFSR <cit.> with simple CNN layers. The model was trained and validated only on the official datasets <cit.>. 


The key point for end-to-end raw image super resolution is the degradation process of original high resolution images.

    y=(x  ⊗ k)↓ s + n


As a general degradation model, <ref> assumes that the low resolution(LR) image y is corresponding to a High Resolution(HR) image x, on which the blur kernel k is applied, then downsampled by the scale factor s.  Finally, the result is affected by the hardware-specific noise sources, modeled by the profile of the noise factor n.     



The team utilized the noise model, blur kernel,  and degradation model as demonstrated in BSRAW <cit.>.

Their approach of adding noise is inspired by the strategy deployed in DiT <cit.> that gradually adds Gaussian noise as a form of degradation in the forward process. Diffusion models<cit.> have been proposed for various imaging tasks, including image super-resolution, demonstrating their capability for end-to-end training to transform pure Gaussian noise into meaningful data representations. They investigate the applicability of gradual magnitude Gaussian noise, as utilized in diffusion models, for addressing the degradation process inherent in Raw Image Super-Resolution tasks. Team McMaster adopted the additive noise model from <cit.>, exposing the input RAW images to noise using the forward diffusion definition described in <cit.>. In the forward diffusion process, the input data undergoes a degradation process the iterative addition of Gaussian noise across 1000 discrete steps. In the backward denoising process of DiT, the model is optimized for the forward diffusion process, maximizing the data likelihood via the variational lower bound.



The architectural configuration of the Team McMaster proposed model is depicted in <ref>.

In the proposed method, a SwinFSR-based design <cit.> performs RAW domain image feature extraction, combined with feature upsampling, matching the size of high-resolution images via a complex convolution operator. SwinFSR builds on the success of SwinIR <cit.>, with an additional data modality given by the Frequency Domain Knowledge, through the FFT image representation <cit.>. This proves to be a superior strategy, combining spatial and spectral features as a way to balance the local information of the spatial domain, and the global information accessed through the spectral representation. It introduces a novel cross-attention module for efficient information exchange between the two modalities and adapts to rectangular input patches for flexibility. 



For the proposed model, only the feature extraction branch of SwinFSR is deployed.



  
Implementation details
The solution was optimized solely on the NTIRE 2024 official challenge data <cit.>, using the proposed Development Phase submission set for validation. This dataset contains 1064 4-channel DSLR-specific RGGB RAW images for training, and an additional 40 raw images set for validation. The images were pre-processed, applying white-black level correction, then being normalized to the unit interval. Since the degraded low-resolution raw images are characterized by low quality, with a considerable level of details being lost, a data augmentation technique is applied, to improve the training procedure in terms of stability, convergence, and the achieved performance level. The strategy combines simple horizontal or vertical flips with channel shifts and mixup augmentations. The training objective is based on the  L_1 loss.




 §.§ Spatially-Adaptive Feature Modulation for RAW Super-Resolution










Team NUDT RSR



Jinyang Yu,
Kele Xu,
Qisheng Xu,
Yong Dou




National University of Defense Technology, Computer Dept., Changsha, China




Contact: <a804235820@gmail.com>









The solution proposed by Team NUDT RSR addresses three major components of the proposed challenge, extending on the degradation pipeline, model design, and model supervision, achieving a significant performance level in terms of restoration fidelity. 

 For the image signal degradation pipeline, the considered degradations include diverse blur kernels, exposure defects, image downsampling, finally coupled with a noise model characteristic to real-world raw image data. 
 
 Following <cit.>, the proposed solution is benefitting from the amplitude and phase components, added to the SAFMN <cit.> backbone. the model is fusing frequency domain and spatial domain information, for global-local level feature mixing.
 
 Moreover, the knowledge distillation is deployed to the described solution, using a NAFNet <cit.> teacher, with multiple complexity level feature supervision. Finally, we apply a progressive training strategy, gradually increasing the patch size at each stage to accommodate larger test inputs. 




Degradation pipeline:
    Inspired by <cit.> and <cit.>, in order to enable the model to learn real degradation information, the Bayer pattern RAW images are cropped, following by degrading the RAW signal in a sequence of operations composed of multiple blurring operations, exposure compensations, downsampling, and hardware specific added noise.

    The first step is represented by a blurring operator, based on randomly generated Gaussian blur, generalized Gaussian blur, with a plateau-shaped distribution, and their an-isotropic version. The augmented PSF kernels provided by starter kit are also considered. All kernel sizes are ranging from 7×7 to 25×25.

    Then, the pipeline continues with linear adjustment for image exposure.  As <cit.> discussed, to simulate the artifacts caused by underexposure and overexposure, the pipeline implements exposure adjustment by linearly scaling the image. The adjustment factor is tuned to the [-0.25, 0.25] interval, applied in the unit interval normalized images.

    Next, the image suffers further downsampling, considering different downsampling kernels, including bicubic interpolation, bilinear interpolation, and an average-pooling operator. To build multi-scale training pairs, the input image is either upsampled or downsampled to a random size first, and rescaled back to half of the original size as for 2× super-resolution tasks.

    On the downscaled image, heteroscedastic Gaussian noise <cit.> is then applied, followed by the practical shot-read noise <cit.> for different exposure levels. An image with higher exposure factor in step 2 is more likely to get noised by heteroscedastic noise, and the shot-read noise for low-light images.

    The last step of the degradation pipeline is a second blurring operator. To expand the degradation space like the high-order degradation model <cit.>, a random operation based on a set of second blurring kernels (same kernels considered in the first step), characterized by smaller standard deviations, is applied in the final stage. 








  
Network
  According to <cit.>, the Fourier spectrogram of an image has similar amplitude to its downsampled one's, while the phase is related to the noise observed in the acquired image signal. Although it is designed for low light image enhancement tasks, Team NUDT RS started with the observation that blind RAW Image super-resolution can also benefit from refining these two components of an low resolution image.
  
  Thus, the Team NUDT RS proposed model is represented in  <ref>, where the input image is encoded by a 3 × 3 convolution layer for shallow feature extraction, and the FFM_FFT blocks are used for deep feature extraction. Following <cit.>, the main blocks are divided into spatial branches and Fourier branches. In each block, the input is simultaneously sent to both branches, then the processed features of the branches are fused. After a residual connection, a final pixel-reshuffle operator is used to upscale the feqature set to the resolution of the reference image <cit.>.

  
  In the spatial branch, the cross domain communication is performed through efficient Feature Mixing Module (FMM) blocks of SAFMN <cit.>. A FMM block consists of a SAFM block and a CCM block. SAFM splits the channels into different parts, fuses their features at different scale levels and obtain attention map after GELU activation. Then, the original input multiplies the attention map. The CCM block consists of a 3 × 3 convolution layer and a 1 × 1 convolution layer, which works as a channel mixer to capture local context information.
  
  In the Fourier branch, an image is transformed to frequency map by Fast Fourier Transform (FFT) operator to get amplitude and phase component, which are later processed by two 1 × 1 convolution layers with GELU activations respectively.  Next, these refined components are combined to a new frequency map, and the inverse FFT its used to transfer back to the spatial domain. Features from different branches are concatenated and fused by another 1 × 1 convolution. The proposed model uses  8 such blocks with width 36. To avoid expensive computations on high dimensions, the SFT layers in the final reconstruction stage are removed <cit.>. This results in an efficient estimator, with the total parameter count being 272.068 K for the proposed solution. The team presents an ablation study in <ref>.





  
Training strategy
A two-stage optimizing strategy is applied. All the training is based on the development dataset and there are no external datasets used.
  
Firstly, a large version of NAFNet <cit.> model is trained as a teacher network, with increased complexity degradations on patch size 128.  Then this NAFNET model is used to apply knowledge distillation, as part of the optimization technique used for the proposed solution. To reduce the performed computations, the knowledge  does not rely on distances computed between multi-level feature sets, but rather on statistics defining the feature see <cit.>.
  
Secondly, to accommodate the high resolution test images, the student model is progressively finetuned on increased resolution patches. The training patch size increases from 128, 256, 352 to 448.

The optimization objective is based on the L! distance simultaneously applied in the spatial and Fourier domains <cit.>. The total loss is defined in <ref>.


    ℒ_p = I_1 - I_2_1


    ℒ_f = ℱ(I_1) - ℱ(I_2)_1


    ℒ_kd = 1/|D|∑_i∈ D𝒢(𝒩_i(I_1)) - 𝒢(𝒩_i(I_2)) _c


    ℒ_total = ℒ_p + λℒ_f + μℒ_kd


I_1, I_2 denotes the restored image prediction and the corresponding reference image. ℱ(·) is the FFT operator, 𝒩_i(·) is the feature extracting operator determined by the network, where i∈ D is a set of the intermediate layers and 𝒢(·) is defined to the square of the channel-wise mean of intermediate features. The ·_c represents Charbonnier loss. λ and μ are used to control the weights of different components. The finetuning stage only uses the ℒ_p and ℒ_f loss terms.






  
Implementation details
The experiments are based on the Pytorch framework for implementation. The NAFNet teacher model uses a width 64, with 2, 2, 4, 8 encoder blocks and 2, 2, 2, 2 decoder blocks at each stage. The middle block num is set to 12. The teacher module is trained for 600000 iterations. The learning rate is set to 2e-4, and halves for every 100000 steps after 300000 steps. 
The training procedure applied for the teacher network uses only the ℒ_p loss. For knowledge distillation, the procedure starts with an initial learning rate warm-up stage, lasting for 30000 iterations. Then, a cosine annealing stage is applied, decreasing the learning rate to 1e-6. 

The FFT L1 loss is balanced with a weight λ=0.05, and the distillation loss with μ=0.1. For the progressive finetuning, each training stage for the proposed model lasts for 200,000 iterations with learning rate 5e-5 that is reduced to 1e-6 with a cosine scheduler. Alongside the augmentation involved in the degradation pipeline, some traditional data augmentation methods, such as random cropping, flipping, and rotation are deployes to further diversify the training dataset. The testing phase deploys the geometric self-ensemble strategy <cit.>, which averages 8 outputs corresponding to 8 augmented versions of the input image. All the experiments are conducted on a NVIDIA GeForce RTX 2080Ti GPU, using the AdamW optimizer with β_1=0.99, β_2=0.9. 




















ieeenat_fullname



