A Dynamic Kernel Prior Model for Unsupervised Blind Image Super-Resolution
Zhixiong Yang Jingyuan Xia Shengxi Li Xinghua Huang Shuanghui Zhang Zhen Liu Yaowen Fu Yongxiang Liu


Characterizing the Age of Information with 
Multiple Coexisting Data Streams

    
Yoshiaki Inoue and Michel Mandjes
This work was supported in part by JSPS KAKENHI Grant Number
21H03399 and the NWO Gravitation project NETWORKS under grant agreement no. 024.002.003.

Y. Inoue is with 
Department of Information and Communications Technology, 
Graduate School of Engineering, Osaka University, Suita 565-0871, Japan 
(e-mail: yoshiaki@comm.eng.osaka-u.ac.jp).

M. Mandjes is with Mathematical Institute, Leiden University, P.O. Box 9512, 2300 RA Leiden, 
The Netherlands (e-mail: m.r.h.mandjes@math.leidenuniv.nl). He is also affiliated with Korteweg-de Vries Institute for Mathematics, University of Amsterdam, Amsterdam, The Netherlands; Eurandom, Eindhoven University of Technology, Eindhoven, The Netherlands; Amsterdam Business School, Faculty of Economics and Business, University of Amsterdam, Amsterdam, The Netherlands.


    
====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

Zhixiong Yang and Jingyuan Xia contributed equally to this work (*Corresponding author: Jingyuan Xia).
This work is supported by National Natural Science Foundation of China, projects 61921001, 62131020, 62322121 and 62171448, and the NSFDYS of Hunan 2022J110067.


Deep learning-based methods have achieved significant successes on solving the blind super-resolution (BSR) problem. However, most of them request supervised pre-training on labelled datasets.
This paper proposes an unsupervised kernel estimation model, named dynamic kernel prior (DKP), to realize an unsupervised and pre-training-free learning-based algorithm for solving the BSR problem. 
DKP can adaptively learn dynamic kernel priors to realize real-time kernel estimation, and thereby enables superior HR image restoration performances. 
This is achieved by a Markov chain Monte Carlo sampling process on random kernel distributions. 
The learned kernel prior is then assigned to optimize a blur kernel estimation network, which entails a network-based Langevin dynamic optimization strategy. 
These two techniques ensure the accuracy of the kernel estimation.
DKP can be easily used to replace the kernel estimation models in the existing methods, such as Double-DIP and FKP-DIP, or be added to the off-the-shelf image restoration model, such as diffusion model. 
In this paper, we incorporate our DKP model with DIP and diffusion model, referring to DIP-DKP and Diff-DKP, for validations. 
Extensive simulations on Gaussian and motion kernel scenarios demonstrate that the proposed DKP model can significantly improve the kernel estimation with comparable runtime and memory usage, leading to state-of-the-art BSR results. 
The code is available at <https://github.com/XYLGroup/DKP>.












    


§ INTRODUCTION



Deep learning provides a new avenue for solving the blind super-resolution (BSR) problem, which aims to reconstruct high-resolution (HR) images from the low-resolution (LR) observations with unknown blur kernels, and is known to be highly non-convex and ill-posed.
To alleviate the non-convexity and ill-posedness, 
most of learning-based BSR methods 

incorporate image priors via supervised learning based on paired LR-HR samples. However, pre-defined labeled training datasets are expensive, time-consuming, and even not feasible in specific scenarios, such as for high speed targets (e.g., satellites, aircraft) and medical images (e.g., beating heart). 

Thus, unsupervised learning-based solutions are highly demanded for BSR problem.


The existing BSR methods can be roughly divided into model-based and learning-based strategies in terms of the priors adopted to provide performance guarantee.
Model-based approaches <cit.> typically adopt hand-designed and explicit constraints as regularizations on image properties, or expert knowledge of the blur kernel.
Meanwhile, learning-based BSR methods <cit.> aim to train an end-to-end network with paired LR-HR image samples to leverage data priors for boosting performances. 
However, these methods highly demand the data and need to undergo throughing pre-training before applications, leading to limited generalization ability towards varying blur kernels. 
To alleviate this issue, quite a few methods <cit.> substitute the cumbersome training in advance by a well-trained diffusion model with significantly less fine-tuning samples in an off-the-shelf fashion. 
On the other side, a slice of works <cit.> propose to replace the HR image data priors by kernel priors, which are more substantial, economical and efficient to be trained. 
However, both of these advances are underlying the supervised learning scheme with necessity of training on labeled datasets, still hindering the flexibility and generalization ability towards the BSR tasks with different kernels and unknown HR ground truths. 










In this paper, we propose a dynamic kernel prior (DKP) generation model that can be plug-in with the majority of image restoration (IR) models, to solve BSR problem in an unsupervised way.
The proposed DKP model consists of two modules: random kernel sampling (RKS) module and prior kernel estimation (PKE) module. 
In the RKS module, a Markov Chain Monte Carlo (MCMC) sampling strategy on  kernel distributions iteratively generates random kernels as kernel priors, which are then assigned to the PKE module.
The PKE module is employed to estimate the blur kernel with respect to the kernel prior generated from the RKS module, the observed LR input and estimated HR image from the IR model. 
The estimated blur kernel is then assigned to an adopted IR model for the HR image restoration.
Along with the alternative solving processes, the MCMC process in RKS module converges to a desired kernel distribution with respect to the LR observation and the estimated HR image to guarantee a rational kernel prior. 
Meanwhile, a network-based Langevin dynamics (NLD) paradigm is proposed to optimize the kernel estimator in our PKE module with respect to the RKS output kernel prior and the data consistency based on the LR image reconstruction error.
The RKS module realizes an unsupervised kernel prior learning. 
The PKE module achieves promising kernel estimation via the NLD update scheme, which further alleviates the non-convexity and ill-posedness in the view of optimization strategy. 
In this way, the DKP model is capable of providing the plug-and-play kernel estimation without training in advance on paired LR-HR samples, and is flexible to be applied to the existing IR models for solving the BSR problem.











Two applications are proposed to validate the feasibility and performance of our DKP model: deep image prior (DIP) <cit.> and diffusion model <cit.> adopted as the IR model, referring to DIP-DKP and Diff-DKP, respectively.
For the DIP-DKP, we simultaneously optimize the parameters of DIP and DKP models from scratch during the alternative solution process. 
For the Diff-DKP, the adopted diffusion model is off-the-shelf from <cit.> and is applied as the fixed HR image restorer. The DKP model is optimized from scratch as well. 
Extensive simulation results show that the DIP-DKP achieves comparable performance than the existing methods, while the Diff-DKP achieves the state-of-the-art performance in both of Gaussian and motion kernel scenarios.
The main contributions are summarized as follows:















  * The RKS module is proposed to generate a rational kernel prior from the MCMC sampling on random kernel distributions. This way, an unsupervised kernel prior learning is achieved to substitute the pre-training phase.

  * In PKE module, the NLD is proposed to optimize the kernel estimator, ensuring good convergence and concise estimation of the blur kernel from the perspective of optimization strategy.

  * The proposed DKP model enjoys the ease use on the popular IR models without the necessity of pre-training/re-training towards different scenarios. The two applications, i.e., DIP-DKP and Diff-DKP, validate the state-of-the-art performance and excellent flexibility of our DKP model.




§ RELATED WORK








To alleviate the non-convexity and ill-posedness, early model-based approaches <cit.> typically construct image priors in explicit formulations, such as the total variation (TV) <cit.>, gradient profile <cit.>, hyper-Laplacian <cit.> and sparsity <cit.>. 
In contrast, learning-based methods <cit.> typically train an end-to-end network on labelled image samples to incorporate data priors. 
Wang et al. <cit.> proposed a CNN-based deep network with degradation feature representation module to learn image degradation feature from supervised training on paired LR-HR images.
Li et al. <cit.> proposed a transformer network to learn multi-scale image feature via self-attention mechanisms. To reduce the high training costs of time and data, recent advances <cit.> are proposed to solve BSR problem by an off-the-shelf diffusion model <cit.>. 
Lin et al. <cit.> proposed to partially fine-tune the parameters of diffusion model with significantly less labeled images. 
Wang et al. <cit.> further formulated a diffusion-based BSR algorithm that iteratively solves super-resolution tasks with the given kernel without re-training. 
Different from the end-to-end models that are trained on paired image samples, recent methods tend to resolve BSR problem via pre-training on kernel datasets <cit.> or pre-defined kernel priors <cit.>. An alternative framework between the kernel estimation and image restoration is typically adopted in these methods <cit.>, such as double-deep image prior (Double-DIP) <cit.>.
On the basis of this framework, Liang et al. <cit.> established a flow-based kernel prior (FKP) network that is pre-trained on labeled kernels to enroll kernel priors while the HR image is estimated by DIP network in an online fashion. 
Yue et al. <cit.> proposed a hand-crafted kernel prior model to improve the robustness towards the Gaussian kernel scenario. 
Despite the fact that these methods approximately bring down the data requirements and training costs, the necessity of training in advances or hand-crafted design still limits the flexibility and generalization ability towards the varying kernel scenarios (Gaussian and motion) without ground truths. 

































§ DYNAMIC KERNEL PRIOR (DKP)




Problem Formulation.
The degradation model of BSR problem is commonly expressed as follows,

    y=(x⊗k)↓ _s+n,
 
where y denotes the LR image, x denotes the HR image, ⊗ indicates the convolution operation, ↓ _s denotes the down-sampling operation with scale factor s, and k denotes the blur kernel.

The BSR problem (<ref>) can be formulated as a maximum a posteriori (MAP) problem:

    x,kmax  p_(y|x,k) p_(x) p_(k),

where p_(y|x,k) denotes the likelihood of the observed LR image y, p(x) and p(k) are the HR image and kernel priors, respectively.
Image priors <cit.> have been well-designed and fully-studied in the past decade.
In contrast, researches on kernel priors p(k) are in the ascendant, as kernel samples are less expensive to obtain and the training phase is more efficient <cit.>.



In this paper, we propose the DKP model, which comprises two modules: RKS and PKE. The RKS module is employed to generate rational kernel priors, which are assigned to the PKE module to support the estimation of blur kernel. Let t=1,2,⋯,T denote the alternative iterations among these two modules and the adopted IR model, k^t and x^t denote the estimated blur kernel and HR image at the t^th iteration, respectively. The details of DKP model is given below.






RKS module.
At the t^th  iteration, the RKS module plays the key role of generating a rational kernel prior k^t_p from the MCMC simulation. The overview diagram is shown in Fig. <ref>. 




Let p(k_r|Σ_r) denotes that the random kernel k_r is conditioned by the latent variable Σ_r, in which  p(Σ_r) determines the category of blur kernel.
Then the distribution of the kernel prior k^t_p can be formulated as

    p(k^t_p) = ∫_Σ_rp(k_r|Σ_r)p(Σ_r)dΣ_r.

Here, Σ_r is the parameter of kernel (e.g., the variance of Gaussian kernel or the length of motion kernel).
It is not easy to sample all the possible Σ_r, and therefore, we convert (<ref>) into the Monte Carlo simulation in the following form:


    p(k^t_p) ≈∑^L_l=1 p(k_r^l|Σ_r^l)Σ_r^l,

where l denotes the index of the Monte Carlo sampling, Σ_r^l denotes the l^th sampled latent variable, k_r^l is the l^th sampled kernel, conditioned on the Σ_r^l.





















To ensure the rationality of randomly generated kernels towards the BSR problem, as well as the optimization during the iterations, the MCMC simulation is proposed as follows,

    p(k^t_p|x^t-1,y)    ≈∑^L_l=1 p(k_r^l|x^t-1,y) p(k_r^l| Σ_r^l)Σ_r^l,

where p(k_r^l|x^t-1,y) denotes the kernel weight ω^l, that is conditioned on the observed LR image y and the estimated HR image x^t-1 with respect to the  MCMC loss ℒ_MCMC in the following form

    ω^l = p(k_r^l|x^t-1,y) ∝1/ℒ^l_MCMC,

where 

    ℒ^l_MCMC = y-(x^t-1⊗k_r^l)↓ _s_F^2 + δ,


δ is the noise to prevent ℒ^l_MCMC=0. 

In this way, k^t_p can be formulated as

    k^t_p = 1/L∑_l=1^L ω^l k_r^l.

The obtained k^t_p is then assigned to the PKE module as a rational kernel prior, which will be introduced next.

We note that the obtained kernel prior k^t_p is an expectation of L times sampling according to (<ref>). 
The number of the sampling times L plays the role of annealing/tempering in MCMC simulations as a hyper-parameter. Details of the tuning on  L will be given in Section <ref>.














PKE module.
In our DKP model, the PKE module is employed to estimate the blur kernel by a lightweight network G_k with parameters ϕ_k as follows 

    k^t = G_k(ϕ^t_k).

The network G_k takes a fixed noise that is randomly initialized as input, and we neglect it for demonstration convenience as ϕ^t_k are the main variables. 

This kernel estimator G_k is optimized in the NLD paradigm with respect to the data-consistency term and kernel prior term, as shown in Fig. <ref>. 
The data-consistency term is computed by the LR image reconstruction error, which is given by 

    log p_( ϕ^t-1_k|x^t-1,y) = - y-(x^t-1⊗G_k(ϕ^t-1_k))↓ _s_F^2.

The kernel prior term is computed based on the difference between the network-estimated kernel G_k(ϕ^t-1_k) and the random-sampled kernel k^t_p from the RKS module as follows,

    log p_(ϕ^t-1_k|k^t_p) = - G_k(ϕ^t-1_k) - k^t_p_F^2.

By combining (<ref>) and (<ref>), the network parameters ϕ^t-1_k can be updated as follows,

    ϕ^t_k = ϕ^t-1_k + δ^2/2   ∂log p_( ϕ^t-1_k|x^t-1,y)/∂ϕ^t-1_k
              + δ∂log p_(ϕ^t-1_k|k^t_p)/∂ϕ^t-1_k,

where the second term is the data-consistency update,
the third term is the additional update based on the random kernel k^t_p.




It has been proved to be effective that the random noise-based disturbation can prevent being trapped into bad local modes for the variable update in Langevin dynamics <cit.>. 
More details of Langevin dynamics refer to the supplementary material.
At this stage, the random kernel sample from the RKS module can be regarded as the random “noise" for the ϕ^t-1_k update. 
Eq. (<ref>) can be reformulated as follows,

    ϕ^t_k = ϕ^t-1_k + δ^2/2   ∂log p_( ϕ^t-1_k|x^t-1,y)/∂ϕ^t-1_k + ζ^t-1_ϕ_k,

where ζ^t-1_ϕ_k = ∂log p_(ϕ^t-1_k|k^t_p)/∂ϕ^t-1_k denotes the parameters correlated Langevin dynamics disturbation.
























































The pipeline of our DKP at the t^th iteration is given in Algorithm <ref>.
The whole DKP model is implemented in a plug-and-play style, in which training in advance is not required. 
Besides, the random kernels from the RKS module are self-adaptively sampled through the MCMC simulation, without the need of labeled training data. We should also note that the DKP model only brings neglectable run-time and memory cost in applications, as the adopted network G_k is typically lightweight. This leads to high flexibility and low computational complexity.
These three merits promise our DKP the convenience of being applied to the existing image restoration approaches, including the untrained DIP model and off-the-shelf pre-trained diffusion model, which will be detailed in the next section. 



























































§ DKP-BASED BSR METHODS









 §.§ Pipeline

The overview of the proposed DKP-based BSR method is illustrated in Fig. <ref>.
The DKP model (gray box), including RKS module (blue box), PKE module (lilac box), and IR model (red box) alternatively optimize the blur kernel and refine the HR image, respectively.


For each iteration, the estimated HR image x^t-1 and LR image y are first fed to RKS module f_RKS to generate kernel prior




    k_p^t = f_RKS(x^t-1,y),

where x^t-1 denotes the estimated HR image from the last IR model output.
Then, the kernel prior k_p^t will be assigned to the PKE module f_PKE, which estimates kernel as follows,

    k^t = f_PKE(x^t-1,y,k_p^t),

where k^t is the estimated kernel at the t^th kernel estimation iteration, which will be assigned to the IR model.
The t^th HR image x^t can be estimated by the IR model as follows

    x^t = f_IR(x^t-1,y,k^t),

where f_IR denotes the adopted IR model.
In this paper, two representative IR models, DIP <cit.> and diffusion model <cit.> , are applied to evaluate the DKP-based BSR solutions, referring to DIP-DKP and Diff-DKP, which are introduced in the sequel.




































 §.§ The proposed DIP-DKP


DIP-based Image Restoration.
DIP <cit.> is designed for capturing low-level image statistics, and estimates HR image x=G_x(z_x,ϕ_x) from a fixed random noise input z_x (we omit z_x in the rest of this paper for demonstration convenience). A typical formulation of DIP-based BSR methods <cit.> is given as follows 





 ϕ^*_x, ϕ^*_k = ϕ_x, ϕ_kmin y- 

                        (G_x(ϕ_x)⊗G_k(ϕ_k))↓_s_F^2, 

x^* = G_x(ϕ^*_x), k^* = G_k(ϕ^*_k). 








Double-DIP <cit.> and FKP-DIP <cit.> have exploited the effectiveness towards the BSR problem. However, the kernel prior of G_k(ϕ^*_k) either adopts the untrained network with limited performances on the kernel estimation <cit.>, or pre-trained kernel network, referring to FKP <cit.>, that requests supervised training in advance. 
As shall be shown in experiments, pre-trained networks do not perform well to generate reasonable kernel estimations when the kernel categories vary.  











Proposed DIP-DKP.
We replace the untrained or pre-trained networks for kernel priors in the existing DIP-based alternative framework by the proposed DKP model, which we refer to as DIP-DKP.
The objective of our proposed DIP-DKP can be formulated as follows,






 ϕ^*_x, ϕ^*_DKP = ϕ_x, ϕ_DKPmin y- (G_DKP(ϕ_DKP) ⊗

          G_x(ϕ_x))↓_s_F^2 + G_DKP(ϕ_DKP) - k_p)_F^2, 

x^* = G_x(ϕ^*_x), k^* = G_DKP(ϕ^*_DKP), 

where G_DKP(ϕ_DKP) is the kernel network of the proposed DKP model.







The overall solution procedure of the proposed DIP-DKP is given in Algorithm <ref>.



At each t^th iteration, the kernel k^t is estimated in the DKP-based kernel estimation stage and then is assigned to the DIP model for HR image restoration in the forward propagation. 
In the back propagation, the parameters of DIP and DKP, i.e., ϕ_x and ϕ_DKP, are updated while solving the BSR problem via an unsupervised inference.
With DKP, DIP-DKP realizes an adaptive kernel learning along the convergence trajectory of the BSR objective function, enabling accurate and dynamic kernel estimation. 
Therefore, without expensive labeled data and long training time in advance, DIP-DKP can estimate HR image and blur kernel simultaneously in a plug-and-play style. 
















 §.§ Diff-DKP


Original DDPM Inference Process.
Denoising diffusion probabilistic models (DDPM) <cit.> defines a T-step forward process to add noise to data and a T-step reverse process to restore desired data from the noise. When an off-the-shelf DDPM S_θ is applied to solve image restoration problem, the reverse process is implemented as inference process to estimate the high quality image as follows,

  x_0|t = 1/√(α^t)(x_t-S_θ(x_t,t)√(1-α^t)), 

x_t-1 ∼p(x_t-1|x_t,x_0|t), 

where x_0|t denotes the estimated HR image x_0 at the t^th step, and α^t is the hyper-parameter. To ensure that HR images x_0 ∼ q(x) can be reconstructed from random noise x_T ∼𝒩(0, I), the existing methods typically re-train <cit.> or fine-tune <cit.> the DDPM model  via supervised learning on LR-HR datasets, or provide ground truth kernel <cit.> to enroll task-specific knowledge for convergence guarantee. However, the performance of DDPM is unstable, even when trained by a large number of labeled dataset.  
















































Proposed Diff-DKP.
The instability of DDPM mainly comes from the training process that involves multiple image processing tasks.
In this case, the off-the-shelf diffusion model cannot concentrate on BSR objective, thus leading to image distortion and content mismatch.
To alleviate this issue, the proposed Diff-DKP incorporates the DKP model to provide task-specific data-consistency knowledge on the basis of the vanilla DDPM reverse iterations.   

Specifically, an external DKP incorporated data consistency refinement of x_0|t is inserted between (<ref>) and (<ref>), given by 

    x̂_0|t = x_0|t + γ^t_x∂log p_( x_0|t|y,k^t)/∂x_0|t,

where γ^t_x is the update step, and

    log p_( x_0|t|y,k^t) = - y-(x_0|t⊗k^t)↓ _s_F^2,

which enables the inference process to converge to the right direction along with the data-consistent solution. 





The overview of the Diff-DKP algorithm is presented in Algorithm <ref>.
Let t=T,T-1,…,1 denote the index of the diffusion reverse step. 
At each step, the diffusion model first estimates the x_0|t.
Then, 
the DKP model adaptively generates kernel prior with respect to the latest x_0|t, while x_0|t is further updated with respect to the data consistency Eq. (<ref>), thus, ensuring the inference process is underlying the BSR objective. 
It is noteworthy to point out that the parameters of the diffusion model are fixed and only the parameters of lightweight kernel estimator network are optimized in the inference process.  

In this way, the off-the-shelf diffusion model plays the role of HR image estimator, while the estimated HR image is further refined by the BSR task specific prior knowledge, referring to Eq. (<ref>). Different from those methods that incorporate prior knowledge of BSR task via supervised re-training/fine-tuning, Diff-DKP behaves a plug-and-play scheme, thus without data demands and training cost before implementation. 



































 





















§ EXPERIMENTS






 §.§ Experimental Setup
 

Data Preparation. 
Following the widely adopted kernel assumption <cit.>, we conduct  the experiments on anisotropic Gaussian kernels and motion kernels, which are shown in Fig. <ref>.
The kernel sizes are set to (4s+3)×(4s+3).
For the Gaussian kernel, the width ranges are set to [0.175s, 2.5s], and the rotation angle range is set to [0,π], with a scale factor s = 4, respectively.
For the motion kernel, we adopt random motion kernel generation method proposed by <cit.>, which simulates realistic and complex blur kernels from random trajectories. Detailed formulations of Gaussian and motion kernels are given in the supplementary material. 
We synthesize LR images with random kernels with respect to Eq. (<ref>) for testing data based on five popular public benchmark datasets, including Set5 <cit.>, Set14 <cit.>, BSD100 <cit.>, Urban100 <cit.> and RealSRSet <cit.>.
We compare these kernels in terms of the peak signal to noise ratio (PSNR), and compare HR images in terms of PSNR and structural similarity (SSIM) <cit.>.





Comparison Methods.
The proposed DIP-DKP and Diff-DKP are compared with existing baselines including: Double-DIP <cit.>, DIP-FKP <cit.>, DASR <cit.>, BSRDM <cit.>, DCLS <cit.>, DARM <cit.> and DiffBSR <cit.>.
Specifically, Double-DIP tends to provide kernel priors by training a FCN network only with respect to the LR image restoration error. 
DIP-FKP incorporates the FKP model as kernel prior which is pre-trained on kernel datasets. KernelGAN+ZSSR and DARM are self-supervised and train an interal generative adversarial network (GAN) to estimate the blur kernel. 
BSRDM formulates an elaborate degradation modelling on noise and kernel as handcrafted priors. 
DASR is a representative end-to-end method that is pre-trained on DIV2K <cit.> and Flickr2K <cit.> HR image datasets. 
DiffBSR is fine-tuned on BSR labeled datasets before applied to estimate HR images.  
















Implementation and Hyper-parameters.

The adopted kernel estimation network G_k of PKE module in this paper is a three-layer fully-connected network (FCN).



The adopted DIP model follows the original settings in <cit.>, and the diffusion model is the vanilla version <cit.> that is trained on ImageNet <cit.>.



The number of sampling times in the MCMC simulation L is the only hyper-parameter in the proposed DKP model. 
The hyper-parameter tuning results are given in Table <ref>.It is explicit that the performance reaches equilibrium around L∈[4, 8]. 
To balance the efficiency and effectiveness, we set L=5 in this paper.




























































































 §.§ Comparison with State-of-the-Arts




Evaluation on Gaussian Kernel Scenario.
Quantitative evaluation results on four datasets with scale factors s=4 are presented in the upper half part of Table <ref>.
We can see that the proposed DIP-DKP and Diff-DKP achieve the second and the best results on all datasets. 
We note that DIP-DKP only realizes slightly higher performance than the existing state-of-the-art (SotA) methods, while Diff-DKP achieves significantly better performances. 
This recalls our demonstrations in Section <ref>: DIP-DKP is totally trained while solving from scratch, and the DKP model plays the role of providing better convergence guarantee. 
Diff-DKP utilizes the DKP model to guide the well-trained diffusion model with fruitful data priors to converge to BSR task for better HR image restoration performances. 
In Table <ref>, we further show that our DKP model achieves the accurate kernel estimation with  higher kernel PSNR.







Evaluation on Motion Kernel Scenario.
The lower half part of Table <ref> shows the simulation results on the motion kernel scenario.
The supervised learning methods, i.e., DASR and DiffBIR, are re-trained/fine-tuned on motion kernel degraded HR image datasets. 
DIP-FKP is re-trained on the motion kernel dataset. 
The proposed DIP-DKP and Diff-DKP show significantly better performance on the motion kernel scenario, which validates that the proposed DKP model has good generalization ability towards different kernel categories. 
Specifically, Diff-DKP presents stable PSNR/SSIM scores when being applied to estimate motion kernels, while the rest suffer significant performance drop. This indicates that the proposed DKP is expected to handle kernel varying tasks.









Visual Results.
The visual results of different methods on synthetic and real-world images are shown in Fig. <ref>.
We can see that 
1) In the case of Gaussian kernel, all methods are capable of producing satisfactory deblurring results,
while our DIP-DKP and Diff-DKP yield better results with more accurate kernel estimation.
2) In the case of motion kernel,
certain distortion on the estimated kernel can be seen in FKP-DKP and BSRDM fail to estimate motion kernel.
Meanwhile, our DIP-DKP and Diff-DKP achieve approximately accurate motion kernel estimation.
3) In the case of real image, both DIP-FKP and BSRDM estimate the Gaussian-like kernels, whereas our DIP-DKP and Diff-DKP tend to estimate the non-Gaussian kernels. 
This verifies that an adaptive and flexible kernel estimation discipline is learned by our DKP model, which may fit the real-world applications better.







 §.§ Ablation Studies

Ablation study of RKS module. The ablation studies are carried on the MCMC sampling of kernel priors.
“Without RKS" denotes that the adopted DKP updates the kernel network only by the data-consistency term without the learned kernel prior.
In Fig. <ref> (left), it can be seen that the estimated kernels without RKS have significant distortion, leading to remarkable PSNR drop of the estimated HR image, while DIP-DKP can estimate Gaussian kernels precisely with respect to the ground truth (with red frame).
Fig. <ref> (right) shows that the accurate motion kernel estimation no longer exists when the RKS module is absent. It is thus obvious that without the kernel prior learned from the MCMC process, the Diff-DKP fails to converge to a rational motion kernel estimation.
The average kernel and image results are shown in Table <ref>.
Without kernel prior learned from the RKS module, the kernel estimation performances of DKP-based BSR methods have a significant performance drop, leading to poor image restoration performance as well.

Ablation study of PKE module.
Since PKE essentially estimates blur kernels on the basis of the random kernel priors and LR observations, thus it is indispensable and we conduct ablation study on the different structures of kernel network in PKE module in Table <ref>.
We find that the kernel network performs well when it has 3-7 layers with 100-1000 units in each layer.
This indicates that the kernel network has good generalization-ability for the structure without the necessity of elaborately designing the network.








 §.§ Model Size, Runtime and Memory Usage

The kernel network of our DKP model has a total of 562K parameters (FLOPs: 536K) while Double-DIP and DIP-FKP have 641K parameters (FLOPs: 600K) and 143K parameters (FLOPs: 178K), respectively.
The runtime and memory usage of our DIP-DKP on a GeForce RTX 3090 GPU for generating a HR image of size 512×512 are about 92 seconds and 11GB memory, which is comparable with the Double-DIP (91 seconds and 11.2GB) and DIP-FKP (90 seconds and 10.6GB).
As for Diff-DKP, the 512×512 image needs to be divided into four 256×256 images for restoration, which costs a total of 60 seconds and 4GB memory.
Considering that our DIP-DKP and Diff-DKP are unsupervised and plug-and-play, it is reasonable to say that our methods have moderate computational costs.

Due to the page limitations, more experimental results are given in the supplementary material.


§ CONCLUSION

In this paper, we propose a dynamic kernel prior (DKP) model to solve the BSR problem in an unsupervised and pre-training-free paradigm. 
DKP realizes the rational kernel prior learning from MCMC sampling on random kernel distributions, providing accurate kernel estimation and thus leading to better HR image restoration. 
DKP can be easily incorporated with existing image restoration model, such as DIP and diffusion model, by replacing their kernel modeling modules or adding as an external kernel prior generator. 
When applied to solve the BSR problem, DKP is trained while solving the task with respect to the LR image restoration error, enabling no training necessity and labeled data demands. 
Extensive experiments on Gaussian and motion kernel scenarios with synthetic LR images and real-world images validate that DKP-based methods improve the kernel estimation accuracy significantly and thus lead to superior BSR results.





We believe that the concept of using a trainable sampling process to provide adaptive priors will lead to a new direction in solving low-level tasks, aiming to achieve superior performance with modest computational costs in the way of unsupervised inference.


ieeenat_fullname






