Learning to Translate Noise for Robust Image Denoising
denoising;generalization;robustness;noise translation
Dear reviewer k3tR,

Thank you for your detailed review and feedback on our paper. We have posted the rebuttal addressing each of your comments and concerns. We appreciate your constructive insights, which have guided us to enhance the clarity and depth of our paper.

Thank you once again for your time and valuable feedback.

Best regards, The Authors
Dear reviewer jgKR,

Thank you for your detailed review and feedback on our paper. We have posted the rebuttal addressing each of your comments and concerns. We appreciate your constructive insights, which have guided us to enhance the clarity and depth of our paper.

Thank you once again for your time and valuable feedback.

Best regards, The Authors
Dear reviewer Nvpd,

Thank you for your detailed review and feedback on our paper. We have posted the rebuttal addressing each of your comments and concerns. We appreciate your constructive insights, which have guided us to enhance the clarity and depth of our paper.

Thank you once again for your time and valuable feedback.

Best regards, The Authors
Dear reviewer UMZZ,

Thank you for your detailed review and feedback on our paper. 
We have posted the rebuttal addressing each of your comments and concerns. 
We appreciate your constructive insights, which have guided us to enhance the clarity and depth of our paper. 

Thank you once again for your time and valuable feedback.

Best regards,
The Authors
Dear reviewer 6pYh,

Thank you for your detailed review and feedback on our paper. 
We have posted the rebuttal addressing each of your comments and concerns. 
We appreciate your constructive insights, which have guided us to enhance the clarity and depth of our paper. 

Thank you once again for your time and valuable feedback.

Best regards,
The Authors
### [Reviewer jgKR, k3tR] Is this evaluation setting realistic? Why not try translating from synthetic noise?

Our research addresses real-world noise, which is challenging to model with a specific distribution due to various factors such as camera settings and scene conditions. In our experiments, each real-world noise dataset exhibits a unique distribution, so training on a specific dataset does not consistently enhance performance on others. In this context, good generalization is the ability to adapt effectively to any realistic noise distribution. Therefore, we do not need to assume that real-world noise follows a specific distribution, such as Poisson noise, as our setting is already obviously realistic. We empirically validate that our framework can handle arbitrary noise types within a realistic scope, including Poisson-distributed noise.


### [Reviewer Nvpd, k3tR] Is training with only the SIDD dataset sufficient?

We first refer to Figure 9 in our supplementary document to verify its effectiveness on unseen real images. As presented, our framework trained solely on SIDD dataset performs well even on the images captured by smartphone. This success is due to (1) noise translation being more generalizable than direct denoising (less prone to overfitting) and (2) the use of noise injection during both training and inference, which enhances our framework’s robustness against noise type shifts at test time.
Global Rebuttal 4
### [Reviewer jgKR] Is explicit loss necessary for the performance?

Although our method seems to work well with baseline translation only with Gaussian injection terms, the optimal performance is achieved when all terms are applied. We emphasize that achieving higher PSNR is a lot more difficult when the performance is already quite high. The average performance gain on the OOD datasets from Gaussian injection is 0.34 dB, and the gain from the explicit loss is 0.25 dB, indicating that the contribution from explicit loss is also significant.

### [Reviewer UMZZ] Is L_freq component in explicit loss necessary?

We introduce L_freq because element-wise Gaussian noise does not necessarily guarantee spatial uncorrelation. To illustrate this, consider 2D Gaussian noise after applying a 3x3 averaging kernel. The result retains spatial correlation, even though it follows a Gaussian distribution element-wise in the spatial domain.
Global Rebuttal 3
### [Reviewer Nvpd, UMZZ] Is it possible for a single noise translation network to pair with a different denoising network?

To validate the generalization ability of our noise translation network, we first train a noise translation network in combination with a pre-trained NAFNet. We then evaluate its performance by pairing the noise translation network with a pre-trained Restormer. As shown in the below table, this cross-performance still remains high, demonstrating the network's generality.

|           | Sidd  | Poly  |  CC   | HighISO | iPhone | Huawei | OPPO  | Sony  | Xiaomi | OOD Avg. |
|-----------|:-----:|:-----:|:-----:|:-------:|:------:|:------:|:-----:|:-----:|:------:|:-------:|
| NAF-Res   | 39.19 | 38.64 | 37.27 |  39.95  |  41.93 |  39.64 | 40.51 | 43.99 |  36.11 |  39.76  |
| NAF-NAF   | 39.17 | 38.67 | 37.82 |  39.94  |  41.94 |  39.74 | 40.45 | 44.17 |  36.14 |  39.86  |
Global Rebuttal 2
### [Reviewer 6pYh, UMZZ, jgKR] Additional comparison with related works

[LAN: Learning to Adapt Noise for Image Denoising]

We would like to emphasize that LAN is a self-supervised method, which adds learned parameters to the input noisy image by backpropagating 5~20 iterations per every single image. In contrast, our method translates noise to the target distribution with minimal computational complexity using a lightweight noise translation network. Due to the computational burden, the LAN paper mostly performed evaluations at a 256x256 resolution. In contrast, our approach is much less computationally intensive, as it requires no additional iterations and uses a lightweight noise translation network, allowing for 1024x1024 inference in a single pass on Restormer. 
For quantitative comparison with LAN, we evaluate our method under the same setting with LAN at a 256x256 resolution in the PolyU dataset. Our framework achieves 39.34dB with Restormer, which obviously exceeds the maximum performance of LAN (39.30dB with Restormer, 10 iterations).
We will add more detailed discussion and analysis with LAN in the revised version.


[Dual Adversarial Network: Toward Real Noise Removal and Noise Generation]

As Reviewer 6pYh suggested, our noise translation and denoising network could be trained simultaneously. However, we highlight that a key contribution of our work is the efficient adaptation of a pre-trained denoising model through a lightweight noise translation network. Note that training our noise translation network, in contrast to training an entire network from scratch, requires only 0.5-1% of the computational cost.

Additionally, we compared the performance of our framework with DANet, as shown in the table below. Our method consistently outperforms DANet on most OOD datasets, while DANet even falls short compared to Restormer baseline (OOD Avg PSNR of 38.85 dB)


|           |  SIDD   |  Poly   |   CC    | HighISO | iPhone | Huawei |  OPPO  |  Sony  | Xiaomi | OOD Avg |
|-----------|:---------:|:---------:|:---------:|:---------:|:--------:|:--------:|:--------:|:--------:|:--------:|:---------|
| DANet | **39.46** |  37.53  |  36.15  |  38.34  | 40.45  |  38.40  |  39.85  | **44.35** |  35.57  |  38.83  |
| Ours  |  39.17  | **38.67** | **37.82** | **39.83** | **41.94** | **39.71** | **40.45** |  44.17  | **36.14** | **39.86** |
Global Rebuttal 1
Yes
No ethics review needed.
Please see my "weakness" part.
The idea is clear, reasonable and interesting.
The experiments seem to be comprehensive, and the superiority of the proposed method is evident.
This study proposes a new image denoising method, mainly aiming to improve its generalization performance to unseen practical complex noise. The main methodology is to build a noise translation network, purposing to translate diverse noises with unknown types into Gaussian, and thus make the image denoising pretrained on Gaussian noisy data reasonable to be used in such images.
Although the proposed main methodology is reasonable, but I still have the following major concerns. 

Firstly, two networks are involved in the proposed framework, including an image denoising network and a noise translation one. It seems that the two networks have been separately trained, and the former is pretrained and fixed in the training process of the latter one. Such a training manner tends to make the noise translation network's performance highly dependent on the capability of the former one. I think the two components should be more rationally trained simultaneously, to make them ameliorate each other's ability mutually. I highly recommend authors to read the following paper:
Dual Adversarial Network: Toward Real Noise Removal and Noise Generation. ECCV, 2020
which also investigated the real complex noise issue, but can learn the noise generation mechanism and denoiser together in a consistent and unified framework.

Secondly, when training the noise translation net, the so-called "explicit loss" should play an important role since it encode the stochastic knowledge of noises into the learning process. But the aimed labels are a preset Gaussian, with pregiven mean and variance (actually evaluated from the current predicted noise). This is a too ideal pre-assumption in my opinion. If the real noise is very complicated (like non-iid or non-Gaussian), and the pretrained network is not very accurate, I think such a label guidance can not guaranteed to be stably and robustly performed. Not with a solid theoretical support, I think such ideal assumption inclines to make the proposed method too heuristic and applicable only in a narrow domain. 

Thirdly, when handling complicated real noise cases, the proposed method seems to be highly dependent on the pre-collected noisy and clean image pairs. This, however, should be extremely difficult in more real scenarios, that is, in real applications, it is generally difficult to achieve large amount high-quality ground truth clean images for those real noisy ones. In this sense, I think the manner used in Dual Adversarial Network or other unsupervised manners should be more useful and meaningful.