Noise Re-sampling for High Fidelity Image Generation
Generative models;Diffusion based models
Yes
No ethics review needed.
See the weaknesses part.
1. The method is simple and does not need any training.
2. The results show improvements in the cropped regions in the generated images.
The paper proposes to use noise re-sampling to generate high-fidelity images. The method takes place in the inference stage of the diffusion model where 1) an image is sampled first using the original diffusion process 2) a patch is cropped and upsampled and then sent to the encoder to get the latent features 3) a random Gaussian noise is added to the latent and denoise back to get clean image 4) paste back to the image. The paper shows comparisons on multiple images especially human images which demonstrates the effectiveness of the method.
1. How would the method affect the inference speed? As the proposed method requires some more operations than the vanilla generation process, it would be better to understand the efficiency of the method as well.
2. The method contains randomness as it involves the addition of the noise and the random selection of the cropped region, how would the method's robustness be? Will it keep outputting high-quality images regardless of the randomness?
3. Could the authors also present some results on SD 3, the paper only shows some original results in Figure 1 but lacks the results output by the proposed method.
Yes
No ethics review needed.
See the previous section.
- The identified issue in latent diffusion models is interesting and requires attention for better image generation.  
- The perspective from signal processing is interesting, and provides a feasible direction for optimization with diffusion models.  
-  From experimental evaluations, the authors demonstrated the generalizability of their approach for different models and in all cases, they further boosted the performance of the the backbone model.
The paper proposes an interesting noise re-sampling technique aimed at improving the fidelity of image generation in Latent Diffusion Models (LDMs), particularly for small, detailed objects that are challenging to render accurately. Traditional LDMs suffer from detail loss due to Variational Autoencoder (VAE) compression, especially in high-frequency regions critical for intricate textures. The authors address this by locally increasing the sampling rate in noise space, allowing LDMs to generate high-resolution details without altering global structures. Additionally, they introduce a VAE-based upscaling method to ensure consistency between upsampled local patches and the global image. The approach demonstrates superiority over baseline models, including Stable Diffusion and super-resolution methods, by enhancing image detail and fidelity across multiple scales.
Firstly, the paper writing is poor. It's a little hard to capture the overall idea of this paper. A few more concrete comments include: 
- The whole method section makes it hard to capture the high-level idea. I am not sure if I understand this paper correctly, does the whole logic of the proposed method be as follows: Identify and decode local patches --> Upsampling in image space --> Re-encoding with added noise and denoising --> Merging with global image?   
- Following the previous point, the whole method section should be revised with a clear problem formulation (i.e. what are the input and outputs, and what are the goals), a straightforward model pipeline, and consistent and well-explained notations. 
- Quite a few texts are redundant. The authors repeatedly discuss the motivation of their approach in the introduction, method, and experimental evaluations.   
- Table 1 is hard to read. Does SD1.5, SD2 mean backbone models, and for all the compared models, they are built upon SD?  

Next, the novelty of this paper is limited. The proposed method is more like an engineering-wise post-processing to enhance the fine-grained details of the diffusion models. There is no tech contribution from this paper, and the signal processing perspective does not fully align with the proposed method. Even if it is, the perspective itself is more like a way to explain why diffusion models miss fine-grained details rather than a contribution of this paper. 

Other questions:
- high-frequent components or fine-grained details? In the paper, the authors take a unique perspective on signal processing, and reveal that VAE has a higher error on high-frequent objects. However, they essentially propose a resampling method to restore fine-grained details. Taking Figure 6 as an example, the proposed method did better restore the details, but the high-frequent components (boundaries) are not clearly better. In the method section, the ``resampling'' is simply crop-and-resize. In other words, the proposed method itself is not quite aligned with signal processing.  
- How about the inference time? The proposed method runs two rounds of denoising and at least two rounds of VAE encoding and decoding, will the inference time doubled, and how to further optimize the inference time?  
- In real practice, how would the proposed method work, without manually selecting resampling regions?
Yes
No ethics review needed.
Please see the Weaknesses part.

Overall, I'm satisfied with the paper's quality and am willing to improve my score if the authors can address my questions.
1. The idea is quite simple but overall I think it is a good paper. Instead of directly increasing the sampling rate, using VAE to perform local space re-sampling in the image space is quite interesting, as it avoids the need for a higher-resolution dataset and re-training. Figure 5 also demonstrates that this method indeed captures more high-frequency information.

2. The proposed re-sampling strategy is straightforward and can be plug-and-play for LDMs.

3. The author's exploration of the impact of VAE compression on the generation quality of LDMs aligns with intuition, and this paper reminds me of a work IDM [1] that uses cascaded Diffusion Models for continuous super-resolution

5. The authors provide a demo code which is worth encouraging. I've tried it and it works fine.

[1] Gao S, Liu X, Zeng B, et al. Implicit diffusion models for continuous super-resolution[C]. CVPR 2023.
This work focuses on the noise sampling rate/resolution in LDMs, and proposes a local reference re-sampling strategy to improve the high-resolution image generation performance. 
The proposed method uses the normal global noise-based denoised image as the reference, to further guide another branch which is focused on local region. 
Meanwhile, the method deploys a training-free mixed upsampling strategy to provide corresponding local latent noise aiming at generating patches with better quality.
Extensive experiments show the proposed method achieves good performance, especially for complex objects and other local details.
1. Figure 4 is kind of ambiguous. 

     (1): If I understand correctly, the blue boxes in subfigures (a) and (b) should be the same size. I also suggest indicating the size of each box for clarity.

     (2): I suggest removing the "Direct latent space upsampling" branch below subfigure (c), or replacing it with the noise re-sampling branch corresponding to Figure 5 (d). Additionally, the title for subfigure (c) appears twice.

2. The statement in lines 367-369, "The final re-sampled clean latent code can be directly pasted back to the corresponding local region of the original output after decoding" is ambiguous. Based on Figure 4 (a), it should be decoded first, and then pasted back to the output image.

3. I am a bit confused by the statement in line 395, "This reference ensures that the denoising process aligns with the overall image". Since the local reference image only selects a portion of the patch without including information from the surrounding areas, how does it ensure continuity between the patch edges and the rest of the image?

4. I suggest conducting an ablation study on the selection of  𝑁 and the size of the local patch 𝑠 to explore the impact of different combinations on performance and efficiency.

5. How to determine the ratio 𝛼_𝑡 of latent upscaling and VAE-based upscaling in Equation 8? Does it also gradually decrease as 𝑡 increases?

6. I think the Figure 3 (d) in line 264 should be Figure 3 (b).

7. Some typos: for example, "to generated" should be "to generate" in line 299

8. There are some minor errors in the reference. For example, Self-Cascade [1] in ECCV is not correctly referenced and is repeated in lines 044 and 048. Also ScaleCrafter [2] is published in ICLR23.

[1] Guo L, He Y, Chen H, et al. Make a cheap scaling: A self-cascade diffusion model for higher-resolution adaptation. ECCV 2024.
[2] He Y, Yang S, Chen H, et al. Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models[C]. ICLR 2023.
Yes
No ethics review needed.
See above
1. The paper introduces the concept of Noise Resampling, which is somewhat novel to me in the context of latent diffusion.
2. The paper studies the error in VAE reconstruction and attempts to address it with noise resampling. It's somewhat valuable.
3. The paper is well organized.
The paper proposes a Noise Re-sampling strategy to enhance high-fidelity image generation in Latent Diffusion Models (LDMs). This method selectively increases sampling rates in detailed local regions. Additionally, a VAE-based upscaling technique ensures consistency across local and global regions by upscaling latent spaces in the image domain. Experiments shows that the approach outperforms existing super-resolution and high-resolution adaptation methods on SD backbones.
1. The proposed solution is straightforward. To address the low-quality fine-grained details in SD generation, the method adds noises to generated patches then conduct the backward denoising process again. Although some methods such as VAE upscaling has been proposed to improve this method, this idea is straightforward without many insights into the diffusion process. 

2. The proposed method requires massive denoising and VAE upsampling over different resolutions of the generation. This would introduce significant computational costs. Unfortunately, I didn't see any discussions on method efficiency in the paper.

3. The performance improvement on global metrics, such as FID and KID, is trivial.
Yes
No ethics review needed.
Stable Diffusion 3 is known for its ability to generate text well. Can noise re-sampling correct errors when text generation goes wrong?
- This paper highlights the issue that VAE compression errors are more prominent in lower-resolution images, a problem that the generative AI community has not paid close enough attention to.

- The proposed method is training-free, making it easy to apply with minimal effort to any newly developed latent diffusion model.
This paper proposes a noise re-sampling strategy and VAE-based upsampling to address the lossy compression problem of VAE in latent diffusion models, which are almost essential in the text-to-image field. The authors observe distortions in small objects generated by latent diffusion models like Stable Diffusion 1, 2, and 3, and propose increasing the sampling rates of local regions. The proposed noise re-sampling method crops the local region of the latent image generated by the diffusion model, upscales it using the proposed VAE-based upsampling method to match the size learned by the diffusion model, and then refines the upsampled crop before attaching it to the original image. The paper compares this method with approaches like Stable SR and Multi-Diffusion, demonstrating superior qualitative and quantitative performance.
- The most fundamental way to address the VAE's lossy compression issue is by using a better VAE. For instance, OpenAI's consistency decoder [1] replaces the decoder of Stable Diffusion 1's VAE with a new diffusion model and releases model weights that can rapidly decode latents into images via consistency distillation [2]. Samples from the consistency decoder demonstrate strong reconstruction performance, particularly on low-resolution (256x256) images where reconstruction errors, like those mentioned in this paper, often occur. This paper should also include a comparison with the consistency decoder.

- Another approach is to ensure the diffusion model generates better latents. Recent research on diffusion guidance [3] suggests that strong classifier-free guidance at middle noise levels enhances image detail. The proposed noise re-sampling method also requires experiments with various guidance scales.

- Some implementation details are missing. The local patch chosen for upsampling should ideally be a small object prone to distortion, but how is this patch selected? What is the exact upsampling ratio used in experiments? When denoising the local patch with the diffusion model, what text prompt is used?

- The proposed VAE-based upsampling seems counterintuitive because it relies on the same VAE that the paper argues to suffer lossy compression issues. This suggests that the upsampled patch might contain significant artifacts.

- The paper lacks of qualitative results of SD3 and SDXL.

[1] Open AI, https://github.com/openai/consistencydecoder

[2] Song et al., Consistency Models, ICML 2023.

[3] Kynkäänniemi et al., Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models, 2024.
A Noise Re-sampling plugin for current LDMs, which enhances the generation quality of LDMs by enabling multi-scale generation and preserving high-frequency information
@inproceedings{
anonymous2024noise,
title={Noise Re-sampling for High Fidelity Image Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GD4Tlqvwrq},
note={under review}
}
Latent diffusion models (LDMs) have emerged as powerful tools for generating diverse and realistic samples across domains. However, their efficacy in capturing intricate details and small-scale objects remains a challenge. 
Our investigation reveals that VAE compression induces errors in the latent space and limits the generation quality. Furthermore, LDMs trained on fixed-resolution images struggle to produce high-resolution outputs without distortions, making simple resolution increases ineffective.
In this paper, we propose a novel **noise re-sampling** strategy that enables multi-scale generation of LDMs, allowing LDMs to "zoom in" and improve generation quality of local regions. By increasing the sampling rates from the noise perspective in the latent space, we effectively bypass the constraints imposed by VAE compression, thus preserving crucial high-frequency information. Our approach, a simple yet effective plugin for current LDMs, enhances the quality of image generation in local regions while maintaining overall structural consistency and providing fine-grained control over the scale of generation in latent diffusion models.
Through extensive experimentation and evaluation, we demonstrate the efficacy of our method in enhancing the generation quality across various LDM architectures. Our approach surpasses existing methods, including stable diffusion (SD) models, SD-based super-resolution methods and high-resolution adaptation methods, in generating high-fidelity samples of complex objects.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Generative models Diffusion based models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/044802c95b5f40d2d91d4502e7c45ae0d335bb5d.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/16a1bc2ff3a9924a042301144c5db5cb2719a44d.zip
Noise Re-sampling for High Fidelity Image Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission