CutDiffusion: A Simple, Fast, Cheap, and Strong Diffusion Extrapolation Method
Mingbao Lin Zhihang Lin Wengyi Zhan Liujuan Cao Rongrong Ji









CutDiffusion



















Lin et al.











^1Skywork AI     ^2Xiamen University


<https://github.com/lmbxmu/CutDiffusion>

CutDiffusion: A Simple, Fast, Cheap, and Strong Diffusion Extrapolation Method
    Mingbao Lin1 Zhihang Lin2 Wengyi Zhan2  Liujuan Cao2Correspondence: caoliujuan@xmu.edu.cn; First Author: linmb001@outlook.com  Rongrong Ji2
    April 29, 2024
===============================================================================================================================================









Transforming large pre-trained low-resolution diffusion models to cater to higher-resolution demands, i.e., diffusion extrapolation, significantly improves diffusion adaptability.

We propose tuning-free CutDiffusion, aimed at simplifying and accelerating the diffusion extrapolation process, making it more affordable and improving performance.

CutDiffusion abides by the existing patch-wise extrapolation but cuts a standard patch diffusion process into an initial phase focused on comprehensive structure denoising and a subsequent phase dedicated to specific detail refinement.

Comprehensive experiments highlight the numerous almighty advantages of CutDiffusion:

(1) simple method construction that enables a concise higher-resolution diffusion process without third-party engagement;

(2) fast inference speed achieved through a single-step higher-resolution diffusion process, and fewer inference patches required;

(3) cheap GPU cost resulting from patch-wise inference and fewer patches during the comprehensive structure denoising;

(4) strong generation performance, stemming from the emphasis on specific detail refinement.





   
  





§ INTRODUCTION




As a member of diffusion architecture models <cit.>, also called a latent diffusion model <cit.>, stable diffusion, primarily used to generate detailed images conditioned on text descriptions, has been considered to be a crucial piece of the ongoing bloom of generative artificial intelligence.


Coupled with its realistic image generation, stable diffusion is open-sourced and supports running on most consumer hardware equipped with a modest 4 GB VRAM GPU. Thus, it marked a departure from many proprietary text-to-image models such as DALL-E series <cit.> and Midjourney program <cit.>, accessible only via cloud services.



One of the dominant real-world applications is to generate high-resolution images.

Initial releases of stable diffusion were trained on a 512 × 512 resolution dataset, resulting in quality issues like degradation and inaccuracies when user-specified resolution deviates from its “expectation” <cit.>.

Though the subsequent version 2.0 strengthened the ability to generate 768 × 768 resolution images <cit.> and Stable Diffusion XL (SDXL) released native 1024 × 1024 resolution <cit.>, retraining of stable diffusion models is slow because of its repeated, sequential nature, and eats a lot of memory because of the operations in latent pixel space, which becomes huge for high-resolution images.

It was reported to train the initial release using 256 Nvidia A100 GPUs on Amazon Web Services for a total of 150,000 GPU-hours, at a cost estimation of $600,000 <cit.>.

The charge drastically increases in higher resolution, which as consequences is almost prohibitive for individuals, even for most small- and medium-sized enterprises, to train large diffusion models one by one for a certain resolution size, in particular, taking into account the ultra-high resolution such as 4096 × 4096 or 8192 × 8192.




















To address the impediments of training an initial model, diffusion extrapolation, which relies on a well-pretrained low-resolution model and achieves higher-resolution image diffusion, has gained significant interest.

Recent research workers opt for fine-tuning involvement <cit.>, or even tuning-free implementation <cit.>, to realize more user-specific generation outputs.


In Table <ref>, we assess major pros and cons of these methods considering factors of tuning-free, fast inference speed, cheap GPU costs, equal parameters, performance and method simplicity.


We embark on method analyses here and specific reports will be given in Sec. <ref>.

On the lines of fine-tuning involvement, most methods make efforts for a parameter-efficient optimization to enable fast adaptation to “unexpected” resolution generation. For example,


DiffFit <cit.> fine-tuned the bias term and newly-added scaling factors in specific layers and Self-Cascade <cit.> upgraded time-aware feature upsampling modules to inject the side information from newly acquired higher-resolution data.





These tuning methods excel in the accurate portrayal of local structural details. Their ability to scale to the target resolution in one step facilitates rapid inference speeds.

However, direct inference at higher resolutions incurs sustainable peak GPU memory expenses due to more pixels, and parameter tuning adds complexity, sometimes extra parameters <cit.>.




 
In tuning-free implementation, all methods keep the same parameter amount as the original models. A straightforward approach is direct higher-resolution inference, which offers fast speed but requires high GPU consumption, similar to fine-tuning methods.

Results in <cit.> with SDXL (denoted as SDXL-DI) showed poor object repetition, twisted object and unreasonable structure, etc.

Attn-SF <cit.> simplified its methodology by scaling attention entropy in balance with the token quantity, which however still endures poor quality of object repetition.

ScaleCrafter <cit.> identified the limited receptive field of convolutional kernels, and emphasized on complex adjustment of dilated stride, injected step, etc. Also, larger kernels affect inference speed.


Another tuning-free implementation performs diffusion extrapolation in a patch-wise manner, reducing the inference GPU costs. For instance, MultiDiffusion <cit.> split an entire higher-resolution noise into multiple overlapping patches and binded them together through multiple diffusion generation processes.

Albeit its simplicity merit, lack of global information leads to issues like repetitive and distorted generation.

Recent DemoFusion <cit.> constructed residual connection and dilated sampling to incorporate global semantic information. Despite its better performance, these operations make DemoFusion sophisticated.

Moreover, DemoFusion adopts a fashion of progressive upscaling, where for example, the generation of a 4096 × 4096 image is constructed on intermediate diffusions of 1024 × 1024, 2048 × 2048, and 3072 × 3072, causing more diffusion steps and slower inference latency.








We dedicate our efforts to exploring the potential of tuning-free diffusion extrapolation through a patch-wise generation approach. Our primary objective is to simplify and expedite the diffusion extrapolation process, ultimately making it more cost-effective and enhancing its overall performance.

With this goal in mind, we present CutDiffusion in this paper, an almighty method that cuts the conventional patch diffusion process into two separate stages, each incorporating distinct sub-patch partitioning methodologies.



During the preliminary stage, non-overlapping patches are arbitrarily sampled from an independent Gaussian distribution and recovered in a spatial pixel interaction fashion, ensuring that each patch focuses on denoising the entire structure. In essence, every individual patch strives to comply with the same textual directive and produce analogous content.

In continuation of the subsequent stage, we reassemble the identical spatial pixels of the first-phase sub-patches to create a more refined higher-resolution Gaussian noise. 

Then, we follow MultiDiffusion <cit.> to develop overlapping patches, where each individual patch is designed to focus on refining and enhancing partial details within the overall image.


As listed in Table <ref>, in addition to its tuning-free completion and absence of parameter introduction, the proposed CutDiffusion also offers substantial advantages in terms of:


(1) Simple method construction. CutDiffusion is constructed upon a standard diffusion process, only altering sub-patch sampling, which eliminates complex third-party engagement and creates a more concise higher-resolution process.





(2) Fast inference speed. CutDiffusion performs a single-step upscaling to the target resolution, an efficient alternative to progressive upscaling, and necessitates fewer inference patches during denoising the comprehensive structure.




(3) Cheap GPU cost. CutDiffusion employs a patch-wise approach for extrapolation and decreases the quantity of non-overlapping patches required for structure denoising, effectively mitigating the necessity for peak GPU memory.






(4) Strong generation performance. Albeit its single-step upscaling approach and absence of third-party engagement, CutDiffusion gracefully executes partial detail refinement, consequently achieving superior generation performance.










§ RELATED WORK




 §.§ Diffusion Models


Diffusion models, often known as probabilistic diffusion models <cit.>, fall under latent variable generative models. 


A classic diffusion model is comprised of the forward process, the reverse process, and the sampling methodology <cit.>.


The forward process initiates at a given starting point, and then consistently introduces noise into the system.


The reverse process generally entails teaching a neural network to remove Gaussian noise from blurred images. The model is designed to counteract the noise addition process in an image, and employed for image generation by refining and denoising a randomly noisy image.


The sampling mechanism operates by navigating parameters and diminishing the variance of the samples, resulting in more accurate approximations of the desired distribution.

DDIM <cit.> allows the utilization of any model trained on DDPM loss <cit.> to sample with a few steps omitted, trading a modifiable quantity of quality. 

LDM <cit.> initially compresses images, applies a diffusion model to represent the distribution of encoded images, and finally decodes them into images.

The supreme performance has made it widely applied to generate text-driven images.





 §.§ Diffusion Extrapolation
 







  §.§.§ Fine-tuning Involvement.



It is a logical idea to fine-tune pre-trained diffusion models to accommodate the generation of higher-resolution images. 

Owing to the considerable training costs of these models, seeking parameter-efficient optimization methods has become a central focus for researchers and practitioners in the fine-tuning field.

In their research, Xie et al. <cit.> took an innovative approach by freezing most of the parameters within the latent diffusion model. They focused on training specific parameters such as the bias term, normalization, and the class conditioning module. Additionally, they incorporated learnable scale factors into the model to further enhance its performance and adaptability.


Guo et al. <cit.> developed an innovative cascade framework that included a low-resolution model at its initial scale and additional new modules at a higher scale. 




  §.§.§ Tuning-free Implementation.


Another line of researchers seek for tuning-free diffusion extrapolation. 



In Attn-SF <cit.>, lower resolution images showed incomplete objects, while higher resolution ones had repetitive chaos. Therefore, a scaling factor was added to address attention entropy shifts and fix defects.

He et al. <cit.> identified limited kernel perception as causing object repetition and structural issues in diffusion models. They suggested dynamic perception field adjustment and proposed dispersed convolution and noise-damped guidance for ultra-high-resolution generation.

MultiDiffusion <cit.> demonstrated the capability of combining several overlapping denoising paths to create panoramic images.


The study DemoFusion <cit.> has innovatively enhanced the injection of global information. This enhancement is achieved by using intermediate noise-inversed representations as skip residuals. Additionally, the use of dilated sampling in the denoising paths further contributes to the improvement.









§ METHODOLOGY




 §.§ Background


Latent Diffusion Model.


Let 𝐳_T ∈ℝ^h × w × c represent a noise tensor following a Gaussian distribution in the latent space. A large pre-trained latent diffusion model (LDM) <cit.> utilizes its three components to denoise 𝐳_T conditioned on a user-specified text prompt 𝒫, resulting in a clean image 𝐈_0 ∈ℝ^H × W × 3:


(1) A text encoder τ_θ, transforms conditional text prompt 𝒫 to an embedding space τ_θ(𝒫). This transformation allows for a more comprehensive and meaningful representation of the user-specified text prompt.



(2) A U-Net ε_θ, executes consecutive T denoising diffusion steps, each of which estimates the noise and recovers a cleaner latent 𝐳_t-1 from 𝐳_t:


    𝐳_t-1 = √(α_t-1/α_t)𝐳_t + (√(1/α_t-1 - 1) - √(1/α_t - 1))·ε_θ(𝐳_t,t,τ_θ(𝒫)),




where t = T → 1 and {α_t}_t=1^T is a set of prescribed variance schedules.


(3) A VAE decoder 𝒟, is responsible for decoding the final image. It accomplishes this by converting the latent representation back into pixel space 𝐈_0 = 𝒟(𝐳_0).

We visualize the decoding of latent 𝐳_t, i.e., 𝐈_t = 𝒟(𝐳_t) and the final clean image 𝐈_0 in Fig. <ref>, where our motivation originates as described in Sec. <ref>.


The above simply introduces essential notations for subsequent content. We recommend the original paper <cit.> for a thorough understanding of LDM.













  §.§.§ Diffusion Extrapolation.



An LDM model is mostly pre-trained on a fixed resolution size, such as H=1024, W=1024 for SDXL <cit.>, leading to latent resolution h = 128, w=128. Diffusion extrapolation refers to the case of generated resolution h̅ (H̅), w̅ (W̅) beyond the pre-trained resolution.


A variety of existing methods, as referenced in studies like <cit.>, engage in the process of conducting direct higher-resolution inference, which consequently contributes to a significant challenge in terms of sustainable peak GPU expenses.

Recent patch-wise techniques <cit.> utilized a shifted window to sample overlapping patches {𝐳̅_T^i}_i=1^L from higher-resolution latent 𝐳̅_T ∈ℝ^h̅×w̅× c, where 𝐳̅_T^i ∈ℝ^h× w× c and L = (h̅-h/d_h+1)·(w̅-w/d_w+1) is patch number, d_h and d_w are vertical and horizontal strides.

Then, patch-wise denoising via Eq. (<ref>) is conducted to derive 𝐳̅_t-1 where the overlapping sections are calculated by taking the average. Finally, 𝐳̅_0 is decoded to a higher-resolution image 𝐈̅_0 ∈ℝ^H̅×W̅× 3.



Most of these methods require complex third-party engagement such as fine-tuning <cit.>, low-resolution image generation <cit.>, progressive upscaling <cit.>, etc.

We develop our method by employing a patch-based strategy, which is specifically designed to simplify and effectively manage the complex operations involved in these intricate processes, ensuring a more efficient and user-friendly approach.







 §.§ Motivation and Framework










  §.§.§ Motivation.




In Fig. <ref>, LDM progressively denoises random noise in Fig. <ref>(b) via iterative steps until a predetermined count (t = 50 → 1) is reached. The resulting Fig. <ref>(a) captures the prompt 𝒫 concept.


Upon examining the progression from Fig. <ref>(b) to Fig. <ref>(k), and ultimately the clean image in Fig. <ref>(a), we can deduce that the diffusion process involves an initial stage of denoising the entire structure, followed by a subsequent stage of refining partial details.

At earlier time steps, various contour profiles become more recognizable. At step t = 40, the contour profile resembling a dog becomes clear. As we move to step t = 35, the tree contour starts to form. Further to step t = 30, the chair's silhouette begins to emerge, making it more distinct, and finally by step t = 25, the profiles of the beach, sky, and sea become clearer, allowing for a better overall structure of the scene.


During later time steps from t = 20 to t = 5, the refinement of local details within the generated contour profiles takes place to enhance the realism of the objects. For instance, the sand appears to collapse around the chair legs, the sky and clouds seamlessly blend together, the dog's fur transforms into a more natural, limp state, and numerous other improvements occur.


Consequently, we recognize a fresh perspective that the diffusion extrapolation can be developed by adhering to the law of progressive image generation. This involves prioritizing comprehensive structure denoising initially, followed by refining specific details, for a more seamless higher-resolution diffusion image.



  §.§.§ Framework.



To develop concise diffusion extrapolation, we introduce the framework of our CutDiffusion in Fig. <ref>. In the premise of a standard diffusion process, CutDiffusion incorporates a patch sampling approach as its only modification. 








In the comprehensive structure denoising phase, we randomly sample multiple non-overlapping patches {𝐳̅_T^i}_i=1^L_1 from Gaussian noise, where 𝐳̅_T^i ∈ℝ^h × w × c. We then perform reverse diffusion steps t = T → T' to denoise each patch's structure. Identical spatial pixels undergo random interactions to enhance textual directive uniformity and generate similar content within denoised patches {𝐳̅_T'^i}_i=1^L_1.

In the specific detail refinement stage, we reassemble the structurally enhanced higher-resolution Gaussian noise, 𝐳̅_T'∈ℝ^h̅×w̅× c, by placing identical spatial pixels of {𝐳̅_T'^i}_i=1^L_1 adjacently. We then extract overlapping patches {𝐳̅_T'^i}_i=1^L_2 from 𝐳̅_T' using a shifted window, 
and finally, the reverse diffusion process continues to restore locality for realistic image generation.









 §.§ Comprehensive Structure Denoising






To generate a higher-resolution image 𝐈̅_0 ∈ℝ^H̅×W̅× 3, one could infer from higher-resolution latent noise 𝐳̅_T ∈ℝ ^h̅×w̅× c <cit.>, but this leads to significant peak GPU memory usage. Another line employs a shifted window to sample overlapping patches {𝐳̅_T^i}_i=1^L from 𝐳̅_T where 𝐳̅^i_L ∈ℝ^h × w × c and perform patch-wise inference <cit.>. We show our comprehensive structure denoising facilitates the sampling process and reduces the required number of patches.






As detailed in Sec. <ref>, the shifted window technique yields L = (h̅-h/d_h+1)·(w̅-w/d_w+1) patches. In contrast, we propose setting L_1 = h̅·w̅/h · w, corresponding to the target scale. For example, with h = w = 128, d_h = d_w = 64, and h̅ = w̅ = 384, the shifted window generates 25 patches, while ours requires only 9 patches. Our method relies on two simple intuitions: (1) Each patch focuses on denoising the entire structure, adhering to the same textual directive and producing similar content, unlike the shifted window that recovers partial details. (2) A minimum of L_1 = h̅·w̅/h · w patches suffices to reassemble a higher-resolution latent 𝐳̅_T ∈ℝ^h̅×w̅× c.



As shown in the top-left of Fig. <ref>, we sample non-overlapping patches {𝐳̅_T^i}_i=1^L_1 from the Gaussian noise and conduct patch-wise denoising as t = T → T':



    𝐳̅^i_t-1 = √(α_t-1/α_t)𝐳̅^i_t + (√(1/α_t-1 - 1) - √(1/α_t - 1))·ε_θ(𝐳̅^i_t,t,τ_θ(𝒫)).






  §.§.§ Pixel Interaction.




While Eq. (<ref>) generates latents from the same textual description 𝒫, it only ensures images related to the same subject, not containing similar content. We therefore introduce spatial pixel interaction among distinct patches. As shown in the top of Fig. <ref>, before denoising, we randomly exchange pixels from different patches with the same spatial coordinate (x, y) as:


    (𝐳̅_t^1)_x, y, :, (𝐳̅_t^2)_x, y, :, ..., (𝐳̅_t^L_1)_x, y, : = randperm({𝐳̅_t^i}_i=1^L_1),


where randperm() returns a random permutation of its input sequence, enabling pixels to contribute to the denoising of other images and promoting similarity in content generation across patches. Sec. <ref> studies the details.




 §.§ Specific Detail Refinement





After steps t = T → T', comprehensive object structures emerge within low-resolution patches {𝐳̅^i_T'}_i=1^L_1. Our approach strategically reassembles pixels from these patches to form a target higher-resolution latent representation. We then refine specific details through the remaining steps, t = T' → 1, ultimately obtaining a higher-resolution latent 𝐳̅_0 ∈ℝ^h̅×w̅× c to decode the final clean image.




  §.§.§ Pixel Relocation.



We previously established the viewpoint that a minimum of L_1 = h̅·w̅/h · w low-resolution patches {𝐳̅^i_T'}_i=1^L_1, where 𝐳̅^i_T'∈ℝ^h × w × c, would be adequate for constructing a higher-resolution latent 𝐳̅_T'∈ℝ^h̅×w̅× c. In pursuing this objective, as shown in the middle of Fig. <ref>, we reassemble pixels from the low-resolution latents {𝐳̅_T'^i}_i=1^L_1 to form a higher-resolution latent, prior to carrying out specific detail refinement. This process can be expressed as:


    (𝐳̅_T')_x,y,: = (𝐳̅^ m · w_s+n+1_T')_m + 1, n + 1, :, 
                           m =(x -1) % w_s , 
                           n = (y-1) % h_s,



where h_s = h̅/h, w_s = w̅/w, and the “%” computes the remainder. 

We relocate the identical spatial pixels of {𝐳̅_T'^i}_i=1^L_1 in adjacent regions of 𝐳̅_T'. For instance, the first pixels of low-resolution patches are placed in the pixels situated in the top-left corner of the higher-resolution image. The consequent higher-resolution latent 𝐳̅_T' is thus featured with a complete integration of structural information.






Following the acquisition of the reassembled higher-resolution latent 𝐳̅_T', as shown in the bottom of Fig. <ref>, we proceed to extract overlapping patches {𝐳̅_T'^i}_i=1^L_2 using a shifted window, following MultiDiffusion <cit.> in Sec. <ref>. As we advance through time steps from t = T' → 1, we continue patch-wise denoising with Eq. (<ref>), averaging overlapping regions. This stage refines specific details in the high-resolution latent structure, following principles in Fig. <ref>.







§ EXPERIMENTATION




 §.§ Experimental Setting




We conduct experiments using the pretrained SDXL <cit.> where T=50 defaultly to show CutDiffusion's simplicity, efficiency, cheapness, and performance. We evaluate inference time and GPU memory consumption on a single 3090 GPU.

Given CutDiffusion's tuning-free nature, we compare it with SDXL-DI <cit.>, Attn-SF <cit.>, ScaleCrafter <cit.>, MultiDiffusion <cit.>, and DemoFusion <cit.>. 


We employ three metrics for quantitative evaluation: FID (Frechet Inception Distance) <cit.>, IS (Inception Score) <cit.>, and CLIP Score <cit.>. FID_r and IS_r compare high-resolution generated images to real images but require resizing to 299^2. Following <cit.>, we crop and resize 10 local patches at 1× resolution to compute FID_c and IS_c. CLIP score evaluates cosine similarity between image embeddings and text prompts. We use 10,000 Laion-5B <cit.> images as real images and 1,000 text prompts for CutDiffusion to generate high-resolution images.






 §.§ Method Comparison


In this subsection, we show almighty capabilities of our CutDiffusion method w.r.t. method simplicity, inference time, GPU cost and generation performance.



  §.§.§ Method Simplicity.




It is important to note that our CutDiffusion renders MultiDiffusion <cit.> a special case when T' = T, under which circumstances no comprehensive structure denoising is applied. We merely modify the sub-patch approach as random Gaussian sampling, incorporating additional pixel interaction and relocation operations. Consequently, CutDiffusion retains the simplicity of MultiDiffusion while simultaneously reducing the number of patches required in the initial stage. Compared to more complex third-party engagements, such as fine-tuning <cit.>, low-resolution image generation <cit.>, and progressive upscaling <cit.>,  our CutDiffusion method provides a more streamlined and straightforward process for generating higher-resolution images.










  §.§.§ Inference Time.



The results in Table <ref> align with our conclusion in Table <ref> that direct higher-resolution inference methods typically deliver faster speeds. An exception to this trend is ScaleCrafter <cit.>, whose use of larger kernels impacts inference speed, particularly in ultra-high-resolution settings. Despite adopting a patch-wise inference approach, our proposed CutDiffusion method achieves rapid inference speeds. These speeds are not only comparable to those achieved by direct inference methods <cit.> but are also significantly faster than those achieved by patch-wise counterparts <cit.>. For instance, CutDiffusion is three times faster than DemoFusion <cit.>. 

CutDiffusion's speed advantage stems from its need for fewer inference patches during comprehensive denoising, unlike MultiDiffusion's many sub-patches. DemoFusion's prolonged inference time is due to its progressive upscaling strategy requiring more diffusion steps.












  §.§.§ GPU Cost.




Table <ref> shows minimal GPU memory needs for image extrapolation methods. Direct methods have faster inference but higher GPU demands, especially at increased resolutions <cit.>. Patch-based methods, like CutDiffusion, offer consistently lower peak memory usage, making them more accessible <cit.>. Besides, CutDiffusion's reduced patch count in comprehensive structure denoising mitigates the long duration of GPU running <cit.>.








































  §.§.§ Generation Performance.




We further present visual results for qualitative comparison and metric performance for quantitative comparison, showcasing the strong generation capabilities of our proposed CutDiffusion.


















Qualitative Comparison. 

We present qualitative comparisons under different aspect ratios (1:1, 1:2, and 2:1) and extrapolation ratios (2×, 4×, and 9×). In Fig. <ref>,[Specific prompts and additional results are provided in the Appendix.] SDXL-DI <cit.> exhibits severe repetition and structural distortion. Despite adjusting the attention scaling factor, Attn-SF <cit.> struggles with the same issues. Although ScaleCrafter <cit.> generates non-repetitive images, local details like human hands and panda feet lack fidelity. MultiDiffusion <cit.> suffers from repetitive and inconsistent content due to insufficient global structural information. DemoFusion <cit.> occasionally produces repeated small objects, such as the small panda beneath the tree trunk. Owing to the collaborative two-stage generation process, CutDiffusion results in high-quality, higher-resolution images.



Quantitative Comparison.



Table <ref> demonstrates that CutDiffusion outperforms other image extrapolation methods, achieving the best FID_r scores at resolutions of 1024×2048 and 2048×1024. Although CutDiffusion's FID_c and CLIP score are marginally lower than MultiDiffusion's <cit.>, it is important to note that FID_c indicates detail generation ability. MultiDiffusion generates local details throughout the entire process, while CutDiffusion does so only in the second stage, resulting in slightly inferior detail. Nevertheless, in Fig. <ref>, MultiDiffusion exhibits significant object repetitions despite its higher CLIP and FID_c. Given both quantitative and qualitative results, CutDiffusion surpasses MultiDiffusion. 



Progressing to the resolutions of 2048× 2048 and 3072×3072, CutDiffusion seems to achieve the second-best. A closer look at Fig. <ref> reveals that DemoFusion's skip connection generates a 3072×3072 image with the same content as a pre-trained 1024×1024 image <cit.>. This implies that DemoFusion's performance advantage is due to the low-resolution rather than its inherent capabilities. To confirm this, we interpolate images from 1024 × 1024 to 3096 × 3096 resolution, obtaining the best FID_r (58.50) and IS_r (17.38). Thus, we conclude that mimicking the low-resolution distribution yields a higher FID score from the pre-trained SDXL model. However, as shown by Fig. <ref> and Fig. <ref>, this approach leads to repetition issues, and even lacks high-resolution details for interpolation. Excluding unfair comparison with DemoFusion, our CutDiffusion attains the best.









 §.§ Ablation Study



We shift our focus to ablate CutDiffusion, simple for its single hyper-parameter T'. We also ablate the pixel interaction. All ablations are conducted using prompt 𝒫 = “A photo of cat.” with 2048×2048 resolution under different random seeds.




  §.§.§ Ablations on Hyper-Parameter.


CutDiffusion cuts denoising process into two stages: comprehensive structure denoising and specific detail refinement, regulated by hyper-parameter T'. A higher T' means more steps to detail refinement and less to structure denoising, and vice versa. Fig. <ref> shows T' = 50 guides CutDiffusion to resemble MultiDiffusion <cit.>, causing repeated objects; T' = 1 emphasizes structure denoising, leading to a clear structure but vague details. Visual assessment indicates that T' = T/2 = 25 strikes the optimal balance.













  §.§.§ Ablations on Pixel Interaction.


As outlined in Sec. <ref>, our pixel interaction guarantees images not only pertain to the same subject but also share similar content. Fig. <ref>(a) substantiates this assertion. The pixel interaction ensures the sub-patches depict similar cats, culminating in a high-quality, high-resolution image. Conversely, the absence of pixel interaction results in sub-patches featuring distinct cats, thereby compromising the generation of high-resolution images.



Another potential way to ensure consistent content involves sampling a single Gaussian noise instead of L_1 = 4, creating four copies in the step T', and executing pixel relocation for a higher-resolution latent 𝐳̅_T'. However, as illustrated in Fig. <ref>(b), this method fails due to the copy operation. The copy operation generates identical pixels in neighboring areas, disrupting the Gaussian distribution and causing irregular denoising during specific detail refinement stages.







§ LIMITATION AND FUTURE WORK


Our CutDiffusion method has limitations: (1) Similar to other methods, the quality of high-resolution images generated by CutDiffusion is somewhat dependent on the pretrained diffusion model. (2) The second stage of CutDiffusion still necessitates as many patches as existing methods, hindering further acceleration.


Future work may investigate higher-resolution generation in a non-overlapping patch-wise fashion, with the potential to enhance inference speeds further.







§ CONCLUSION


We have proposed a mighty diffusion extrapolation method, CutDiffusion, which streamlines and accelerates the process while enhancing performance and affordability. CutDiffusion cuts the standard patch diffusion process into an initial comprehensive structure denoising and a subsequent specific detail refinement. It only modifies the sub-patch sampling approach, integrating pixel interaction and relocation in the respective stages. Unlike existing methods that excel in specific areas, CutDiffusion highlights in all perspectives of simple method construction, fast inference speed, cheap GPU cost and strong generation performance.
















splncs04









§ MORE QUALITATIVE RESULTS


In Fig. <ref>, Fig. <ref> and Fig. <ref>, we present an extensive collection of visually compelling results generated by our innovative CutDiffusion method at various higher-resolution levels, showcasing the effectiveness and versatility of our approach.




§ PROMPT LIST


We provide specific prompts we use to generate Fig. <ref> in the main paper.


    
  * A painting of a beautiful graceful woman with long hair, a fine art painting, by Qiu Ying, no gradients, flowing sakura silk, beautiful oil painting.
    
  * Emma Watson as a powerful mysterious sorceress, casting lightning magic, detailed clothing.
    
  * A cute panda on a tree trunk. 
    
  * A Portrait of Albus Dumbledore. 
    
  * The Great Wall of China winding through mist-covered mountains, captured in the delicate brushwork and harmonious colors of a traditional Chinese landscape painting.
    
  * RAW photo of a mountain lake landscape, clean water, 8k, UHD.



We provide specific prompts we use to generate Fig. <ref> in the main paper.


    
  * A fantasy forest.
    
  * Watercolor Painting, Handmade, Gift, Wall Hanging, Home Decor, Landscape, Nature, Size/ Height 12.9 Inches X Width 10 Inches.
    
  * Snow-capped mountain.
    
  * A picturesque mountain scene with a clear lake reflecting the surrounding peaks.
    
  * The vast starry sky.



We provide specific prompts we use to generate Fig. <ref> in the main paper.


    
  * A corgi wearing cool sunglasses. 
    
  * An astronaut riding a pig, highly realistic dslr photo, cinematic shot.
    
  * A squirrel eating an acorn in a fores.
    
  * A rustic wooden cabin nestled in a snowy forest.
    
  * A young badger delicately sniffing a yellow rose, richly textured oil painting.
    
  * A fox peeking out from behind a bush.
    
  * A rabbit is skateboarding in Time Square.



We provide specific prompts we use to generate Fig. <ref> in the main paper.


    
  * A cute teddy bear in front of a plain white wall, warm and brown fur, soft and fluffy.
    
  * Primitive forest, towering trees, sunlight falling, vivid colors.
    
  * A Dog is skateboarding on the snowy mountain.
    
  * Cute adorable little goat, unreal engine, cozy interior lighting, art station, detailed’ digital painting, cinematic, octane rendering.
    
  * Monster Baba yaga house with in a forest, dark horror style, black and white.
















