In recent years, the field of computer vision and natural language processing has witnessed remarkable progress with the emergence of deep generative models. Among the various types of deep generative models, diffusion models have gained attention as a promising method that address the limitations faced by pre-existing generative models such as Generative Adversarial Networks (GANs) . In particular, diffusion models have demonstrated outstanding performance in image generation tasks and have been leveraged in the development of cutting-edge text-toimage generators such as Dall-E , Midjourney 9 , and Stable Diffusion . These methods enable generating user-desired images based on the given input prompts (e.g. "draw me an armchair in the shape of avocado") with the help of the diffusion model.