Improved Training Technique for Latent Consistency Models
Consistency Model;Diffusion Model
Yes
No ethics review needed.
1. The pseudo-huber loss also seems to be able to be characterized similarly to the Cauchy loss by adjusting c to adjust the degree of suppression of outliers. I'd like to know what the authors think.

2. ﻿ The batch for consistency model training is usually large. What is the increase in total time when using POT to calculate optimal transport for noise data coupling?
1. The analysis and motivation of this manuscript is sound, and to the best of my knowledge, the effect of the potential spatial anomalies they first revealed on consistent model training.

2. Each of the proposed techniques was well ablated.

3. From the visualizations provided by the authors (Figures 4,5), several of the proposed techniques do significantly improve the visuals of data such as Celeba-HQ.
The authors analyzed statistical differences between pixel space and potential space, and found that potential data often contained anomalous values of altitude pulses, which significantly reduced iCT performance in potential space. To solve this problem, they proposed the Cauchy loss, which effectively mitigated the effects of anomalies. Additionally, they introduce diffusion losses at early time steps and employ optimal transport coupling to further improve performance.
1. As an empirical paper, the authors seem to have compared only with iCTs that reproduce in hidden spaces. In fact, there have been many improvements on the consistent model of lifting hidden spaces, such as [1, 2], with which authors should compare or discuss.

[1] Hyper SD: Trajectory Segmented Consistency Model for Effective Image Synthesis

[2] Trajectory consistency disruption

2. The authors' experiments are limited to some simple modal datasets such as the FFHQ, CELEBA-HQ datasets. Empirical evidence without multi-modal datasets weakens persuasion.
Yes
No ethics review needed.
In my experience, some losses that are robust to outliers usually slow down the convergence of the model, due to the fact that these losses are equivalent to reduced gradients in large error regions, which may lead to slower convergence in early training than standard losses. Have the authors observed this phenomenon in their experiments?
1. The motivation of this article are very sound and the introductory section does a good job of presenting the motivation and contribution of the article. Given the popularity of potential spatial diffusion modeling, this research may have important implications for the practical application of accelerated diffusion.

2. The authors found significant differences between latent space training and pixel space training. The latter is usually normalized and the former may have impulse noise. This factor may seem simple, but it has been long neglected.
This paper presents a series of techniques for enhancing the training of consistent models in potential space, including Cauchy loss, early diffusion loss, and optimal transport coupling between noise and samples.
1. The link to TD training in DQN in section 4.1 seems somewhat redundant. There is no evidence that DQN has a similar problem with impulse noise. And there is no solution proposed by the authors that is not derived from it.

2. The authors mention in the introduction that the aim is to address the potential proliferation of large-scale applications such as text-to-image or video generation. However, instead of using a text-to-image model like LCM, the authors ended up experimenting on some simple pattern datasets, which is a minor shortcoming of the experimental evaluation.
Yes
No ethics review needed.
Please see the Weakness.
1.	The writing of the paper is clear.
2.	The paper analyzes the reasons for the poor performance of improved consistency training in latent space.
3.	The results of the proposed method achieved a great improvement than improved consistency training in latent space.
The paper proposes a new strategy for consistency training of latent consistency models. It analyzes the reasons for the poor performance of improved consistency training in latent space, particularly addressing the issue of outliers by suggesting the use of Cauchy loss as a remedy. Additionally, by using x0 as the ground truth when t is small, the paper reduces the errors caused by model fitting. Compared to improved consistency training, the proposed method shows significant improvements in latent space.
1.	Equation (9) should be ||f(xt)-x0||^2.
2.	Please briefly explain how the constant in Equation (11) is determined.
3.	Batch size has a significant impact on generative models, especially in consistency training; please include the batch size in Table 1.
4.	Please include a comparison of the results from latent consistency distillation, including the resources used for training.
Yes
No ethics review is needed.
No ethics review needed.
1. What is the correlation between the major contributions?
2. Could you please provide more quantitative comparisons of your models with LCM and original CM?
3. Why do you choose such a schedule in equation (11)?
1. The performance reported in the paper is good compared to the self-implemented iCT.
2. The motivation that delivers Cauchy loss, optimal transport, adaptive scaling-c scheduler, and non-scaling layernorm to handle highly impulsive outliers is convincing.
This paper targets the efficient training problem for consistency models on latent space. They find that the highly impulsive outliers in the latent data significantly degrade the performance of iCT. Therefore, they employ a series of training tricks, including Cauchy loss, diffusion loss at early timesteps, optimal transport, adaptive scaling-c scheduler, and non-scaling layernorm to improve the performance.
1. The soundness of the contribution in this paper is not good. The major contributions (Cauchy loss, optimal transport, adaptive scaling-c scheduler, and non-scaling layernorm) are mostly engineering training tricks. The correlation between the major contributions is not so strong, which makes the entire paper look like an A+B+C work. Since CM is already a complex model with so many hyper-parameters and training tricks, introducing more training tricks into CM doesn't appeal to me enough.

2. The comparison of this paper with SOTA models is not sufficient. While the authors mention LCM in the paper, they do not compare their model with LCM in Table 1. This paper is mainly based on the impulsive outliers observed from their self-implemented iLCT, which is not so convincing to me. Does LCM also show impulsive outliers? Is it possible to adopt your training tricks to LCM? In addition, since iCT is not open-source, I recommend the authors show more comparisons of your model and the original CM. For example, what about L2 loss and LPIPS loss in Table 2 (b)?

3. The ablation study to the hyper-parameters is not sufficient. Please provide more discussion on why you choose such a schedule in equation (11). 

4. The presentation of this paper is somehow repetitive and complicated. For example, Lines 164-175 are quite similar to the abstract and introduction part. And sections 4.4 and 4.6 are all descriptions with pure texts. Introducing more equations or figures may help to improve the presentation of this paper.
@inproceedings{
anonymous2024improved,
title={Improved Training Technique for Latent Consistency Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PQjZes6vFV},
note={under review}
}
Consistency models are a new family of generative models capable of producing high-quality samples in either a single step or multiple steps. Recently, consistency models have demonstrated impressive performance, achieving results on par with diffusion models in the pixel space. However, the success of scaling consistency training to large-scale datasets, particularly for text-to-image and video generation tasks, is determined by performance in the latent space. In this work, we analyze the statistical differences between pixel and latent spaces, discovering that latent data often contains highly impulsive outliers, which significantly degrade the performance of iCT \citep{song2023improved} in the latent space. To address this, we replace Pseudo-Huber losses with Cauchy losses, effectively mitigating the impact of outliers. Additionally, we introduce a diffusion loss at early timesteps and employ optimal transport (OT) coupling to further enhance performance. Lastly, we introduce the adaptive scaling-$c$ scheduler to manage the robust training process and adopt Non-scaling LayerNorm in the architecture to better capture the statistics of the features and reduce outlier impact. With these strategies, we successfully train latent consistency models capable of high-quality sampling with one or two steps, significantly narrowing the performance gap between latent consistency and diffusion models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Consistency Model Diffusion Model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e08bbd18d3d64125bd56dbc68ff84bec60c25288.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/1b731c0f9914b34a60d7308182466e939b835ef8.zip
Improved Training Technique for Latent Consistency Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission