From Noise to Factors: Diffusion-based Unsupervised Sequential Disentanglement
Sequential Disentanglement;Deep Learning;Generative Models
Sorry for the inconvenience caused. I have updated my review.
Update of the review.
Reviewer FuiQ,

Can you correct your reviews ASAP to allow the authors sufficient time to respond?

AC
Dear Reviewer FuiQ,

We would like to bring to your attention that the review provided for our submission appears to pertain to a different submission. We kindly request that you revise your response to accurately reflect the content of our submission. We sincerely appreciate the time and effort you have dedicated to reviewing our work and thank you for your attention to this matter.

Best regards,
The Authors
Yes
No ethics review needed.
1. It's not required, but would be good if the authors include the derivation of the training objective from the score matching objectives.
2. Would like to see the comparison and results on longer sequences.
1. The PGM factorization is natural and elegant, and the network design are simple but effective.
2. Quantitative results shows good reconstruction and swapping. Qualitative results also demonstration good disentanglement and generation capacity.
The paper proposes a probabilistic model for disentangling static and dynamic features from a series of videos, expanding on the idea of DiffAE. Specifically, the authors expand DiffAE to denoising a series of images, and factorize $z_{sem}$ to $s$ and $d$ which stands for static and dynamic features. A simplified score matching object is also utilized and empirically shows reasonable results for reconstruction. Moreover, the authors experimented on various disentanglement-oriented experiments including conditional/unconditional and zero-shot swapping, and also latent probing.
1. Experiments are mostly shown on short sequence face/motion datasets, where the static feature is appearance and dynamic feature being the keypoints/expressions. Under the framwork that the author proposed, ideally longer sequences should facilitates the disentanglement of $s$ and $d$.
2. It's not clear if the effect of disentanglement comes mostly from the factorization, or from the fact that $d$ has small size and $s$ has larger size.
Yes
No ethics review needed.
* It would be great to see the method shine in more (real-world) domains, for example audio (e.g. speaker vs content) disentanglement, multi-object video (controlling aspects of objects as is shown in this work, but then on the object rather than video level), multi-view video (e.g. controlling viewpoint)

Relevant works:
* View point synthesis: "DORSAL: DIFFUSION FOR OBJECT-CENTRIC REPRESENTATIONS OF SCENES"
* Object disentanglement: "Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models"
* Audio benchmark: "Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation"
* Generally well-written and clear.
* Strong empirical results on the chosen benchmarks.
* The proposed solution to obtaining disentangled representations is interesting in that it relies on the structure of the model rather than auxiliary losses.
The paper is concerned with the problem of learning disentangled generative models for sequential (e.g. video) data. It proposes a method based on the Diffusion VAE framework that splits the latents into two groups - static (does not vary with frame) and dynamic (varies per frame). The proposed method is benchmarked on a number of (single) person-centric datasets.
**Major**
* The paper touches on the problem of disentangled representation for sequential data in general. However, the presented experiments are limited - they only make use of simple (single person) video data. This does not give a fair picture of the methods usefulness and applicability because (i) it targets only a single modality; and (ii) because even within this modality the model targets very structured datasets (e.g. single view, single object).

**Minor**
* The Introduction section gives a skewed picture of the state of image generation with VAEs and GANs (e.g. citing works from 2016 and 2018) to support the claims of GANs being unstable and VAEs blurry. A large body of work has been introduced since the introduction of these methods that makes GAN training very stable in practice, and improves VAE generation quality.
* Please double-check citations. For example, line 651 has an incorrect author name.
Yes
No ethics review needed.
Please refer to weakness.
1. The paper proposes to deal with a practical, meaningful and challenging problem setting, that is unsupervised sequential disentanglement.
2. The paper presents a diffusion sequential disentanglement autoencoder (DiffSDA) framework for the unsupervised sequential disentanglement problem.
3. The paper conduct experiments on three practical datasets, and demonstrate the effectiveness of the proposed DiffSDA compared with given baselines.
In this paper, the authors present a diffusion sequential disentanglement autoencoder (DiffSDA) for unsupervised sequential disentanglement. The proposed DiffSDA is based on DiffAE but improves DiffAE from three perspectives. Moreover, the authors introduce three new datasets for the evaluation and verify the effectiveness of the proposed DiffSDA by comparing with several SOTA baselines.
1. One major concern of the work is on the technical novelty. The disentanglement function is decoupled with the diffusion module but is done in the semantic encoder. However, the way of extracting the dynamic and static information in the semantic structure, i.e., using LSTM to explore the temporal relations and its last hidden to calculate the static one, has been well studied in existing studies, e.g., disentangled sequential VAE and S3VAE. To this reviewer, this work is a conditional diffusion with the extracted dynamic and static feature representations as conditions. In this sense, the technical significance of the proposed DiffSDA is limited.
2. The paper misses some important related works. Using diffusion for animation generation, e.g., Animate Anyone, MagicAnimate, not only transfers face/appearance but also changes the motion to follow the driven sequence. There are also a lot of image2video works that generate (zero-shot) target video sequences using a single target image and a source driving videos. Although these methods do not explicitly use the terminology ‘disentanglement’, they have similar application scenarios with the paper.   
3. It is always encouraging to introduce and test new datasets. However, unsupervised sequential disentanglement is a well-defined/studied problem setting, and testing on some benchmark datasets (maybe naïve) is necessary. The authors may refer to the datasets used in disentangled sequential VAE and S3VAE. 
4. To show the effectiveness of the disentanglement, it is necessary to show the reconstruction results with only one latent factor, e.g., keeping dynamic one and set the static one as zeros. Moreover, a project page with some video results is encouraged, which clearly helps the readers to understand your results as well as to compare with the other baselines.
5. Since the method is built on DiffAE, the reviewer wonders how the performance of the proposed DiffSDA compared with DiffAE where frame-by-frame swap can be done.
6. To show the superoirty of the proposed method in the fast sampling, complexity analysis or time-cost analyses, e.g. rtf, need to be done.
Yes
Although I did not flag this paper for additional ethical review, I believe there are potential ethical concerns, especially given the qualitative examples presented. I am noting this here to raise the authors' awareness.
No ethics review needed.
Please see my detailed Weaknesses section. While there are several minor issues throughout the manuscript, I have highlighted what I consider to be the major concerns. Overall, I believe the paper is not yet ready for publication.
The paper extends the static diffusion space decoders to their dynamic sequential version, which can be applied to video editing tasks. It is an interesting application scenario.
The paper proposes a decoder framework for the latent spaces within diffusion generative models. By incorporating sequential aware neural networks (LSTM) into the proposed DiffSDA method, it is able to extract some sequential features that allow for video editing tasks.
- Above all, I believe that it is not appropriate to show a video example with gender being modified in the first illustration figure, the authors can choose many other alternative examples to convey the same idea other than this particular case. On top of that, the results shown in this first figure are some qualitative examples of "video editing" (at least to me), and there is a misalignment between what is “disentanglement” and “editing or semantic manipulation tasks” within this context.
- A major weakness of this paper is the lack of a clear problem definition. The abstract and introduction start with an ambitious, high-level goal—representation disentanglement in an unsupervised learning setting—but the focus quickly shifts to generative modeling across a range of architectures, including VAEs, GANs, and DMs. There is a significant gap in explaining how this broad question is formulated within a generative framework.
- Building on my previous point, the authors may risk overclaiming the contribution of the proposed framework. It functions as a post-hoc decoder that applies additional disentanglement to the diffusion latent space, rather than as an unsupervised learning framework that directly addresses disentanglement challenges.
- Even within this diffusion generative context, many recent works have demonstrated the intrinsic ability of diffusion models to learn disentangled features, such as [a,b,c]. The authors fail to discuss and/or compare these related works in the paper.
- I see a strong implicit assumption in the problem formulation in Section 4.1, where the authors assume the data distribution can be factorized into a "state-independent distribution density of static (time-invariant) and dynamic (time-variant) factors," treating them as independent variables. This assumption seems questionable and requires, at a minimum, some justification. Additionally, Eq. (4) is somewhat unclear and unprofessional—what is V? It is mentioned in Line 185 but not defined.


[a] "Diffusion models already have a semantic latent space." ICLR 2023.

[b] “Boundary Guided Learning-Free Semantic Control with Diffusion Models”. NeurIPS 2023.

[c] “Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions.”. 2024
@inproceedings{
anonymous2024from,
title={From Noise to Factors: Diffusion-based Unsupervised Sequential Disentanglement},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=YFKH1vO0W2},
note={under review}
}
Unsupervised representation learning, in particular, sequential disentanglement, where the goal is to learn disentangled static and dynamic factors of variation, remains a significant challenge due to the absence of labels. Existing models, based on variational autoencoders and generative adversarial networks, achieved success in certain domains, but they often struggle with disentangling sequences, especially when dealing with real-world complexity and variability. Further, there is no real-world evaluation protocol for assessing the effectiveness of sequential disentanglement models. Recently, diffusion autoencoders have emerged as a new promising generative model, offering semantically rich representations by gradual noise-to-data transformations. Despite their advantages, these models face limitations: they are non-sequential, fail to disentangle the latent space effectively, and are computationally intensive, making them difficult to scale to sequences. In this work, we introduce our diffusion sequential disentanglement autoencoder (DiffSDA), a novel approach effective on real-world visual data and accompanied by a new and challenging evaluation protocol. DiffSDA is based on a new probabilistic modeling and is implemented using latent diffusion models and efficient samplers, facilitating processing of high-resolution videos. We test our approach on several real-world datasets and metrics, and we demonstrate its effectiveness in comparison to recent state-of-the-art sequential disentanglement methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Sequential Disentanglement Deep Learning Generative Models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d002feb59ce1f8dac8a9a6a7334318b76f57519b.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
From Noise to Factors: Diffusion-based Unsupervised Sequential Disentanglement
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission