Conditional density estimation for video prediction with score-based models
temporal prediction;diffusion models;denoising;conditioning;occlusion boundaries;cue combination
Yes
No ethics review needed.
It will be very helpful to include a discussion that draws the boundary between the proposed framework and well-known diffusion models.

A demonstration of the proposed framework on high resolution videos will help to strengthen the experiments. If this is not possible, a discussion about the scalability might be beneficial.
The main contribution of this paper is mostly conceptual. The key ideas of (1) employing a direct regression approach for learning score-based models and (2) the sampling strategy from the learned denoising function are both intriguing. It is perhaps a bit surprising to see how they can work even on some toy data. 

The proposed framework presents an interesting and significantly simplified alternative to existing diffusion models.

The paper is well-written overall. The results, including a detailed analysis of the learned representations, are elaborated.
The paper presents a simplified diffusion-based framework for modeling conditional density, with demonstrations in video prediction. The key idea, derived from the empirical Bayes formulation of score-based models, involves learning a denoising function that implicitly infers noise levels and removes noises of arbitrary magnitude from an input. This denoiser yields an estimation of a family of score functions across noise levels. This framework thus removes the time axis in a standard diffusion model, and pursues a direct regression approach, allowing easy analysis of the learned representations. This framework is evaluated on synthetic videos, as well as natural image sequences.
Despite the conceptual novelty, the practicality of the proposed framework is somewhat questionable. Considering the problem of video prediction, it is probably fair to say that the proposed framework provides at best an alternative solution to diffusion models. While diffusion models have demonstrated impressive results for video generation and prediction, the proposed framework is solely demonstrated on “toy” data (small scale, low resolution synthetic and real videos). It is not clear if the proposed framework can scale up to larger datasets or higher resolution videos.
Yes
No ethics review needed.
- equation (1) is not clear due to index s having two distinct roles (also, it would be helpful to clarify whether s>t is feasible)
- it would be good to clarify whether f(y,c) is E[x|y,c] as in (2) or f(y,c) is E[x|y,c] - y as in (7)
- forecasting future frames in video is an important problem
- generative modelling is an appropriate tool for the task at hand due to ability to account for multimodal future
The manuscript formulates probabilistic forecasting of the next video frame as a generative modeling task. The proposed method allows to recover plausible instances of the next frame by iteratively sampling a denoising deep generative model through Tweedie's formula. In the experiments, the denoising model is a U-Net with up to \tau=2 conditioning frames. The model optimizes the L2 reconstruction loss which corresponds to log p(x|y,c). The experiments have been performed on a synthetic dataset (moving leaves) and DAVIS32.
- the proposed method appears quite straight-forward and in line with previous recent work in the field
[a] Gabriel Loaiza-Ganem, Brendan Leigh Ross, Luhuan Wu, John P. Cunningham, Jesse C. Cresswell, Anthony L. Caterini. Denoising Deep Generative Models. ICBINB 2022

- the manuscript does not report quantitative comparison with related work in the field; for instance, it would be insightful to report MS-SSIM and LPIPS on Cityscapes and KITTI, as in [b] and references therein.
[b] Yue Wu, Qiang Wen, Qifeng Chen. Optimizing Video Prediction via Video Frame Interpolation. CVPR 2022: 17793-17802.
Yes
N/A
No ethics review needed.
- The temporal sampling of an image sequence is an artifact of the technology; how does this impact the modeling proposed.  The basic technical premise is disarming: "The next frame in an image sequence is a single event with several possible outcomes..."  The next frame in an image sequence has very very many possible outcomes; it's also not clear what "an event" means (in the context of the earlier part of this bullet).  It would be helpful if the paper included a discussion of the impact of this discrete sampling of continuous time along with a discussion of the potential impact this has on the method's performance.
- The paper motivates a strong problem about the lack of transparency in next-frame generation (or general video generation) papers.
- The base formulation for the next prediction is sound.
This paper describes a probabilistic formulation for next-frame-prediction.  The paper develops statistical machinery based on learned denoising functions in what appears to be a sampling approach, although the details are vague and hard to follow.  The method is tested on a synthetic, procedural dataset and a small natural image sequence dataset.
- The development of the technical ideas in the paper are difficult to follow, moving from an initial problem formulation to denoising without any reasonable discussion.   Then the discussion jumps around with relatively basic statistical machinery without giving any details of the actual method.   Better description of what the novelty in the method would be helpful.  More concrete relationship between the current work and the related art is very important to help the reader understand the current method and its contributions.

- The overall approach seems similar to other probabilistic generative methods, even those noted in the paper.

- The evaluation approach is not well described or convincing.  There are not comparisons to component methods or baselines.  It would improve the paper if comparative baselines were included.  The notion of occlusion, which may be useful is introduced seemingly out of nowhere.  It would improve the paper if there is more discussion motivating this evaluation.  However, evaluation should also describe general generation techniques.  The natural image sequences tested are few.  It would improve the paper to demonstrate capability on establish video generation benchmarks with established video generation evaluation protocols, beyond PSNR.  The details of the networks used are unclear.  It would improve the paper to have a thorough network description, even if in the appendix.

- (Minor) Typos exist in the paper; e.g., line 89 focuese
Yes
No ethics review needed.
Could the authors clarify how their approach to conditional density estimation for video prediction differs from existing score-based or diffusion models?

What specific insights or improvements does this work offer over current state-of-the-art methods in video prediction?

How does the analysis of the network's adaptive behavior contribute to our understanding of video prediction models?
The paper defines score matching formulation for video prediction in a really accurate way. They also show how adding noise is helpful both in training and inference.
The paper proposes a score-based model for conditional density estimation tailored to video prediction. The authors demonstrate that single-step denoising results in blurred predictions, highlighting the necessity of iterative denoising—a standard approach in score matching, diffusion, and flow matching models—to capture sharper, more plausible outcomes. Through a straightforward dataset and network, they further illustrate the distinct contributions of past frames and the current noisy frame in shaping accurate predictions.
It’s challenging to identify the novel contribution of this paper, as it seems to be positioned somewhat independently, with limited engagement with prior work on video prediction, particularly with diffusion models. More explicit connections to existing approaches and an analysis of what this model uniquely offers in comparison would strengthen the work.
Probabilistic video prediction with conditional denoiser networks relies on representations that automatically adapt to the signal.
@inproceedings{
anonymous2024conditional,
title={Conditional density estimation for video prediction with score-based models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=mHkbi3XM58},
note={under review}
}
Temporal prediction is inherently uncertain, and representing the ambiguity in natural image sequences is a challenging high-dimensional probabilistic inference problem. For natural scenes, the curse of dimensionality renders explicit density estimation, let alone integration, statistically and computationally intractable. We describe an implicit regression-based framework for learning and sampling the conditional density of the next frame in a video given the past few observed frames. We show that sequence-to-image deep networks trained on a simple resilience-to-noise objective function extract adaptive representations for temporal prediction. Synthetic experiments demonstrate that this score-based framework handles occlusion boundaries:  unlike classical methods that average bifurcating temporal trajectories, it chooses among likely trajectories, selecting more probable options with higher frequency. Furthermore, local linear analysis of networks trained on natural image sequences reveals that the representation automatically weights predictive evidence by reliability, which is a hallmark of probabilistic inference.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
temporal prediction diffusion models denoising conditioning occlusion boundaries cue combination
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/af4382e25a63ab54adfa44b79e7b06faa142a735.pdf
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Conditional density estimation for video prediction with score-based models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission