Denoising with a Joint-Embedding Predictive Architecture
AIGC;JEPA;Diffusion Model;Flow Matching;Image Synthetic
## Conclusion

We sincerely appreciate the valuable comments from the reviewers. After the paper concludes the double-blind review process, we will open-source all the code and models for D-JEPA, including the code for audio, video, and multimodal generation as mentioned in Appendix G.
Conclusion
## More Experiments on T2I

Recently, we have extended the T2I experiments in section G.2. We built a 2.6B text-to-image model based on D-JEPA, and on the GenEval metrics, we have surpassed the results of the highly representative diffusion model, SD3.0 (4B version) (0.65 vs. 0.64). The detailed metrics are listed in the table below:

|        | #Params | Overall | Single Obj | Two Obj | Counting | Colors | Position | Color Attr |
|:------:|:-------:|:-------:|:----------:|:-------:|:--------:|:------:|:--------:|:----------:|
| SD3.0  |  2.0B   |  0.62   |    0.98    |  0.74   |   0.63   |  0.67  |   0.34   |    0.36    |
| D-JEPA |  2.6B   |  0.65   |    0.98    |  0.80   |   0.59   |  0.87  |   0.22   |    0.47    |
| SD3.0  |  4.0B   |  0.64   |    0.96    |  0.80   |   0.65   |  0.73  |   0.33   |    0.37    |

It is noteworthy that this is the first work to surpass diffusion models in the T2I task using a next-token prediction-based architecture. This achievement lays a solid foundation for the future design of unified multimodal model architectures.
More Experiments on T2I
## Comprehensive Comparisons with Recent Methodologies

We have noticed that several highly relevant works have been published on ArXiv after the ICLR submission deadline. According to the guidelines, we are not required to compare with these works, but to fully demonstrate the superiority of the D-JEPA architecture, we have conducted a comprehensive comparison here.

The core idea of D-JEPA is to bring the advantages of representation learning into generative tasks, rather than pursuing a unified model for both representation learning and generative tasks, which Reviewer dQp1 has misunderstood. Based on this premise, REPA has made similar discoveries. REPA effectively shortened the training time of DiT by adding additional feature learning tasks. D-JEPA, on the other hand, is a novel architecture that, apart from being used for image generation, has more crucial potential roles. Firstly, it is capable of generating across various continuous modalities (something both MAR and REPA cannot achieve), including audio and video. Secondly, D-JEPA employs a Next-token-prediction approach, laying the groundwork for constructing unified multimodal large models in the future. These two key differences are at the heart of the D-JEPA work. For a better comparison with REPA, we present the results in the table below:

|                 | #Params | #Iters |  FID  |
| :-------------: | :-----: | :----: | :---: |
| SiT-XL/2 + REPA |  675M   |  400K  |  7.9  |
| SiT-XL/2 + REPA |  675M   |   1M   |  6.4  |
| SiT-XL/2 + REPA |  675M   |   4M   |  5.9  |
|    D-JEPA-L     |  687M   |  300K  | 2.32  |

As seen in the table above, D-JEPA-L, with a comparable parameter size, achieves results far superior to REPA with 4M iterations using only 300K iterations. This clearly demonstrates the superiority of the D-JEPA-L architecture.

REPA: Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, [https://arxiv.org/abs/2410.06940](https://arxiv.org/abs/2410.06940)
Comprehensive Comparisons with Recent Methodologies
## Regarding Model Scale and Experimental Results

Both Reviewer gTwo and Reviewer dQp1 have expressed concerns about the fairness of the experimental comparisons in Tab.1. They noted that D-JEPA-H has more parameters compared to MAR-H (1.4B vs. 943M). Consequently, although D-JEPA-H achieved superior results with just 320 epochs of training compared to MAR-H’s 800 epochs (FID 2.04 vs. 2.35), they question the fairness of this comparison.

Firstly, let us clarify the principle behind the D-JEPA model configurations at different scales. To fairly complete the representation learning experiments in Appendix E, we chose to base our work on the existing ViT-B/L/H frameworks. As shown in Table 7, existing tasks focusing solely on representation learning have been based on the standard ViT B/L/H. Thus, our D-JEPA B/L/H models respectively adopt the corresponding ViT B/L/H as the backbone, resulting in parameter sizes of 212M, 687M, and 1.4B. For the MAR series models, MAR-B also uses the ViT-B structure, but MAR-L/H do not follow the standard ViT-L/H, which restricts our ability to maintain parameter consistency.

Secondly, concerning the experiments in Table 1, it should be noted that our D-JEPA-L model already outperforms all previous works (including MAR-H) with 480 epochs of training and 687M parameters. This adequately illustrates the superiority of the D-JEPA architecture, even without considering the results of D-JEPA-H. Nevertheless, we trained the D-JEPA-H model to further demonstrate the scaling laws of the D-JEPA architecture, and unsurprisingly, it achieved even better results. Detailed analysis on this part is provided in lines 375 to 409 of the paper. Below is an excerpt from Tab 1, offering a more intuitive comparison:

|          | #Params | #Epochs |  FID  |  IS   | Pre.  | Rec.  |
| :------: | :-----: | :-----: | :---: | :---: | :---: | :---: |
|  MAR-L   |  479M   |   800   | 2.60  | 221.4 | 0.79  | 0.60  |
| D-JEPA-L |  687M   |   480   | 2.32  | 233.5 | 0.79  | 0.62  |
|  MAR-H   |  943M   |   800   | 2.35  | 227.8 | 0.79  | 0.62  |
| D-JEPA-H |  1.4B   |   320   | 2.04  | 239.3 | 0.79  | 0.62  |
Regarding Model Scale and Experimental Results
# Preface

We would like to extend our heartfelt thanks to each reviewer for their valuable comments. Before we proceed with a detailed, point-by-point response, I hope that the reviewers and the Area Chair could spare a moment to browse through this blog content. It primarily addresses potential misunderstandings regarding our paper and provides comprehensive comparisons with recent methodologies (those published on ArXiv after the ICLR submission deadline).
Preface
Yes
No ethics review needed.
My main concerns with this paper relate to its performance compared to the similar approach, MAR. The absence of ablation studies on model components and evaluation on other datasets makes it challenging to assess the impact of each design choice.
(1) This paper introduces a straightforward and effective approach to enhance generation quality by bridging representation learning and image generation.
(2) In the appendix, the authors offer an in-depth analysis of the model design, including additional insights on representation learning as well as applications in video and audio generation, showcasing the versatility of the proposed methods across multiple tasks.
(3) The presentation is clear, and the ideas are easy to understand.
This paper extends the JEPA framework—a masked-image-based self-supervised representation learning model—to image generation by incorporating an additional MLP head that operates on image patches. This design unifies representation learning and image generation within a single framework, demonstrating improved image generation quality over recent state-of-the-art methods in specific model configurations.
(1) The performance is comparable to similar approaches like MAR, which does not use the JEPA loss. In Table 1, the proposed method requires more training epochs (1400 vs. 800 for D-JEPA-B VS ) to achieve similar results to MAR-B. While D-JEPA-L/H outperforms MAR-L/H, it also involves more parameters. Similar trends are observed in Table 2.
(2) There is no comparison to baseline methods, such as the effect of removing the JEPA loss.
(3) How does the model perform in unconditional generation tasks or on more complex datasets, such as COCO?
Yes
N/A
No ethics review needed.
Listed in Weakness.
1. Suffieicnt experiments, including generative reuslts on Imagenet-256, sufficient abaltion studies and the exps about representation learning.

2. Authors present the effectiveness of D-JEPA  over multi-modalities including videos, images and audio.

3. Authors provide a theretical support about the proposed models.

4. Beyond the generative results, authors provides the empirical study about the linear performance of the proposed D-JEPA with the pixel/latent-level inputs.
The proposed D-JEPA is the combination of Joint-embedding Predictive Architecture (I-JEPA) with MAR's dfiffusion parts, ie, leveraging a feautre from JEPA as the condition of diffusion parts.  Good generative performance is achieved by the proposed model.
And authors provide the exps over multi-modalities.
1. The organization and the structure of the current version should be improved. The writing of chapter 3 is very confusing for me. And some important results and discussions should be re-located in the main paper not the appendix.

2. The novelity is limited:
 It seems that this work just replace the MAGE parts of MAR. The simple combination of I-JEPA and MAR's dfiffusion parts. It doesn't solve the core issue between the gap of representation learning and generative modeling.
Such an archecture cannot bring huge improvement on representation learning with latent-level input (Results in AppendixE over  down-stream tasks are good, just for pixel-level inputs), and the corresponding performance are very poor.
Could you please provide the similar settings for MAR? Use both the pixel-level and latent-level inputs to check the representation performance? I think your design with JEPA cannot solve the inherent limitation (latent input -> bad representation results) of such an TOKEN learning + diffusion framework.

3. The core of I-JEPA is the small-block-wise masks to perform the feature-level augmentation to achieve the better linear performance without the help of data-level augmentation. But in your work, the random mask strategy with high ratio is applied. Could you please provide some empirical evidence and intuitions about such a design？


4. Line 200： “It is important to note that training in latent space is not a necessity for D-JEPA; it can be trained in raw space and still achieve excellent results.” WHERE is the corresponding generative results？ I haven't found it.

5. It is unfair to compare Huge scale models in Table1 and 2. The parameters of the proposed D-JEPA is not aligned with MAR-H. 

6. How about the D-JEPA-H linear and finetuning performance in Table 7/8.

7. Please provide the exps on Imagenet-512x512 to compare with  SOTAs in a more comoutation situation.
Yes
No ethics review needed.
n/a
* Interesting combination of JEPA representation learning with generative AI, showing that representation learning can help generative AI
* Strong SOTA results on ImageNet 256x256, better or equal than MAR
* When applied for representation learning, the context encoder achieves good results on ImageNet classification
* Experiments showing generalization to text conditioned image generation (and not just class conditioned)
* Experiments showing that it works with audio as well, class conditioned video generation, and CFG
The author describes a JEPA based architecture which combines a prediction loss and a diffusion loss.
* The prediction loss learns to predict the representation of the mask tokens
* The diffusion loss learns to predict the raw pixels (or the latent representation of the patch in case of VAE)

The architecture itself is based on a three ViTs:
* The context encoder takes the unmasked tokens and outputs Y
* The feature predictor takes Y, pad it, and learns to reconstruct the full representation Z
* The target encoder takes all tokens (no masking) and produces the target representation Z'
* The prediction loss is a smooth l1 loss between Z and Z'
* The diffusion loss takes Z as input and tries to predict the pixels / latents
* The target encoder is itself an EMA of the context encoder

The paper explores this architecture and shows that:
1) The diffusion loss prevents the collapse of traditional JEPA architectures
2) This architecture is scalable, as the results improves with increased compute budget
3) At inference, instead of predicting all unmasked tokens, we can predict a subset of them, as do MaskGIT, to provide better results
4) They reach SOTA performance on class conditional generation on ImageNet 256x256, equating the results of MAR with CFG
* It would be interesting to have generation results at higher resolution than 256x256 px to see if inference speed suffers when bi-directional attention meets lots of tokens
Yes
No ethics review needed.
1. Could the authors provide an ablation study that isolates the MAE branch to elucidate its specific impact on the AR branch's performance?

2. How does the model's performance on representation learning tasks vary between training in pixel space and latent space, and what does this indicate about the model's ability to serve as a unified architecture?

3. It’s interesting to see from Tables 1 and 2 that D-JEPA has more parameters than MAR. I’m curious—shouldn’t these models have similar architectures? Can the authors explain why D-JEPA is larger? What additional components does D-JEPA include that MAR doesn't?
The ideas tha integrating joint-embedding predictive architectures(e.g. MAE and MIM) with diffusion-based generative modeling makes sense. 

The experiments in the paper are robust, covering a wide range of applications, which showcases the versatility of the D-JEPA model. This extensive experimental validation strengthens the paper's claims about the model's effectiveness across different types of generative tasks.
The paper, D-JEPA, explores a novel approach by merging joint-embedding predictive architectures with generative modeling. It applies masked image modeling as a generalized next-token prediction strategy for autoregressive data generation. D-JEPA employs a combination of diffusion losses and MAE loss to model token distributions effectively. It aims to enhancing generative tasks across various domains like images, videos, and audio. Extensive experiments demonstrate the model's scalability and effectiveness, claiming state-of-the-art performance on ImageNet class-conditional benchmarks.
However, the paper presents a significant methodological oversight—it appears as a mere combination of existing MAE (Masked Autoencoder) and MAR (Masked Autoregressive Model) methods without a critical analysis or ablation study showing how the MAE branch impacts the AR branch. Specifically, the absence of experiments where the MAE branch is removed casts doubt on the incremental benefit of this integration.

The representation learning capability of D-JEPA is also questionable. According to Table 7, the best ImageNet accuracy is obtained when the model is trained on pixel-level data, while the generative experiments focus on a D-JEPA trained on VAE latent space. This discrepancy points to a lack of a truly unified model approach, as the performance seems to depend heavily on the training specifics rather than the model architecture itself.
In this paper, we introduce Denoising with a Joint-Embedding Predictive Architecture (D-JEPA), pioneering the application of JEPA in generative modeling.
@inproceedings{
anonymous2024denoising,
title={Denoising with a Joint-Embedding Predictive Architecture},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=d4njmzM7jf},
note={under review}
}
Joint-embedding predictive architectures (JEPAs) have shown substantial promise in self-supervised representation learning, yet their application in generative modeling remains underexplored. Conversely, diffusion models have demonstrated significant efficacy in modeling arbitrary probability distributions. In this paper, we introduce Denoising with a Joint-Embedding Predictive Architecture (D-JEPA), pioneering the integration of JEPA within generative modeling. By recognizing JEPA as a form of masked image modeling, we reinterpret it as a generalized next-token prediction strategy, facilitating data generation in an auto-regressive manner. Furthermore, we incorporate diffusion loss to model the per-token probability distribution, enabling data generation in a continuous space. We also adapt flow matching loss as an alternative to diffusion loss, thereby enhancing the flexibility of D-JEPA. Empirically, with increased GFLOPs, D-JEPA consistently achieves lower FID scores with fewer training epochs, indicating its good scalability. Our base, large, and huge models outperform all previous generative models across all scales on class-conditional ImageNet benchmarks. Beyond image generation, D-JEPA is well-suited for other continuous data modeling, including video and audio.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
AIGC JEPA Diffusion Model Flow Matching Image Synthetic
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b0befa6bcbfd94dcc389e621a4d55aa367dc9a5b.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Denoising with a Joint-Embedding Predictive Architecture
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission