L-C4: Language-Based Video Colorization for Creative and Consistent Colors
Video Editing;Diffusion Model;Colorization
Yes
No ethics review needed.
Please refer to weakness.
1. The paper is well-organized and well-written.
2. The design of the components in the proposed method generally makes sense.
This paper proposes a language-based video colorization model and employs temporally deformable attention and fusion module to handle inter-frame color consistency and instance-aware colorization.
Method Issues

1.	The proposed method can only process frames with fixed and low resolutions, specifically 256x256, making it incapable of handling high-resolution scenes. The results shown in the paper are also low-resolution, which might obscure some detailed issues. However, diffusion-based methods often struggle with details, so there are doubts about whether the proposed method can handle fine details effectively.

Performance Issues

1.	Lack of comparisons with state-of-the-art (SOTA) colorization methods. The comparisons shown in the paper omit more recent methods, such as ColorMNet[1] and DDColor[2].
2.	Some of the qualitative comparisons in the paper seem unfair and meaningless. For example, Figure 5 presents a comparison between the exemplar-based methods and the proposed text-based method under completely different conditions (e.g., the exemplar shows an orange fruit, while the text description specifies a pink fruit). Such comparisons lack significance, and the pink fruit in the figure also exhibits noticeable color overflow issues, where the pink in the highlight areas turns gray.
3.	Some of the videos in the demo still show color flickering and inconsistency issues, especially in small objects like shoes.
4.	Some of the displayed images still exhibit color bleeding issues.
5.	For diffusion-based methods, different seeds can lead to varying results. Does the proposed method perform consistently across different seeds? If not, what is the success rate of the proposed method?
6.	In video processing, processing speed is also essential. Please provide comparisons with the speeds of other video colorization methods.

[1] Yang, Yixin, et al. "ColorMNet: A Memory-based Deep Spatial-Temporal Feature Propagation Network for Video Colorization." ECCV, 2025.

[2] Kang, Xiaoyang, et al. "Ddcolor: Towards photo-realistic image colorization via dual decoders." ICCV. 2023.
Yes
No ethics review needed.
Please refer to the Weaknesses section
# Strengths

- The design of proposed submodules are reasonable and technically sound
- Extensive experiments demonstrate the effectiveness of the proposed framework
# Summary

The paper proposes a stable diffusion based video colorization framework. The paper leverages the temporal deformable attention for color consistency, cross-modality prefusion for instance-awareness, a skip window design for long-term temporal consistency. Experiments show the proposed method outperforms the selected baselines.
# Weaknesses

- The "Limited creative imagination" in L044 might not be a weakness of exemplar-based video colorization considering the user can use text-based image colorization methods to obtain exemplars
- It seems the paper overclaims its contribution in L091-092. It's unclear how reliable the proposed framework can "assign colors to each instance" considering the proposed framework still uses CLIP-based embeddings
- It seems that the design of LumEncoder is same as the previous works, but the paper does not cite them
- The paper does not mention how the exemplars were selected for the exemplar-based baselines. For L-C4 colorization with user prompts, the exemplar-based baselines might also need to adopt text-based colorization results as exemplar for fair comparisons
- Fig 4 and Fig 5 only contain one frame for each method, which seems inadequate to visualize the visual quality for video colorization. Visible color shifting artifacts can be observed in the supplementary video
- The paper does not mention the detailed evaluation settings: are prompts used in quantitative evaluations, if used, how they are obtained? 
- Diffusion models can generate diverse results. The paper does not mention how results are selected for comparison and does not evaluate the success rate for each generation
- CDC metrics seem to have different scales with previous works
- Considering there is no ground truth for video colorization, using PSNR/SSIM for evaluation seems strange
- The paper does not compare with recent text-based video colorization methods like [1] and [2]
- The scale of user study seems relatively limited
- In Fig 7 right part, the boy's shirt in L-C4 seems suffering from color bleeding issue
- The paper does not mention runtime statistics, how long does the proposed framework take to colorize a video with 300 frames?

- [1] Towards Photorealistic Video Colorization via Gated Color-Guided Image Diffusion Models
- [2] Versatile Vision Foundation Model for Image and Video Colorization
Yes
No ethics review needed.
Please see the weakness section.
1. Innovative Integration of Components: The combination of CMPF, TDA, and CCF addresses key challenges in video colorization, such as instance-aware color assignments and temporal consistency.
2. Comprehensive Evaluation: The paper conducts extensive experiments, including quantitative metrics, qualitative analyses, user studies, and ablation studies, providing a well-rounded assessment of L-C4's performance.
3. Clear Motivation: The authors effectively identify the limitations of existing video colorization methods and articulate how L-C4 overcomes these issues through its novel framework.
4. State-of-the-Art Performance: L-C4 achieves superior results on multiple benchmarks, indicating its effectiveness in real-world applications like film restoration and artistic creation.
The paper introduces L-C4, a language-based video colorization framework designed to assign semantically accurate and creatively flexible colors to monochrome videos based on user-provided language descriptions. The proposed approach leverages a pre-trained cross-modality generative model and integrates three key components:

- **Cross-Modality Pre-Fusion (CMPF)**: Generates instance-aware text embeddings to facilitate precise color assignments based on language inputs.
- **Temporal Deformable Attention (TDA)**: Ensures inter-frame color consistency by dynamically capturing object features across frames.
- **Cross-Clip Fusion (CCF)**: Maintains long-term color consistency in extended video sequences by aggregating information across multiple clips.

Experimental evaluations on DAVIS, Videvo, and an additional VRIPT dataset demonstrate that L-C4 outperforms existing automatic, exemplar-based, and language-based image colorization methods across various quantitative and qualitative metrics. User studies further support the method's effectiveness in achieving temporally consistent and creatively satisfying colorizations.
1. **Incremental Novelty**: While L-C4 combines existing techniques in a new way, components like deformable attention and cross-modality fusion are based on established methods. The paper would benefit from highlighting more unique innovations or providing deeper theoretical explanations for the observed performance improvements.

2. **Model Complexity and Efficiency**: Integrating TDA and CCF adds significant complexity to the pipeline, resulting in longer inference times and larger model sizes compared to baseline exemplar-based methods.

3. **Performance vs. Efficiency Trade-offs**: The paper does not discuss the balance between L-C4's performance gains and its increased computational costs, which is important for evaluating its practicality in real-world applications.

4. **Artifacts from Cross-Clip Fusion**: The cross-clip fusion approach may introduce artifacts or inconsistencies at clip boundaries, especially in videos with rapid motion or sudden scene changes.

5. **Fairness in Comparisons**: Despite the authors' efforts to ensure fair evaluations, comparing different methods remains challenging due to varying inputs and guiding signals.
Yes
No ethics review needed.
- What's the generation process of color-related masks? What if the text prompts are complicated or intricate? One example:
  "The sunset painted the sky with hues of burnt umber, casting a surreal glow over the landscape." Could the model handle these cases successfully, and to what extent will the quality and type of text prompts influence its performance?

- Temporally Deformable Attention (TDA) generates offset estimations for reference points. Would the performance get worse or better if we replace the offset estimation with an off-the-shelf model, such as an optical flow estimation model?
- Each component is well-motivated, with clear explanations of how they address specific challenges in video colorization. The paper is well-written, with logical organization and flow that makes it easy to understand.

- The proposed Cross-Modality Pre-Fusion (CMPF) module enhances text embeddings with instance-aware capability, ensuring that the objects are assigned with desired colors, enabling creative and precise colorization given user input.

- The Temporally Deformable Attention (TDA) and Cross-Clip Fusion (CCF) effectively maintain color consistency across frames in a video. These components help prevent flickering and color shifts, leading to smoother and more visually appealing results.

- The experiments are thourough and well-designed, demonstrating superiority over state-of-the-art approaches.
The paper introduces L-C4, a novel framework for language-based video colorization that guides the colorization process of monochrome videos using natural language descriptions, addressing the primary challenges faced in video colorization tasks: ambiguous color
assignment, limited creative imagination, and vulnerable color consistency. The proposed framework builds upon pre-trained cross-modal generative models to utilize their strong language understanding capabilities. To ensure accurate instance-level colorization, the paper proposes the Cross-Modality Pre-Fusion (CMPF) module to enhance text embeddings with instance-aware capability. Deformable Attention (TDA) block and Cross-Clip Fusion (CCF) are designed to ensure temporal color consistency across long sequences. The paper is well-written with clear motivation and rationales. The experiments are thorough, demonstrating their effectiveness.
- Regarding the generation of instance-aware embeddings, the paper does not fully explain or justify the creation of color-related masks. This might limit the model's applicability in complex cases, such as when dealing with intricate or lengthy textual descriptions.

- The paper lacks discussions on limitations. 

- Some typos: Line 311, "cross-flip" should be "cross-clip".
Yes
N/A
No ethics review needed.
See weaknesses.
1. The proposed method is technically sound. 
2. Three main ablation results in Table 1 show the performance gains brought by the proposed techniques. 
3. The paper is generally in good shape.
4. The results in Table 1 and the video are impressive.
The paper proposes a language-based video colorization method (L-C4) to colorize monochrome videos based on the user-input instructions. The proposed method contains four main components, a cross-modality generative model [28] for language understanding and color modeling, a temporally deformable attention for inter-frame consistency, a cross-modality pre-fusion module for instance-aware text embedding generation, and a cross-clip fusion for long-term color consistency. The proposed method was evaluated on two existing datasets and show better results against several existing colorization methods.
1. Originality: The paper handles an existing task, video colorization, with an existing solution of using user-input languages as guidance [5,6,8,23,36]. The main difference is that while previous language-based methods are proposed for image colorization, the proposed method focuses on videos and considers modeling temporal consistency. However, the introduction does not discuss such differences and highlights the challenges and novelties. Comparing the language-based video colorization method to exemplar-based methods does not make sense, given the existence of language-based image colorization methods.

2. While the paper proposes a language-based video colorization method, the capability of understanding language and colors is from the existing image-based generation model [28]. Despite the proposed three methods (i.e., TDA, CMPF, and CCF), TDA and CCF focus on temporal coherency and CMPF is for text-instance alignment. There are no technical insights regarding color modeling from the colorization viewpoint.

3. The novelty of CMPF and CCF are not explained explicitly.

4. In fact, this paper is to extend a single image-based method [28] to handle the video colorization task. However, the paper does not provide a correct context (how existing methods handle temporal consistency and multi-modal alignment issues and what the true novelties of this work are) for evaluating its idea, and accordingly, the evaluations are not convincing (despite two experiments of L-CAD[4]+DF[20] and L-CAD[4]+temporal attention).

5. The authors claims that "Since the process of retrieving appropriate references can be elaborate, and the retrieved images are generally collected in the wild, exemplar-based video colorization methods may restrict the ability to assign creative colors to instances based on users’ imagination (Fig. 1 second row)". However, it is important to note that exemplar-based methods have the potential to incorporate language-based image colorization techniques, which can enable the assignment of creative colors. This raises the question of whether language-based video colorization methods are indeed necessary, or if the integration of language guidance into exemplar-based approaches could suffice to achieve the desired creative control over colorization. It would be beneficial for the authors to explore and discuss the limitations and advantages of both exemplar-based and language-based methods in the context of creative color assignment. A comparative analysis or experimental validation could strengthen their argument for the necessity of language-based video colorization methods over the adapted exemplar-based approaches.

Quality: (-) Most of the comparing methods are automatic and exemplar-based colorization methods, which are not close to the proposed method compared with those language-based image colorization methods [5,6,8,36,23] (the proposed method relies on the [28], which is also an image-based method). Meanwhile, there are also video diffusion models (as discussed in section 2.3) that study temporal coherency. Comparing the proposed method to (a combination of) language-based image colorization and video generation methods would be more convincing. (-) The proposed method lacks in-depth analysis and ablations, such as the sampling rate and hyperparameter alpha in TDA, the luminance encoder [6], and the hyperparameter N^f in CCF.

Clarity: (-) There are only some careless writings/typos: e.g., Temporal (Line 118) or Temporally (Line 54) Deformable Attention? Fig.12, last case (right): the colors should be red and blue. (-) The information about the training and evaluation datasets for existing methods and the proposed method is not clear, which may undermine the fairness of the comparisons.
@inproceedings{
anonymous2024lc,
title={L-C4: Language-Based Video Colorization for Creative and Consistent Colors},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=lessla98Wp},
note={under review}
}
Automatic video colorization is inherently an ill-posed problem because each monochrome frame has multiple optional color candidates.
Previous exemplar-based video colorization methods restrict the user's imagination due to the elaborate retrieval process. Alternatively, conditional image colorization methods combined with post-processing algorithms still struggle to maintain temporal consistency. To address these issues, we present Language-based video Colorization for Creative and Consistent Colors (L-C4) to guide the colorization process using user-provided language descriptions. Our model is built upon a pre-trained cross-modality generative model, leveraging its comprehensive language understanding and robust color representation abilities. We introduce the cross-modality pre-fusion module to generate instance-aware text embeddings, enabling the application of creative colors. Additionally, we propose temporally deformable attention to prevent flickering or color shifts, and cross-clip fusion to maintain long-term color consistency.  Extensive experimental results demonstrate that L-C4 outperforms relevant methods, achieving semantically accurate colors, unrestricted creative correspondence, and temporally robust consistency.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Video Editing Diffusion Model Colorization
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/fd9a97b08ba3e0a20e69ba2db90b496a74a95546.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/ea2320a353b8a201e3a7ad1aba8f4d9f19a83817.zip
L-C4: Language-Based Video Colorization for Creative and Consistent Colors
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission