LM4LV: A Frozen Large Language Model for Low-level Vision Tasks
Low-level vision;Large Language Model;Self-supervised Learning
Yes
No ethics review needed.
None
(1) It proves frozen LLM can solve low-level vision task
(2) The ablation experiment is sufficient to answer that the processing of low-level information is not due to the trainable linear layer, but the text pre-training plays a role
This paper analyzes the reasons why most MLLMs cannot handle low-level features and then puts forward a simple solution. It uses MAE fine-tuning with Image reconstruction tasks to save low-level features. After that, it is then interesting to find that frozen LLM can understand low-level features, which taps into the LLM's uncanny ability to understand visual features.
vision encoder selection is relatively small, exploring more vision encoders will be more convincing
Yes
No ethics review needed.
1. Why fine-tune MAE to perform image reconstruction, instead of using MAGE or MaskGIT? MAGE and MaskGIT use the MIM training paradigm to perform image reconstruction. Will they achieve better performance than MAE?
2. In sec. 5.2, the authors claim they randomly initialize the LLM. After the random initialization, is the LLM frozen or fine-tuned? It would be interesting if the LLM is fine-tuned but still underperforms the frozen LLM counterpart.
+ The proposed method is simple and easy to understand.
+ This work is the first to use LLM for low-level vision tasks.
+ Some conclusions in the paper are interesting. E.g. MAE visual tokens are robust to rotation.
This paper proposes a novel framework, named LM4LV, which applies MLLM to low-level vision tasks. Through preliminary experiments, the authors highlight the advantages of MAE in preserving detailed visual features. Therefore, MAE is used as the visual encoder in LM4LV. Keeping the LLM and visual encoder frozen, the training of LM4LV can be conducted without multimodal data. Extensive experiments are conducted on various low-level vision tasks and different vision modules, to verify the low-level vision task capability of LLM.
- It would be helpful if the authors could specify the number of visual tokens generated by MAE for each image. Moreover, the discussion about computation cost is missing.
- While in sec. 4.1, the authors use MAE-r model as a baseline. Since MAE-r is trained for image reconstruction only, the baseline is not very strong. To establish a stronger baseline, the authors can consider adding the linear adapters into MAE-r, and train the linear adapters for image restoration tasks. Ideally, the linear adapters should be trained separately for each image restoration task, with the same recipe used for training LM4LV.
- Lack of comparison with traditional low-level vision methods. I'm not an expert in low-level vision tasks, so a dedicated part in the related works section would be helpful. Moreover, I don't see any comparison between the proposed LM4LV framework and the previous low-level vision methods. 
- The presentation contains several typos. In line 018 "we purpose LM4LV" and line 039 "purpose to use LLMs", should the word "purpose" be "propose"? In line 471 "(Amid et al., 2022) requires", the citation format can be corrected. In line 914, "ativation" might be a typo.
Yes
No ethics review needed.
NA
Pro:
1. The paper tackles an interesting research direction by investigating whether frozen LLMs can handle low-level vision tasks without multi-modal training, addressing a significant gap in current MLLM research.
2. The authors propose LM4LV, an efficient framework that achieves impressive results across multiple low-level vision tasks using only two trainable linear layers while keeping the LLM frozen.
3. The paper provides thorough empirical analysis through comprehensive ablation studies, particularly demonstrating the necessity of auto-regressive generation, the importance of choosing appropriate vision modules, and the effectiveness of each module, which helps validate the framework's design choices.
The authors introduce LM4LV, a framework that enables frozen LLMs to handle low-level vision tasks (like image denoising and deraining) without requiring multi-modal data or prior training. Through comprehensive experiments, the authors demonstrate the framework's effectiveness by showing significant improvements over baselines and reveal two crucial design choices: the necessity of auto-regressive generation and the importance of vision module selection.
Cons:
1. The paper said, “Furthermore, we cancel the causal attention mask and the ROPE position embedding in the forward process, as they are not the common practice for vision modules.”. However, the ROPE and it variant 2D-ROPE are widely used in large vision transformers (e.g., EVA). This sentence needs revision, and additional experiments with position embeddings would strengthen the analysis.
2. The paper does not explore different LLM variants and sizes. While testing very large LLMs may be resource-intensive, experimenting with smaller or comparable models (like LLaMA-3.2-3B or Qwen2.5 series) would provide valuable insights into how model architecture and scale affect performance.

Suggestion:
1. The paper would benefit from including comparisons with state-of-the-art (SOTA) results from mainstream methods for each low-level vision task. The readers do not expect the method to outperform SOTA methods, while a clear comparison might be helpful for the readers to understand the position of the paper.
2. It would be valuable to investigate the effectiveness of non-autoregressive language models like BERT for low-level vision tasks. While the paper's experiments show that removing auto-regressive generation from LLMs significantly degrades performance, it alters the pre-trained model. Testing with models that are inherently non-autoregressive (like BERT) could provide clearer insights into whether the auto-regressive nature of LLMs is truly essential for these tasks or if alternative architectures might be viable with proper adaptations.
Yes
No ethics review needed.
This paper study the use of LLM for low-level vision tasks, which is under-explored. However,  the motivation to use LLM for these tasks are a bit weak, and the model struggles to preserve high-frequency details in degraded images, suggesting room for further exploration in loss functions or architectural enhancements.
1. The paper addresses an under-explored area in multimodal research by extending LLMs to low-level vision tasks without requiring multimodal training.

2. Comprehensive experiments cover diverse low-level tasks, including denoising, deblurring, and pepper noise removal.
The paper presents LM4LV, a framework that enables a frozen large language model (LLM) to handle low-level vision tasks, such as denoising and deraining, without any multimodal data or prior. By combining a masked autoencoder (MAE) with a frozen LLM and two linear adapters, LM4LV demonstrates non-trivial capabilities in various low-level image processing tasks, thus bridging a gap between large vision-language models (LVLMs) and low-level vision.
1. The paper could benefit from a broader comparison with other non-LLM approaches, particularly in vision-language domains, to better position LM4LV’s relative advantages.

2. The motivation to use LLM for these low-level vision tasks, instead of task-specific models, are still not clear.
Yes
No ethics review needed.
1. In the line 54-57, paper claims that "Enabling MLLMs to process low-level features can lead to a more fine-grained understanding of images and better control in the process of image generation." It's there any fine-grained qualitative or qualitative results to prove that? 

2. I can't understand the Fig.3, is there lack of labels?

3. For the auto-regressive process, the image tokens a limited in the <img> and </img>, so how to control the number of image tokens to make it same with the original image?

4. What is the order of auto-regressive, as row or column? Is there any difference when use different order to generate the image tokens?
1. LM4LV explores the potential of LLM in handling with the vision data without any multi-modal data, which can provide some references.

2. Detailed appendices provide the analysis and the process of implementation. It's easy to follow it. 

3. The paper is well written and easy to understand.
The paper proposes a framework called LM4LV to explore the ability of vanilla Large Language Model in handling with the low-level vision task. LM4LV utilizes the fine-tuned MAE to extract the image features, and take two linear layers as the bridge from vision to language. A paired decoder is used to transform the output tokens to images. It shows competitive quality results in denoising, deblurring, deraining and so on, but the effective in spatial operations like rotation and flipping is ordinary.
1. Contribution. The contribution is limited. LM4LV relies on the existing auto-regressive ability from the training process in LLM. It only constructs a linear layer as the bridge to align the vision semantic space and language semantic space to predict the next token.

2. Soundness. The soundness of the paper is fair. It only utilizes the LLM as a next token predictor, and not explores another way to process vision information by LLM directly. 

3. The qualitative results are not fair. The comparisons with other methods should cover the other denoising, deblurring, deraining methods like MAXIM [1], Restormer [2]

4. The novelty is limited. The framework is composed with a fine-tune MAE and a LLM.

[1] Maxim: Multi-axis mlp for image processing, CVPR 22
[2] Restormer: Efficient transformer for high-resolution image restoration, CVPR 22
LLM can operate on low-level visual features without any multimodal data or alignment
@inproceedings{
anonymous2024lmlv,
title={{LM}4{LV}: A Frozen Large Language Model for Low-level Vision Tasks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ZAx5DxAucB},
note={under review}
}
The success of large language models (LLMs) has fostered a new research trend of multi-modality large language models (MLLMs), which changes the paradigm of various fields in computer vision. Though MLLMs have shown promising results in numerous vision-language tasks such as VQA and text-to-image, no work has demonstrated how low-level vision tasks can benefit from MLLMs. We find that most current MLLMs are blind to low-level features due to their design of vision modules, and thus are inherently incapable of solving low-level vision tasks. In this work, we purpose **LM4LV**, a framework that enables a FROZEN LLM to solve a range of low-level vision tasks without any multi-modal data or prior. This showcases the LLM's strong potential in low-level vision and bridges the gap between MLLMs and low-level vision tasks. We hope that this work can inspire new perspectives on LLMs and a deeper understanding of their mechanisms.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Low-level vision Large Language Model Self-supervised Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1ad0ef4c66b45b198167de2c32b48064854c35cf.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
LM4LV: A Frozen Large Language Model for Low-level Vision Tasks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission