CMC-Bench: Towards a New Paradigm of Visual Signal Compression
Image Compression;Large Multimodal Model;Image Quality Assessment
Yes
No ethics review needed.
1. The method for obtaining the compression rate (CR) values in Section 3.2 is unclear. Different codecs and rate points are expected to yield significantly varied results.

2. The reported -FR and -NR values are the weighted average of multiple metrics, but it is unclear how the proposed weighting ensures a reasonable assessment.

3. In Fig. 6, the gap between the upper and lower bounds of most variation bars exceeds 1, with some even approaching 2. Given that the MOS scale ranges only from 1 to 5, is such a large gap reasonable? Can reliable conclusions be drawn from this? It is also unclear how the rankings in Fig. 6 were determined given the substantial variation in MOS scores.

4. It is unclear how this paper deal with the randomness in CMC-based codecs.

5. According to the visualization results shown in Figs. 8-13, although the decoded images appear perceptually good, they differ significantly from the input images. Moreover, the RD curves suggest that CMC-based codecs are effective only at extremely low rates and cannot be extended to higher rates. Even if CMC-based codecs outperform traditional codecs in some objective evaluations, do they truly have practical applications in real-world compression scenarios? Even if CMC-based codecs outperform traditional codecs in some objective evaluations at extremely low rates, do they really have practical applications in real-world compression scenarios?

6. It appears that all the data used for training and evaluating the FR/NR quality indicators are generated using GPT-4 (OpenAI, 2023) and 12 different T2I models. However, this does not include images generated by other methods, such as traditional codecs or any learning-based codecs. Wouldn't the indicators trained on this data be biased?
1. The idea of conducting a benchmark to evaluate CMC-based codecs is valuable.
2. This paper provides a comprehensive set of experiments studying various combinations of I2T and T2I models.
This paper introduces CMC-Bench, a benchmark designed to evaluate ultra-low bitrate image compression using a Cross Modality Compression (CMC) approach with Image-to-Text (I2T) and Text-to-Image (T2I) models. To assess the coding performance of CMC-based codecs, four collaboration modes are presented along with two indicators: consistency and perception. Experimental results show that some combinations of existing I2T and T2I models outperform traditional codecs in terms of some quality metrics at ultra-low bitrates.
1. This paper does not propose a new model but instead (1) creates a mixed dataset by collecting images from several existing datasets and (2) combines existing I2T and T2I models to evaluate their coding performance. The technical novelty of this paper is therefore questionable.
2. The reported -FR and -NR values are the weighted average of multiple metrics, but it is unclear how the proposed weighting ensures a reasonable assessment.
3. In Fig. 6, the gap between the upper and lower bounds of most variation bars exceeds 1, with some even approaching 2. Given that the MOS scale ranges only from 1 to 5, is such a large gap reasonable? Can reliable conclusions be drawn from this?
4. Although the authors claim that some combinations of existing I2T and T2I models outperform traditional codecs at ultra-low bitrates, this is only demonstrated for specific objective functions. A MOS comparison and subjective evaluation between traditional codecs and CMC-based codecs are not provided.
5. PSNR result is not reported. 
6. The RD curve of GPT-4o + RealVis in Fig.7 looks very weird.
7. It appears that all the data used for training and evaluating the FR/NR quality indicators are generated using GPT-4 (OpenAI, 2023) and 12 different T2I models. However, this does not include images generated by other methods, such as traditional codecs or any learning-based codecs. Wouldn't the indicators trained on this data be biased?
Yes
I will have no ethics concerns if all the interviewers are fairly paid or rewarded when they give subjective annotations.
No ethics review needed.
As mentioned above, I am wondering is there any specific example that proves the proposed benchmark dataset can help resolve the inconsistency issue of cross-modality compression?

Except the issues I mentioned in the Weakness section, I also wonder whether there is any bias during the human annotation. Some experimental details should be added to ensure the subjective scores are reliable and unbiased.
I appreciate the effort that the authors put in annotating subjective scores and building such a relatively large dataset for image compression.
This paper builds a benchmark dataset for Cross Modality Compression (CMC), which aims at achieving much better compression at extremely low bitrates. This benchmark covers tens thousands of image samples and evaluates on 6 Image2Text models and 12 Text2Image models for CMC task. Human annotations are given to evaluate the subjective preference of compression results. There are four modes tested for cross modality compression in this paper, including text-conditioned compression (Text-mode) pixel-conditioned compression (Pixel-mode) traditional codec-based generative compression (Image-mode), and a combination of them all (Full-mode). After analyzing the collected dataset, this paper concludes that without dedicated training for compression, combinations of several advanced I2T and T2I models have already surpassed traditional codecs in multiple aspects.
1. Throughout the paper, I did not find any non-trivial insight from the proposed benchmark for the whole community. The main conclusion of the proposed benchmark assessment is that: in the task of ultra-low rate compression, codecs based on generative models (e.g., T2I models) could perform better than traditional codecs in many aspects. However, actually it is almost a common sense in the compression community, I would suggest the authors to think deeper so that it can get some valuable conclusions, such as “which type of T2I architecture would perform better for CMC task, auto-regressive or denoising diffusion?”

2. Some sentences in this paper are overclaimed. For example, in abstract, it is written as 
“CMC has certain defects in consistency with the original image consistency with the original image and perceptual quality. To address this problem, we introduced CMC-bench…”
How could the benchmark address the issue of inconsistency in CMC. There is no technical algorithm proposed in this paper to address this issue, neither any direct insight in the proposed benchmark dataset that guides the community to address the inconsistency issue.

3. I have some doubts on the reliability of the proposed benchmark dataset. For example, in figure7, the performance of GPT-4o +RealVis fluctuates in FID-bpp. Sometimes FID increases and sometimes FID decreases as bitrates increases.
Yes
No ethics review needed.
1.	For each column in Table 1, it is recommended to provide the full name and corresponding unit (e.g., "dis") in the main text or caption.

2.	According to the description in the text, the radar chart on the right side of Figure 5 represents "{GPT-4o + 12 different T2I models}," but it appears there are only six methods shown on the chart. The description is somewhat confusing.

3.	In Section 4.2, the first sentence of the first paragraph should be changed to “Figure 5 shows the performance of 6 I2T models as encoders and 12 T2I models as decoders in image compression.”

4.	The three colors in the histogram in Figure A3 are not all explained within the figure.

5.	I am curious about how the authors balanced consistency and perceptual quality when establishing the evaluation metrics, or if any experiments or considerations were conducted regarding this. Also, did the authors perform any ablation studies to determine the final weights of each evaluation metric?

6.	It seems the authors could further adjust the organization of the main text and appendix to prioritize more important information in the main body.

7.	The authors have indeed undertaken extensive integration and experiments related to previous data and evaluation methods in establishing the CMC-Bench standardized evaluation platform for I2T-T2I. However, this work seems more like an integration of previous research efforts, including the sources of the new dataset and the evaluation metrics, which resemble normalized methods from prior I2T-T2I tasks. Are the authors concerned that this paper’s originality might be questioned?
This paper introduces a new benchmark platform for Cross Modality Compression (CMC) called CMC-Bench, offering a fresh approach and guiding direction for future ultra-low bitrate image compression. The paper includes 58,000 images and 160,000 human annotations, offering a large-scale and comprehensive evaluation. Furthermore, authors presents rigorous comparative experiments with existing traditional codecs, demonstrating the advantages of CMC, which is highly persuasive. Although it does not propose a complete solution, the dataset and evaluation metrics provided in this work are likely to inspire further research.
The paper introduces CMC-Bench, a benchmark for Cross Modality Compression (CMC) in image compression. Utilizing advancements in Large Multimodal Models (LMMs), the authors explore an Image-to-Text-to-Image (I2T-T2I) approach, where visual signals are first transformed into text and then regenerated into images, achieving compression rates as high as 10,000 times compared to traditional codecs. However, the CMC paradigm faces challenges in consistency and perception fidelity at ultra-low bitrates. CMC-Bench evaluates 6 I2T and 12 T2I models across 58,000 images, covering consistency and perception dimensions with 160,000 subjective human annotations, providing a comprehensive dataset for future codec development. Experimental results indicate that CMC has the potential to outperform traditional codecs but requires further refinement in handling Screen Content Images (SCI) and balancing consistency and perception.
Some details require further discussion, such as the balance between consistency and perceptual quality. Additionally, certain expressions and the structure of the paper would benefit from further refinement and polishing.
Yes
None
No ethics review needed.
Improvement:

(1) I hope the author can add the std of each T2I model. Currently, I only see the mean result on the leaderboard. As you know, the performance of T2I is unstable. Adding std can make the benchmark more complete.

(2) I suggest the author open source all data. Including raw data, test code, and subjective quality labels. As a benchmark, LLM developers need an end-to-end testing pipeline.
(1) Sufficient and comprehensive experiments, namely benchmarks for most current T2I and I2T models. The author has extensively considered mainstream LLMs and explored their optimal combination.

(2) Comparison with traditional codecs such as VVC. Experiments show that CMC has advantages over 7 of the 8 fidelity/quality indicators. This proves the significance of CMC.

(3) Exquisite visualization. Figures 1/2 are both relatively intuitive.
This pioneering work explores the application of LLMs in the context of codec, namely cross-modality compression (CMC). The overall paradigm is innovative, as a combination of I2T+T2I pipeline. Experimental results indicate a satisfying fidelity and extremely low bitrate, which has significant advantages over traditional compression paradigms.
(1) Ultra-low bitrate needs a better definition. Why 1,000 times? Thousand-fold compression is indeed the highlight of this article, but has there been work in the past that can achieve this bpp? This needs to be further explained, as this is the core novelty of this article.

(2) The author experimented with 12 T2Is but only 6 I2Ts. One or two more I2Ts should be added to the benchmark list.
Using LMMs for image compression
@inproceedings{
anonymous2024cmcbench,
title={{CMC}-Bench: Towards a New Paradigm of Visual Signal Compression},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=foKwWau15m},
note={under review}
}
Ultra-low bitrate image compression is a challenging and demanding topic. With the development of Large Multimodal Models (LMMs), a Cross Modality Compression (CMC) paradigm of Image-Text-Image has emerged. Compared with traditional codecs, this semantic-level compression can **reduce image data size to 0.1% or even lower**, which has strong potential applications. However, CMC has certain defects in consistency with the original image and perceptual quality. To address this problem, we introduce CMC-Bench, a benchmark of the **cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models for image compression**. This benchmark covers 18,000 and 40,000 images respectively to verify 6 mainstream I2T and 12 T2I models, including 160,000 subjective preference scores annotated by human experts. At ultra-low bitrates, this paper proves that the combination of some I2T and T2I models has surpassed the most advanced visual signal codecs; meanwhile, it highlights where LMMs can be further optimized toward the compression task. We encourage LMM developers to participate in this test to promote the evolution of visual signal codec protocols.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Image Compression Large Multimodal Model Image Quality Assessment
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/78d0b15f9c1f4b4b827038a3ada02579902037cb.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
CMC-Bench: Towards a New Paradigm of Visual Signal Compression
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission