ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment
data centric machine learning;autoformalization;large language models;reasoning
Yes
No ethics review needed.
### Minor comments

Figure 3, page 5:  
The color bar is labeled "Gzip Alignment" instead of
"ZIP-FIT-Alignment" from Algorithm 1; it may be confusing to readers.

Figure 3, page 5, line 231:  
Please mention also in the figure caption that the test loss is
calculated on ProofNet data.
The paper is concise, sound, well written, and the experimental section shows promise for the method, especially with regard to other embedding-free methods.

The conceptual simplicity combined with the empirical results of the method is an especially strong point of the work.
The paper introduces a new data selection mechanism based on text
compression distances. The concept of using compression methods for
deep learning follows several modern practical results and theoretical
motivations that language modeling is fundamentally based in text
compression. The method's conceptual simplicity combined with strong
empirical results make it stand out as a modern way for filtering for
aligned data.
Ideally, it would be shown how the size of $n$ (i.e., number of
samples from the target domain $p$) influences the performance of the
method. If it is possible to pick $n$ just sufficiently large enough,
it would greatly improve the computational efficiency of the method
for large target datasets.

Experiments in other domains would be really nice to better
demonstrate the generalization capabilities of the method. Possibly
there is data that is not well-suited to compression and accordingly
ZIP-FIT, or where the data's compression factor varies too much
between samples?
Yes
No ethics review needed.
1. Could you provide further insights into how ZIP-FIT might perform with data that have higher variability and diverse syntactic structures, such as conversational datasets?
2. Can you clarify the theoretical basis for using gzip compression over other compression methods that might exploit redundancy differently? Would alternative compression algorithms affect the performance of ZIP-FIT?
1. ZIP-FIT’s embedding-free approach is a refreshing deviation from common embedding-based methods, offering a novel solution by leveraging gzip compression. The concept of using normalized compression distance (NCD) as an alignment metric is insightful and could inspire future research in embedding-free methodologies for various data selection tasks.
2. The empirical results support the claims, showing that ZIP-FIT achieves faster convergence and better performance than established methods. The experiments were conducted on both AutoFormalization and code generation tasks, demonstrating ZIP-FIT's versatility across different domains.
3. The paper is well-structured, with a clear exposition of the algorithm, experimental setup, and results. The figures effectively illustrate the performance benefits of ZIP-FIT.
4. ZIP-FIT could represent a significant advancement in data selection for machine learning, particularly in computationally constrained environments. Its potential to optimize model fine-tuning with minimal resource requirements makes it highly applicable for real-world use cases, especially in domain-specific and low-resource applications.
This paper introduces ZIP-FIT, an embedding-free data selection method leveraging gzip compression to measure the alignment between training and target domains. Unlike existing approaches that rely on neural embeddings, ZIP-FIT uses a computationally efficient compression-based alignment metric, enabling faster data selection while maintaining high relevance to the target task. Empirical evaluations demonstrate ZIP-FIT’s superiority over baselines DSIR and D4 in AutoFormalization and code generation tasks, achieving significantly faster convergence and lower cross-entropy loss with reduced computational costs. ZIP-FIT’s promise lies in its scalability and effectiveness, particularly in low-resource settings, where traditional embedding-based methods may be impractical.
1. While ZIP-FIT achieves excellent results on the tasks tested, its reliance on gzip compression may limit its effectiveness in complex semantic domains where relationships are nuanced and less compressible. Embedding-free approaches, while efficient, may not be ideal for tasks that require deep semantic understanding or complex syntactic relationships.
Yes
No ethics review needed.
As specified in the "Weaknesses" section:
- What is the score of the fine-tuned LLM using ZIP-FIT on benchmarks like HumanEval and PubMedQA compare to LLMs fine-tuned without using ZIP-FIT?
- How does ZIP-FIT compare to prior method like https://arxiv.org/abs/2405.00705 in terms of both running time and final model score?
- This paper is well-presented and well-motivated.
- Studying computation-efficient methods for data selection in LLM instruction fine-tuning is a promising research direction.
- The proposed ZIP-FIT is intuitive and easy to follow.
- The proposed approach bypasses the need for LLM forward computation to obtain embeddings, making it computationally efficient.
- The presented experimental results seem promising.
This paper proposes ZIP-FIT, an efficient, embedding-free method for selecting high-quality, domain-specific fine-tuning data for language models (LMs). Prior methods often rely on computationally expensive neural embeddings or classifiers to filter aligned datasets, while those based on N-gram similarity may lack the structural depth needed for complex tasks like code generation. In contrast, ZIP-FIT leverages gzip compression to evaluate data alignment with target domains, based on the idea that compression algorithms encode information in a way similar to neural networks. The ZIP-FIT approach eliminates the need for LM forward passes to obtain embeddings, making it efficient and particularly suitable for low-resource environments. Experimental results show that ZIP-FIT outperforms prior data selection methods, such as DSIR and D4, as measured by test loss.
- [Major] The proposed method seems very simple and straightforward; using a gzip-style method to embed data appears to be a relatively standard approach.
- [Major] All experimental results are based on test loss, which may not be very reliable. It would be essential to conduct evaluations on some standard benchmarks, such as HumanEval and MBPP for code evaluation, to demonstrate the scores the model can achieve.
- It is unclear how the proposed ZIP-FIT compares to prior, more complex data selection methods in terms of both running speed and final model quality (e.g., [1]), aside from deduplication approaches like D4.
- [Minor] The paper seems to be written somewhat in rush, the figure quality of Figure 2 does not seem to be very high.

[1] https://arxiv.org/abs/2405.00705
Yes
No ethics review needed.
(1) Could the authors provide additional evidence to support the claim that gzip is effective in capturing syntactic and structural relationships in textual sequences?

(2) Would the authors be able to demonstrate the effectiveness of their approach using evaluation metrics beyond cross-entropy test loss, and compare it to relevant baselines, such as those mentioned earlier?

(3) Could you provide more insight into why D4 was excluded from the code generation experiments, and specifically how it affected model performance?
(1) Problem Significance: The author tackles a crucial problem in low-resource settings, addressing the challenge of fine-tune data selection without relying on GPU-intensive and embedding-based methods. This is a highly relevant and impactful research direction.

(2) Innovative Filtering Criterion: The authors' inspiration from gzip compression methods has led to the proposal of a novel and intriguing selection criterion. This approach is not only interesting but also demonstrates out-of-the-box thinking, making it a notable contribution to the field.
This paper introduces an innovative, embedding-free data selection method for efficient fine-tuning of large language models. Drawing inspiration from gzip compression techniques, the authors propose utilizing Normalized Compression Distance as a metric to filter and prune fine-tuning datasets. The authors conduct a comparative analysis with prior embedding-free methods, originally designed for filtering pre-training datasets, on Autoformalization and Python coding tasks.
(1) **Inadequate Baselines**: The authors propose a data selection method for model alignment, but only compare it with prior works such as DSIR and D4, which were primarily designed for data selection during the pre-training phase. A more comprehensive literature review on data pruning methods for model alignment is lacking, including embedding-based methods [1], LLM model response metrics [2], Gradient-based metrics [3],Quality metrics judged by LLMs [4], inference loss on evaluation sets [5].

(2) **Evaluation Metrics**: The authors primarily use test data cross-entropy loss as the evaluation metric, results are thus not surprising given that the data selection method uses the test data to anchor the selection criteria. However, the authors do not compare their results with widely accepted metrics in the research community for the studied downstream tasks, such as:

(a). Autoformalization: proof success rates on miniF2F [6,7]

(b). Python coding: functionality pass rates (pass@k on HumanEval) based on unit-tests [8,9]


(3) **Clarifications on Motivation**: In Section 2.3, the authors argue that n-grams fail to capture syntactic or structural relationships within the data, while hypothesizing that gzip does. However, this hypothesis is not supported by theoretical or empirical evidence, weakening the motivation for the proposed approach. It is also not compared on if the proposed approach is better or worse than high-resource methods, such as embedding-based methods.

References: 

[1] DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection

[2] From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning

[3] LESS: Selecting Influential Data for Targeted Instruction Tuning

[4] Alpagasus: Training a better alpaca with fewer data

[5] Instruction Mining: Instruction Data Selection for Tuning Large Language Models

[6] Autoformalization with Large Language Models

[7] LEGO-Prover: Neural Theorem Proving with Growing Libraries

[8] Evaluating Large Language Models Trained on Code

[9] Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
use gzip to select optimal data for code and autoformalization
@inproceedings{
anonymous2024zipfit,
title={{ZIP}-{FIT}: Embedding-Free Data Selection via Compression-Based Alignment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=4JBEpP6eRS},
note={under review}
}
Selecting high-quality, aligned fine-tuning data is crucial for improving the downstream performance of language models (LMs). Automatic data selection in these scenarios is challenging and often inefficient due to previous approaches relying on neural embeddings or limited n-gram representations to identify aligned datasets. In addition, traditional data selection methods often focus on increasing the size of the training data, making them computationally expensive to use and data inefficient. In this work, we introduce ZIP-FIT, an embedding-free, data-efficient selection framework that leverages gzip compression to measure the alignment between training data and target domains. We show that ZIP-FIT significantly outperforms two leading baselines, DSIR and D4, in selecting high-quality data for ProofNet, a formal mathematical dataset, and HumanEval, a benchmark for code generation tasks. Specifically, ZIP-FIT demonstrates a computational speed advantage, performing data selection up to  65.8\% faster than DSIR and achieving its lowest cross-entropy loss up to 85.1\% faster. Our findings suggest that ZIP-FIT offers a scalable and adaptable approach for data selection, enabling more precise fine-tuning for code generation domains. By demonstrating that embedding-free data selection can outperform established methods like DSIR and D4, our research opens new avenues for optimizing model training, thereby enhancing the effectiveness and efficiency of machine learning workflows.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
data centric machine learning autoformalization large language models reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a706bb5d7465dc4f64fab987f6d76be8adb5a408.pdf
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission