SCAR: Efficient Instruction-Tuning for Large Language Models via Style Consistency-Aware Response Ranking
Style Consistency;Data Efficiency;LLM Alignment;Fine-Tuning
**Response to Concerns on Dataset and LLM Size Limitations:**

We appreciate the reviewer’s feedback regarding the dataset and model size. However, we validated our method across a range of models and data sources, strengthening the robustness and generalizability of our results. Specifically:

- ***Model Variety and Real-World Relevance:*** Our method is tested across four different base LLMs, including CodeLlama, Llama3, and widely recognized open-source models like OLMO and Starcoder, all of which are commonly used in real-world applications. This diversity of models ensures that our approach is not restricted to a single model setup, enhancing the reliability of our findings.

- ***Dataset Diversity and Scale:*** We validated our selection method on the large-scale open-source dataset, the OLMO tulu dataset, with 320,000 examples, which offers a rich source of real-world data. Our own curated dataset (20,000 and 10,000 examples) for data selection is also comparable in scale to widely used benchmarks like Alpaca (52,000 examples), LIMA (1,000 examples), and Guanaco-Commits (13,000 examples). All these prior datasets have been effectively used to train numerous open-source models. This similarity in dataset sizes underscores that our approach is grounded in scales commonly used in prior studies.

- ***Validation Across Data Curation Methods:*** Our approach is further validated by applying it to both open-source datasets and our own curated datasets, collected by simulating two real-world data collection scenarios. This range of data sources demonstrates the flexibility and adaptability of our method, showing that it is effective across diverse data curation contexts and is not limited by dataset scale.

By leveraging multiple models and diverse datasets, including large-scale, real-world sources, our study provides strong validation for the scalability and applicability of our approach.
**Motivation and Positioning on Diversity of Instructions:**

- We appreciate the reviewer's feedback on our motivation. We would like to clarify that our work ***does not contradict the value of diversity*** in instruction-tuning data. Instead, our study aims to explore the role of style consistency within a diverse dataset, offering a complementary perspective to traditional diversity-based selection methods. As demonstrated in Table 1, we intentionally use the same set of instructions across different experimental conditions to isolate the impact of style consistency, thereby excluding diversity effects. This setup enables us to identify phenomena that are independent of instruction diversity and highlight how style consistency can enhance model performance even within diversified datasets. While we recognise the value of diversity, it is beyond the scope of our paper's discussion. Combining these two approaches could be a promising direction for future applications.

- Regarding the observed performance of diversity selection, we explain in the Experiments section that this method is less effective in our context because our full dataset is already inherently diversified due to the crowdsourcing data curation methods. Consequently, diversity-based methods like random sampling and diversity selection do not provide the same advantage here as they might in scenarios with less inherent diversity.

**Clarification on Semantic Surprisal and Use of Perplexity:**

Semantic surprisal is rooted in the well-established concept of text surprisal in NLP, commonly measured by perplexity, as shown in prior works mentioned in the paper [1,2,3]. Using this foundation, we adopt perplexity to quantify surprisal. However, our approach goes a step further by connecting text surprisal to LLM fine-tuning performance—an unexplored area in previous studies. To examine the distinct impacts of semantic and non-semantic components on LLM performance, we decompose responses into segments that capture semantic versus non-semantic content, allowing us to evaluate how each component individually influences alignment.

[1] Byung-Doh Oh and William Schuler. Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? Transactions of the Association for Computational Linguistics

[2] JA Michaelov, MD Bardolph, CK Van Petten, BK Bergen, and S Coulson. Strong prediction: Language model surprisal explains multiple n400 effects. neurobiology of language, 1–71.

[3] Adam Goodkind and Klinton Bicknell. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th workshop on cognitive modeling and computational linguistics (CMCL 2018),

**Explanation of LLAMA2-13B Performance Degradation:**

- The observed performance degradation with LLAMA2-13B occurs in two contexts:
   - ***Data Generation for Fine-Tuning:*** As shown in Table 1, LLAMA2-13B-chat was used to generate data for training the base models, CodeLlama-7b and Llama3-8b. However, the LLAMA2-13B-chat model often produced low-quality data, including hallucinations and unhelpful responses, which negatively impacted fine-tuning during both response rewriting and direct generation.
  - ***Ablation Study – Ranker Training:*** In the Ablation Study, degradation also arises when LLAMA2-13B-chat-generated data is used to train our ranker. Low-quality training data led the ranker to prioritize style-consistent but low-quality responses, which ultimately detracted from the LLM’s performance in SFT. Our findings emphasize the importance of both quality and style consistency in training data for effective ranker training and, consequently, LLM fine-tuning.
Yes
No ethics review needed.
Refer to the above.
1. The paper proposes SCAR, which reduces the size of the training set while improving performance by optimizing data selection in the context of fine-tuning the existing LLM instructions. The core innovation of SCAR lies in its focus on language style consistency and defines two key style elements: language form and semantic surprisal. This systematic focus on style consistency and optimization method is an important contribution.
The paper introduces a system called SCAR that can prioritize instruction-response pairs in a training dataset based on their style consistency. In addition, the paper also explores the relationship between response style, data quality, and LLM performance.
1. The motivation of this paper is unclear. Existing research has widely recognized the importance of ensuring diversity in instruction-tuning data. However, this paper seems to oppose this common understanding without strong justification. The experiments do not persuade me, as they are somewhat weak: both the dataset and the LLM size are limited. The results are unconvincing and, if not thoroughly validated, could potentially mislead the community.

2. Although the paper proposes "linguistic form" and "semantic surprisal" as key style elements, the definitions and measurement methods of these concepts are slightly vague in some sections, especially the concept of "semantic surprisal", which still needs further clarification and explanation.

3. Although the paper provides a lot of experimental data and results, the design and interpretation of some experiments are a bit complicated. For example, in the comparison of different data selection methods, some performance differences are not adequately explained. For some performance degradation cases (such as the performance of LLAMA2-13B), the paper does not explore the reasons behind it in detail.
Yes
No ethics review needed.
1. What is the output format of StackExchange? Does it contain only code or both text and code?
1. Regarding the code, is the linguistic metric meaningful? Since the standard deviation of TTR in code is larger than that in text, which may not be intuitive.
1. Why use max pooling when calculating v_p, which only preserves the information of one token?
1. What parameters need to be trained in your method?
1. Why is the selection ratio of code and text different (12.5% vs 10%)? Is this intentional?
- Investigating the linguistic form and semantic surprisal of SFT data and their impact on fine-tuned performance is meaningful.
- The paper is well-written, easy to follow, and contains rich details.
- The authors conduct a large number of experiments to provide useful information.
- The authors open-source their code and data.
The authors analyze the impact of linguistic form and semantic surprisal on LLMs' SFT performance. They find that consistent data form leads to better outcomes. They propose a selection method, SCAR, to choose a small portion of SFT data, achieving strong performance on certain downstream tasks.
1. The position of the selection method is unclear. Is it proposed for training a general model (e.g., ChatGPT) or a specialized model (e.g., CodeLLaMA)?
   - If your method is proposed for training general models, you should verify its effectiveness using various downstream tasks (e.g., MMLU, GSM8k, HumanEval in TULU evaluation). Please report the performance on the above benchmarks directly using the checkpoints trained in Table 5. This will help determine whether the selection achieves overall improvement or just improvements in a few tasks.
   - If your method is proposed for training specialized models, you should compare it with more relevant baselines, such as directly using existing **high-quality** domain-specific data (rather than StackExchange), evol-instruct in specific domains, or instruction backtranslation in specific domains. If a user wants to train a specialized model, they do not need to select data from large-scale general data but can directly use high-quality domain-specific SFT data.

2. The method seems to select the response whose format is closest to an existing model (e.g., GPT 3.5), rather than detecting format-consistent instances in the dataset.

3. The referenced response is unconvincing to me. Since the referenced prompt contains instructions, the model may correct the response even if asked to ignore them. It might be better not to provide the instruction.
Yes
No ethics review needed.
Q1: The "longest" method discussed in prior work[1, 2] appears to be a strong rule-based baseline. How does its performance compare with SCAR?

[1] Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning
[2] Rethinking Data Selection for Supervised Fine-Tuning

Q2: The performance of SCAR on some objective benchmarks is expected, such as GSM8K, MMLU, BBH, etc.

Q3: Could you provide more details about how were the embeddings created for Figure 1(Left)?
The paper is rich in content and presents extensive analysis The investigation of the impact of styles on LLM fine-tuning effectively motivates the design of the ranking model, which is further validated through experiments on various datasets with ablation discussions.
This work proposes SCAR as a novel approach for instruction-following data selection. It is grounded in the observation that a model performs better when the response styles in its training data are more consistent. Thus, the author trains a reward model to capture response differences in linguistic form and surprisal-determining features. The required dataset consists of quadruples, i.e., an instruction, a human response, an LLM response, and a human-referenced LLM response. The model is trained to optimize the ranking loss and the representation learning loss simultaneously. Experiments on code and open-ended domains show consistent improvements in SCAR over several baselines.
* There is a lack of a simple yet significant rule-based baseline. (See Q1)

* The model trained on general instruction-following data was only tested on AlpacaEval evaluated by LLMs. The data collected for training the ranking model also relies on responses generated by LLMs. This raises concerns that SCAR inadvertently leverages certain style features favored by the LLM judge. (See Q2)

* The font in some figures is difficult to read without zooming in, particularly in Figures 1 and 3. The organization of the experimental setup in Section 4 could be improved.
Yes
No ethics review needed.
1. The colors in Table 1 are not aligned with the colors in other parts.
2. What is exactly presented in Figure 1? For left, are they the complete responses of the 3 categories or extracted functional words from responses of the 3 categories? Similarly, for right, are they the ppl over complete responses or ppl over y_c?  
3. Can you provide a more detailed illustration and analysis of how the takeaways are concluded? 
4. Please further illustrate the experimental results in Figure 2, especially in the Open-ended Domain. It looks like every metric is worse than the Full Data on Human dataset except for ppl, I think it contradicts with most of the previous findings. Can you explain it?
1. This paper proposes that Style Consistency is important for the efficiency of instruction tuning, which has not been well studied. Thus I think the novelty of this motivation is solid. 
2. The experiments conducted by this paper are comprehensive.
The paper introduces SCAR (Style Consistency-Aware Response Ranking), a method to improve the efficiency of instruction tuning for Large Language Models (LLMs) by selecting data that maintains stylistic consistency in responses. The authors propose that consistent response style enhances fine-tuning performance, and define two key elements in response style: linguistic form (the structural presentation of responses) and semantic surprisal (the predictability of response content relative to the instruction). SCAR ranks instruction-response pairs by stylistic consistency, allowing LLMs to achieve comparable or even superior performance using only a very small portion of the original dataset in coding and open-ended question-answering benchmarks.
1. The paper is not well-written and very hard to follow and understand. I can not even find an overall description of the whole workflow. An illustrative main figure should be included. 
2. The definition of Semantic Surprisal in line 054 is not well-aligned with the real metric that is used in line 162. Your definition of Semantic Surprisal, “the choices of solutions, ideas, or approaches in a response that affects how predictably or unexpectedly it addresses the instruction” is largely beyond the capabilities of perplexities. 
3. Please let me know if I am wrong, when you calculate the PPL(y_c|x), you will first filter out all the functional words (y_p). So I think it is probable that this process will directly destroy the consistency and fluency of the original sentences, thus making the resulting ppl less meaningful. 
4. Similar to point 2, your definition of linguistic form is also not well-aligned with the real metric being used. I don’t think the utilization on TTR, MTLD, Flesch score, sentence length and punctuation frequency can support your definition as “elements that shape the presentation of a response, mostly independent of semantics, such as tone (formal or informal), transitional word choice, sentence structure, formatting (bullet points or heading lines), variable naming conventions”. 
5. The settings for this paper are slightly chaotic: In the paragraph in line 148, the LLMs being used are mainly llama2 families, but in the later parts of the paper, it seems that most experiments are conducted on llama3 families. Is this done on purpose? This inconsistency makes it hard to follow. 
6. The method is far too complicated and contains too many components, thus making it hard to get practical usage. For example, it needs LLMs to first generate an analysis on Helpfulness and Correctness, and it needs LLMs to re-generate some of the responses. It also needs to train a customized module for the ranking further. As mentioned in point 1, considering so many components utilized in the method, there is no illustrative figure, which is not reasonable to me. 

I think most of my weaknesses are caused by the unclear writing of this paper. If the paper is modified to be clear, I will be glad to raise the score.
This research introduces Style Consistency-Aware Response Ranking (SCAR), a method that prioritizes stylistically consistent training examples, enabling fine-tuned LLMs to achieve superior performance with significantly reduced data.
@inproceedings{
anonymous2024scar,
title={{SCAR}: Efficient Instruction-Tuning for Large Language Models via Style Consistency-Aware Response Ranking},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=5lokEzttBF},
note={under review}
}
Recent studies have shown that maintaining a consistent response style by human experts and enhancing data quality in training sets can significantly improve the performance of fine-tuned Large Language Models (LLMs) while reducing the number of training examples needed. However, the precise definition of style and the relationship between style, data quality, and LLM performance remains unclear. This research identifies two key stylistic elements in responses: linguistic form and semantic surprisal. We find that, among training data of comparable quality, higher consistency in these response elements leads to better LLM performance. Inspired by this, we introduce Style Consistency-Aware Response Ranking (SCAR), which automatically prioritizes instruction-response pairs in the training set based on their response stylistic consistency. By selecting the most style-consistent examples, sometimes as few as 0.7\% of the full dataset, the fine-tuned LLMs can match or even surpass the performance of models trained on the entire dataset in coding and open-ended question-answering benchmarks. Code and data are available at https://anonymous.4open.science/r/SCAR-0233/.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Style Consistency Data Efficiency LLM Alignment Fine-Tuning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d709997fdebb2c6a6c73d0f7649bb319d404a41c.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SCAR: Efficient Instruction-Tuning for Large Language Models via Style Consistency-Aware Response Ranking
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission