Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining
Large language models;Code generation;Pretraining;Data quality
Yes
No ethics review needed.
1) Could you provide more detailed analysis or ablation studies on how the quality annotator specifically improves the model’s learning? What are the key features or patterns it identifies that contribute most to the performance boost?

2) How does the synthetic data generated by Llama-3.1-70B differ in quality or characteristics from the high-quality tokens selected by the annotator? Could you provide examples or metrics that highlight these differences?

3) Your results suggest that grouping by language before repository improves performance. Could you elaborate on why this approach works better? Have you considered any hybrid methods that combine cross-language learning with language-specific training?

4) Given the success of Arctic-SnowCoder-1.3B with 555B tokens, how do you envision scaling this approach for larger models or different domains? Are there diminishing returns or unique challenges you anticipate?

5) The paper focuses on benchmarks like BigCodeBench and HumanEval+. How do you ensure these benchmarks reflect real-world programming challenges? Have you considered any additional metrics or benchmarks that might better capture practical coding scenarios?
Arctic-SnowCoder demonstrates remarkable strengths among small size model, particularly in achieving state-of-the-art results on BigCodeBench with a 36% performance improvement over Phi-1.5-1.3B, despite using only 555B tokens compared to models trained on trillions of tokens. Arctic-SnowCoder-1.3B outperforms StarCoderBase-3B across all benchmarks and surpasses StarCoder2-3B, trained on over 3.3T tokens, on HumanEval+ with a score of 28.0 compared to 27.4. The model also achieves competitive results on MBPP+ (42.9) and EvoEval (18.0), showing that it can match or exceed the performance of larger models like StableCode-3B and Granite-Code-Base-3B, which are trained on 1.3T and 4.5T tokens, respectively. These results, combined with thorough ablation studies, highlight the effectiveness of its three-phase pretraining strategy, focusing on high-quality and synthetic data, while providing concrete evidence of its efficiency and superior performance in practical and complex coding tasks.
This paper introduces Arctic-SnowCoder-1.3B, a small code model trained on 555B tokens through a meticulously designed three-phase pretraining process. The first phase involves general pretraining with 500B tokens of standard-quality code, filtered and deduplicated. The second phase refines this with 50B tokens of high-quality code, identified using a BERT-based quality annotator trained on positive examples from curated open-source repositories and instruction datasets. In the final phase, 5B tokens of synthetic data are generated using Llama-3.1-70B, seeded from the high-quality data to further enhance model performance. The model achieves state-of-the-art results on BigCodeBench, significantly outperforming larger models on practical and challenging programming benchmarks such as HumanEval+ and MBPP+. The paper highlights the importance of progressively improving data quality and aligning it with downstream tasks, offering comprehensive evaluations, ablation studies, and practical insights into optimal pretraining strategies, such as learning rate schedules and data repetition, to maximize the efficiency of smaller language models in code generation tasks.
While the synthetic data significantly boosts performance, as seen in the 36% improvement over Phi-1.5-1.3B on BigCodeBench, an overreliance on synthetic data risks skewing the model’s understanding of practical coding tasks. Additionally, the performance on HumanEval+ (28.0) and MBPP+ (42.9), although impressive, shows only incremental improvements over models like StarCoder2-3B (27.4 on HumanEval+ and 49.2 on MBPP+), indicating room for optimization in handling more complex or diverse programming tasks. The quality annotator, trained on specific curated datasets, could introduce biases that may not adequately represent broader coding practices, potentially limiting its effectiveness across all programming domains.

The paper’s approach to handling repo-level data in the general pretraining phase is insightful but has room for further exploration. The authors compare two methods: grouping files by repository names and by language before repository. They conclude that partitioning by language yields better results, as evidenced by improved scores on HumanEval+ (17.1 vs. 12.8), MBPP+ (33.9 vs. 30.7), and EvoEval (7.4 vs. 7.0). This method ensures that training documents are more focused and homogenous, which likely aids the model in learning language-specific patterns effectively. However, this method might overlook the potential benefits of cross-language learning, especially in multi-language projects where inter-language interactions are critical. Future work could explore hybrid approaches that maintain language-specific grouping but occasionally incorporate multi-language contexts to enhance the model’s ability to handle real-world, polyglot codebases. Additionally, more granular investigations into the impact of repository size and the diversity of code snippets within a repository could provide deeper insights into optimizing repo-level data grouping for enhanced model performance.

In addition, my concern is that the paper presents compelling results among small language models, particularly with its strong performance on benchmarks like BigCodeBench and HumanEval+. However, the underlying reasons for achieving such high performance despite the relatively small training dataset (555B tokens) are not fully unpacked. While the authors attribute the success to the progressive refinement of data quality and the use of synthetic data, the detailed mechanisms by which these factors translate into superior model performance remain somewhat opaque. 

Finally, I suggest evaluating the model on the CodeMMLU benchmark, which could provide a broader assessment of the model’s capabilities across a diverse set of coding tasks, thereby offering more comprehensive insights into its strengths and potential areas for improvement.

[1] CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities of CodeLLMs, https://arxiv.org/abs/2410.01999
Yes
No ethics review needed.
The authors address a critical aspect of language model development—high-quality data in the code domain—which is essential for improving model performance and applicability. For the experiments, Arctic-SnowCoder-1.3B demonstrates good results, achieving state-of-the-art performance on BigCodeBench and surpassing similarly sized models trained on up to 1 trillion tokens, including a notable 36% improvement over Phi-1.5-1.3B.

However, I have three concerns:

1. Limited Novelty: While the use of a data annotator to extract high-quality data for pretraining is a valuable approach, it is not entirely novel. Similar methodologies have been employed, such as using GPT-4 as a data annotator. This raises questions about the uniqueness of the authors' contributions.

2. Missing Baselines: The evaluation would benefit from the inclusion of additional baselines, such as OpenAI's GPT models. Comparing or discussing these established models would provide a more comprehensive context for assessing Arctic-SnowCoder's performance and highlight its relative strengths and weaknesses.

3. Lower than Phi-1.5-1.3B on HumanEval+ MBPP+ and EvoEval: Despite achieving strong performance on BigCodeBench, Arctic-SnowCoder-1.3B underperforms compared to Phi-1.5-1.3B on more general code generation tasks, such as HumanEval+, MBPP+, and EvoEval. Interestingly, Phi-1.5-1.3B achieved better results with less training data, which suggests that Arctic-SnowCoder's specialized pretraining on high-quality code tokens may not necessarily translate into better generalization across a broader range of benchmarks.
+ Important Area.

The authors address a critical aspect of language model development—high-quality data in the code domain—which is essential for improving model performance and applicability. 



+ Good Performance on BigCodeBench

Arctic-SnowCoder-1.3B demonstrates good results, achieving state-of-the-art performance on BigCodeBench and surpassing similarly sized models trained on up to 1 trillion tokens, including a notable 36% improvement over Phi-1.5-1.3B.
The authors emphasize the critical role of high-quality data in the effective pretraining of language models, particularly within the code domain, while noting that the precise definition of "high-quality" remains inadequately explored. To address this, they introduce Arctic-SnowCoder-1.3B, a data-efficient base code model pretrained on 555 billion tokens through three phases of progressively refined data. The first phase involves general pretraining with 500 billion standard-quality code tokens, processed through basic filtering, deduplication, and decontamination. The second phase continues with 50 billion high-quality tokens, selected from the first phase by a BERT-style quality annotator trained to distinguish good code from random data, using high-quality code files and instruction data from Magicoder and StarCoder2-Instruct. The final phase employs 5 billion synthetic tokens generated by Llama-3.1-70B, using phase two data as seeds and adapting the Magicoder approach for pretraining. Despite the limited dataset, Arctic-SnowCoder achieves state-of-the-art performance on BigCodeBench, outperforming similarly sized models trained on up to 1 trillion tokens, including a 36% improvement over Phi-1.5-1.3B. Across various benchmarks, Arctic-SnowCoder-1.3B performs better than StarCoderBase-3B pretrained on 1 trillion tokens and matches the performance of leading small base code models trained on trillions of tokens.
1. Limited Novelty: While the use of a data annotator to extract high-quality data for pretraining is a valuable approach, it is not entirely novel. Similar methodologies have been employed, such as using GPT-4 as a data annotator. This raises questions about the uniqueness of the authors' contributions.

2. Missing Baselines: The evaluation would benefit from the inclusion of additional baselines, such as OpenAI's GPT models. Comparing or discussing these established models would provide a more comprehensive context for assessing Arctic-SnowCoder's performance and highlight its relative strengths and weaknesses.

3. Lower than Phi-1.5-1.3B on HumanEval+ MBPP+ and EvoEval: Despite achieving strong performance on BigCodeBench, Arctic-SnowCoder-1.3B underperforms compared to Phi-1.5-1.3B on more general code generation tasks, such as HumanEval+, MBPP+, and EvoEval. Interestingly, Phi-1.5-1.3B achieved better results with less training data, which suggests that Arctic-SnowCoder's specialized pretraining on high-quality code tokens may not necessarily translate into better generalization across a broader range of benchmarks.
Yes
No ethics review needed.
There are some questions after reviewing the paper:
1. In line 190,  "increase the Python mix ratio to approximately 50% while keeping the proportions of the other languages unchanged.", why is Python set as the primary language data, and how can it be adjusted for other languages?
2. In line 295,  "We can observe that the second approach, which we finally adopt in general pretraining, performs significantly better than the first one.", could you further explain the reason behind this conclusion?
3. In line 351, "the key to high-quality data is essentially the alignment with downstream application distributions.", what is the difference between alignment and fine-tuning of pre-trained models? And what are the advantages of the method proposed compared to fine-tuning techniques?
1. This paper proposes a method for improving the performance of pre-training models by focusing on multi-stage data quality enhancement. It introduces a high-performing code model with low token usage. 
2. Additionally, the paper analyzestraining strategies, including emphasizing the preparation of training data files and the characteristics of learning rate scheduling.
This paper introduces a  method that includes three different pre-training phases, combined with iterative improvements to the quality of training data. It presents a code model—Arctic-SnowCoder-1.3B—which demonstrates competitive performance compared to current small code models, while significantly reducing the number of tokens used. The paper also provides guidelines for repo-level data grouping, learning rate scheduling, and emphasizes the importance of high-quality data.
This paper primarily focuses on techniques for enhancing and filtering the quality of code training data, with a key emphasis on how high-quality, filtered data improves model performance. However, an important question arises: could this improvement come at a cost, such as reduced generalization ability on non-target domain tasks? 

Additionally, the paper should review some existing techniques for improving training data quality and, where appropriate, include comparative analyses to demonstrate the advantages of the proposed method.
Yes
No ethics review needed.
1. Regarding the openness of this work, will the authors consider making the data pipeline and models publicly available? This will greatly help future studies on Code LM pretraining.
2. Regarding the evaluation, can the authors provide some explanations as to why BigCodeBench results are omitted in most of the tables?
3. Can the authors share more insights on how current setups documented in the paper can be generalized to larger Code LMs (e.g., 10B+)?
The authors documented various valuable practices of pretraining Code LMs from scratch, which can inspire future work in this direction:
- Training a BERT-based classifier to annotate code quality, which is very efficient compared to any LLMs-as-Judges approaches.
- Using re-warmup as the learning rate schedule is quite novel.
-  While previous studies [1] suggest that deduplicating the data will result in better model performance, training on the repeated high-quality code data can further improve the coding capabilities.

The authors also shared a few interesting findings:
- "Textbook" is not all you need, so improving "educational value" in the training data may not be optimal.
- Re-warmup performs much better than other conventional schedules, such as linear and constant.
- When the number of high-quality tokens is limited to 50B, the setup of 12.5B with four repetitions could be more optimal.

[1] Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022, May). Deduplicating Training Data Makes Language Models Better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8424-8445).
The authors provide practical findings through pretraining a small code language model (Code LM), which achieves SOTA performance on representative coding benchmarks like HumanEval, MBPP, and BigCodeBench, among the Code LMs with similar sizes.
There are a few weaknesses in this paper:
- The current evaluations are Python-only, and the evaluation on multilingual (programming languages) code generation may share more interesting findings.
- The authors only study the 1.3B models, which is considered a bit small. While I understand that pretraining LMs is very costly, is it possible for the authors to provide more motivation for studying the ~1B models?
- It is most likely that the data cannot be opened due to legal constraints.
@inproceedings{
anonymous2024arcticsnowcoder,
title={Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=X9JU2gKEkR},
note={under review}
}
Recent studies have been increasingly demonstrating that high-quality data is crucial for effective pretraining of language models. However, the precise definition of "high-quality" remains underexplored. Focusing on the code domain, we introduce Arctic-SnowCoder-1.3B, a data-efficient base code model pretrained on 555B tokens through three phases of progressively refined data: (1) general pretraining with 500B standard-quality code tokens, preprocessed through basic filtering, deduplication, and decontamination, (2) continued pretraining with 50B high-quality tokens, selected from phase one by a BERT-style quality annotator trained to distinguish good code from random data, using positive examples drawn from high-quality code files, along with instruction data from Magicoder and StarCoder2-Instruct, and (3) enhanced pretraining with 5B synthetic data created by Llama-3.1-70B using phase two data as seeds, adapting the Magicoder approach for pretraining. Despite being trained on a limited dataset, Arctic-SnowCoder achieves state-of-the-art performance on BigCodeBench, a coding benchmark focusing on practical and challenging programming tasks, compared to similarly sized models trained on no more than 1T tokens, outperforming Phi-1.5-1.3B by 36%. Across all evaluated benchmarks, Arctic-SnowCoder-1.3B beats StarCoderBase-3B pretrained on 1T tokens. Additionally, it matches the performance of leading small base code models trained on trillions of tokens. For example, Arctic-SnowCoder-1.3B surpasses StarCoder2-3B, pretrained on over 3.3T tokens, on HumanEval+, a benchmark that evaluates function-level code generation, and remains competitive on BigCodeBench. Our evaluation presents a comprehensive analysis justifying various design choices for Arctic-SnowCoder. Most importantly, we find that the key to high-quality data is its alignment with the distribution of downstream applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large language models Code generation Pretraining Data quality
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0a14022ac6e15d7d6b42aa0e2b4b217566332c79.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission