Weak to Strong Generalization for Large Language Models with Multi-capabilities
Weak to Strong Generalization;Large Language Model;Superalignment
Yes
No ethics review needed.
- In line 132, there is a missing comma. Additionally, it would be better to explain the difference between your work and related work in the related work section.
- In Figure 1 (right), it would be helpful to annotate that the performance pertains to the strong model.
- In line 186, "Figure 4" should be "Figure 3". In line 191, "Figure 3" should be "Figure 4". Furthermore, in Figures 3 and 4, it would be better to keep the colors of the "weaken" and "remove" bars consistent.
- In Section 3.5, the experiments show that the same strong model cannot improve itself. However, I am interested in the outcomes when training another weak model or a smaller but strong model.
- The paper is well-written and easy to understand. Moreover, the structure is complete and coherent.
- Section 3 provides valuable insights, and the proposed method makes sense.
This paper first conducts a series of analyses to explore the factors influencing the spectrum from weak to strong generalization in large language models (LLMs) with multi-capabilities. It then proposes a two-stage training framework that uses reward models to select valuable data based on the insights gained from the analyses. Experimental results demonstrate improvements in the multi-capabilities of the LLMs.
- The setups of "WTS-S" and "WTS" are not clearly explained upon their first mention. It appears that "WTS-S" involves training different models for different datasets and computing their average score, while "WTS" involves training all datasets together in a single model.
- In Sections 3.3 and 3.4, it would be better to reference the setups under "WTS" rather than "WTS-S".
- What is your setup for the "mix data" in Section 3.6? Are the datasets directly merged? There is concern about potential conflicting cases in the "mix data" that could confuse the model. The total number of correct cases in "mix data" should be larger than "weak data". So I am doubtful about the conclusion regarding Figure 7 (right) because it seems a little contradictory to Figure 7 (left). Additionally, "mix data" should be better termed as "mixed data".
- The performance advantages of "WTS" in Figure 8 are not obvious when the weak model is scaled up. However, the gap to the strong model remains significantly large.
- I doubt the results regarding "the dependence of data quality across various capabilities" as claimed in lines 191-192. In Figure 4, there are only two cases: "w/o GSM8k" and "w/o ECQA & e-SNLI". While these can provide some insight, they are not comprehensive. If, as the conclusion suggests, the abilities in each capability are independent of each other, then why are the mixed training results of multi-capability data better than the single-capability averages?
- **[Important]** Despite the focus on multi-capabilities in the title, I would prefer to see the performance in single capabilities and comparison results with existing methods. It seems that most of the analysis and the proposed method could also apply to single capabilities, which are more commonly used. This paper does not convincingly demonstrate the necessity of working on multiple capabilities.
Yes
None
No ethics review needed.
1. What is the connection between the conclusion regarding the independence of different capabilities mentioned in Weakness 1 and the subsequent methodology and experiments?

2. In the scaling experiments, it is observed that as model size increases, the difference between the proposed method and the baseline becomes smaller. Considering the reward model's size is consistent with that of the strong model and the data used for the reward model is from the weak and strong model training, does the proposed method have certain limitations? (In simple terms, is the reward model scalable?)
The paper analyzes the current gaps in research on the weak to strong generalization paradigm from a "super-alignment" perspective, proposing a new setup for experimental analysis and providing a novel training approach under this setup.
This paper analyzes a multi-capabilities setting under the weak to strong generalization paradigm. Based on this analysis, a training framework for this paradigm is proposed, which includes a reward model for selecting valuable weak supervision data, followed by a two-stage training of a strong model on both the weak dataset and the selected dataset. Further discussion on the reward model is also provided.
1. The significance of the multi-capabilities setting is unclear. In Section 3.3, the paper concludes that since the quality of weak data is relatively independent across different capabilities and does not interfere with each other during generalization, the setup seems only quantitatively different from a single capability setup. Therefore, is it necessary to study this setup separately? It is recommended to clarify the differences between the single capability setup and the one proposed in this paper.

2. The paper does not clearly define the diversity of the weak dataset. It is mentioned that the diversity of a weak dataset can lead to better results as previous works. The experiment the third chapter focuses on the consistency of results generated by weak and strong models, which is typically not an explicit measure of diversity. 

3. The paper lacks ablation experiments. It only presents the performance without using the reward model for data selection. It is suggested to conduct experiments by randomly selecting the same amount of data as picked by the reward model (as random selection is a strong baseline in LLM data selection).

4. The legends in the figures are unclear, and some contain labeling errors. In the first three sections of Chapter 3, "weak model" is used as the axis label, yet most of the time, the data shown is based on the performance of the strong model, causing confusion. Additionally, there are errors in the numbering of Figures 3 and 4 in Section 3.3. Besides, there is inconsistent usage of "weak to strong generalization" and "weak-to-strong generalization."
Yes
No ethics review needed.
1. I am curious whether the second version of the strong model in Section 3.5 is further trained on the existing trained strong model, or if it is untrained. If an untrained strong model is used to relabel data and then continues to train itself, would that improve performance? Additionally, while greedy search can exacerbate the issue of overconfidence, would sampling help mitigate this? Specifically, by using multiple sampling and voting (self-consistency) to select more diverse and accurate samples, could it lead to greater self-bootstrapping gains?
2. Is a separate reward model trained for each weak-to-strong training instance? If so, wouldn’t this make the overall training pipeline somewhat lengthy? Additionally, I understand that the reward model’s training data and inference data are identical, so how many weak data points with the same labels are actually filtered out? If not, how does the reward model generalize?
3. To what extent might rule-based data selection methods serve as alternatives to the reward model? For instance, could calculating semantic similarity or n-gram redundancy be effective approaches?
4. What is the performance of the strong model before training? How does it compare with the supervised trained weak model and the strong model after weak-to-strong training? I am curious about the significance of studying weak-to-strong generalization.
1. As the first study to investigate multi-capability weak-to-strong generalization, it can provide some relevant conclusions for this field.
2. This study explores data selection directions for weak-to-strong generalization and proposes corresponding methods to improve performance.
This paper first investigates multi-capability weak-to-strong generalization in large language models (LLMs). It then confirms that diverse, high-quality data from weak models improves generalization for strong models. To enhance this process, the authors introduce a two-stage training framework that uses reward models to select optimal weak supervision data.
1. The research on multi-capability weak-to-strong generalization seems rather isolated from the later stages of weak data selection and also lacks a connection to the proposed method. It is a relatively shallow and separate exploration.
2. The validation of self-bootstrapping's ineffectiveness is insufficient. Concerns are detailed in the 'Questions' section.
3. The necessity and effectiveness of the reward model lack sufficient validation and investigation. For example, an ablation study on Equation 7 was not conducted. Additional concerns are detailed in the 'Questions' section.
4. The necessity of the warm-up stage lacks further validation. For example, running only 2 epochs might be insufficient, as the model's optimal performance is achieved after a total of 4 epochs, suggesting that it may be the total training time, rather than the distinct nature of the two stages, that drives performance improvements.
Yes
None
No ethics review needed.
As mentioned in the above weaknesses.
1. The research topic is novel and meaningful, providing valuable insights into multi-capability alignment under the concept of super-alignment.
2. The writing is rigorous and standardized, with a deep and thorough analysis that enhances the overall clarity of the paper.
3. The proposed two-stage approach effectively addresses key challenges within the research paradigm, supported by solid theoretical analysis and experimental validation.
This paper conducts a thorough exploratory study on multi-capability alignment within the realm of weak supervision for strong model training. The authors make several key observations, such as the relative independence of different tasks when weak supervision is applied and the tendency of models to become overly reliant on limited supervision through bootstrapping. These insights highlight the challenges and limitations of current approaches. Based on these findings, the authors propose a theoretically and experimentally grounded two-stage training framework, which includes training reward models for data selection and training strong models. This framework effectively enables learning from weak datasets and the selected data, addressing the inconsistencies between weak and strong datasets.
1. While the paper exhibits excellent logical flow and robust experimentation, the presentation of key figures, such as Figure 1, lacks clarity and attention to detail. The meanings of the axes, calculation formulas, and the implications of the legends are somewhat ambiguous, leading to confusion regarding the data sources. The overlapping results of different methods detract from the visual appeal, and the depiction of performance proportional to model size raises questions, especially when the same-sized methods are represented in different colors. It is advisable to optimize this section for better clarity and presentation.
   
2. The width of the model size representation could be expanded to better align with the theme of super-alignment, thereby more effectively demonstrating the robustness of the proposed method.
Yes
No ethics review needed.
The paper explores the weak to strong generalization for LLMs across multiple capabilities. However, there are several aspects that require further clarification and improvement to strengthen the validity and robustness of the proposed method. Below are my detailed comments:
1. The paper frequently uses the term "performance" to describe the experimental results of the comparison models without clearly specifying the particular evaluation metrics. Could the authors clarify what is meant by "performance"? Is it accuracy, or another metric? Please provide a clear definition and the specific metrics used.
2. In Equation 6, why is Dci considered low-confidence data? Is it because this data is generated by a strong model through relabeling? Could the rationale behind treating it as low-confidence data be to enhance data diversity and thereby prevent model collapse?
3. Could you provide more details about the reward model R(x)? Specifically, what are the training data and model architecture used for R(x)? Are the reward model, weak model, and strong model all instances of Qwen 1.5, or do they differ in their configurations?
4. In the experiments conducted with different model sizes, were the sizes of the datasets (e.g., the labeled dataset and the unlabeled dataset) varied? For instance, was a larger amount of data used for the 4B model compared to the 1.8B model?
The authors present how weak supervision affects strong models, revealing that strong models do not always follow the guidance of weak models. This insight is crucial for understanding the limitations and potential pitfalls of using weak supervision. And the proposed training framework, which uses reward models to select valuable data for weak supervision, is innovative. It addresses the issue of overconfidence in strong models and leverages the diversity of weak datasets to improve generalization. This paper clearly outlines the steps involved in the proposed framework, making it easy to understand.
This paper investigates the weak-to-strong generalization capabilities of LLMs across multiple capabilities. Through extensive experiments, key findings include the relative independence of capabilities, the significant impact of data quality on weak supervision, and the overconfidence of strong models in specific knowledge areas, leading to performance degradation. To enhance weak-to-strong generalization, the authors propose a novel training framework. This framework uses reward models to select valuable weak supervision data, generated by weaker models, for training stronger models. A reward model is trained to identify datasets that differ in distribution from the strong datasets, ensuring diversity. A two-stage training method is then applied, allowing the strong model to learn from both weak and selected datasets, resulting in improved multi-capabilities weak-to-strong generalization.
Although the paper points out that overconfidence is an important problem in strong models, it does not provide a detailed description of the reward model used to select valuable weak supervision data, such as the reward model's framework and training data. In addition, this paper uses 'performance' to refer to the experimental results of the comparison models in several places, without clearly describing the specific evaluation metrics of the performance. Clearer metrics would help readers better understand the improvements achieved by the proposed framework.
@inproceedings{
anonymous2024weak,
title={Weak to Strong Generalization for Large Language Models with Multi-capabilities},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=N1vYivuSKq},
note={under review}
}
As large language models (LLMs) grow in sophistication, some of their capabilities surpass human abilities, making it essential to ensure their alignment with human values and intentions, i.e., Superalignment. This superalignment challenge is particularly critical for complex tasks, as annotations provided by humans, as weak supervisors, may be overly simplistic, incomplete, or incorrect. Previous work has demonstrated the potential of training a strong model using the weak dataset generated by a weak model as weak supervision. However, these studies have been limited to a single capability. In this work, we conduct extensive experiments to investigate weak to strong generalization for LLMs with multi-capabilities. The experiments reveal that different capabilities tend to remain relatively independent in this generalization, and the effectiveness of weak supervision is significantly impacted by the quality and diversity of the weak datasets. Moreover, the self-bootstrapping of the strong model leads to performance degradation due to its overconfidence and the limited diversity of its generated dataset. To address these issues, we proposed a novel training framework using reward models to select valuable data, thereby providing weak supervision for strong model training. In addition, we propose a two-stage training method on both weak and selected datasets to train the strong model. Experimental results demonstrate our method significantly improves the weak to strong generalization with multi-capabilities.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Weak to Strong Generalization Large Language Model Superalignment
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a1a382dcc636fd8655399b2078c21d2c6bba3467.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Weak to Strong Generalization for Large Language Models with Multi-capabilities
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission