Bayesian WeakS-to-Strong from Text Classification to Generation
weaks-to-strong;bayesian;generation;per token
Yes
No ethics review needed.
**(Q1) How exactly were the the $\lambda$’s set?** Related to (W3) above, the appendix does not say exactly what the final $\lambda$'s are for the "Fixed weight" approach in Appendix E and how "the performance of different weak models" was used.

**(Q2) How is PGR calculated when you have multiple weak models?** The definition provided in Sec 5.3 is clear when there is a single weak model but there are perhaps multiple ways to define in the WeakS-to-Strong setting (e.g. max v.s. average over the weak models in). 

**(Q3) Miscellaneous clarifications**
- In Eq. 4, what is $\hat{y}_w^{(i)}$? Should it not just be the label $k$; in general it seems something in the inner sum should depend on  $k$?
- In Eq. 7, my understanding was that we wanted $P(s_1)P(s_2) = P(W)$ but here it seems like their sum will be equal to P(W) instead of their product.
**(S1) Originality and Significance** Studying the WeakS-to-Strong setting is well-motivated and using the proposed Bayesian method here is novel. Furthermore, the expansion to from binary verification tasks to generation is notable since the latter more realistically reflects LLM use cases. 

**(S2) Quality** The paper’s proposed method results in notable improvements on the studied tasks, outperforming the individual weak model teachers in isolation and the naive baseline that simply does a weighted average over the standard CE-based losses. I also appreciated the extra ablations the authors ran to verify the importance of certain aspects of their design (e.g., the aux loss, probability estimation for weak sequences and the $\lambda$ weights across tasks). 

**(S3) Clarity** Overall, I found the writing to be clear, doing a good job of contextualizing methods/results in terms of prior work.
This paper extends work in Weak-to-Strong generalization in two notable ways: (1) considering the setting in which there is an ensemble of multiple weak teachers; (2) expanding from more basic text classification to generation-based tasks. For (1), they show that using a Bayesian approach (based on EDL) is effective for learning from the ensemble of weak teachers, improving upon any of the weak teachers individually as well as simpler baselines that weight typical cross-entropy losses across teachers. For (2), they overcome the key issue that weak and strong models may have different tokenizers and also try applying a modified DPO stage on top, reporting a positive effect.
**(W1) Significance of some results.** I found that certain claims in the manuscript were not as strongly supported by the empirical results. For instance, it appears that the improvements obtained from the DPO-based stage weren't too big (i.e., within standard deviations across runs). Another case was the claim in L461 about the auxiliary loss improving Bayesian methods. That being said, this doesn't affect my opinion about what I consider to be the main result, i.e., the superiority of Bayesian approach. 

**(W2) Other baselines and tasks.** There are some other baselines that I think would be good to consider, namely:
- Methods from data programming / programmatic weak supervision (e.g Snorkel, FlyingSquid) to aggregate the labels generated by different weak models (i.e, treating them as Labeling Functions). In any case, it would be nice to discuss this line of work given that they also operate in a setting where the goal is aggregate multiple weak label sources. 
- In the related work, the authors mentioned that some previous works (Liu & Asahi, 2024; Sang et al., 2024) tried traditional ensembling methods. Perhaps these would also be useful baselines?

In terms of evaluations, it would strengthen the paper if the authors could show similar results on more tasks per category (i.e., classification, generation). Currently the results are limited to one task for each. 

**(W3) Clarification on key assumptions.**  Some further discussion would be helpful on the following
- From cross-referencing Appendix E with the main result tables, it appears that the best results rely on the "Fixed Weight" approach which is based on how good each of the weak models are individually. The assumption here (which I think should be made more explicit), seems to be that one is actually able to accurately estimate the weak model performances. I wonder how applicable this assumption is for real superhuman tasks where presumably getting ground-truth labels is difficult. Perhaps the authors can comment here? 
- Given one stated motivation for the study of WeakS-to-Strong was to consider the variation in human opinions, how might the methods/results generalize to when supervision actually comes from humans rather than weak models? In particular, it seems that in the current experiments, having confidence estimates associated with the weak model labels is important, but this may not readily exist for human-generated labels?
Yes
No ethics review needed.
**Suggest**: As I understand, the regularization loss is critical. I suggest moving the corresponding discussion to the main paper. I also suggest mentioning and discussing the exact values of the weights in the loss to the main paper, as it can provide some insight into how the method works.
**Originality**: The paper is the first to successfully extend weak-to-strong to the weaks-to-strong setting. The paper also contains several useful ideas for performing sequence-level training using soft-labels when the student and the teacher have different tokenizers.

**Clarity**: The paper is written well and is easy to follow.

**Quality**: The experiments are well structured to answer the central questions posed by the paper, i.e., is weaks-to-strong better than weak-to-strong, and is the proposed method of performing weaks-to-strong training better than naive baselines? 

**Significance**: Based on the quality of the experiments and the topic addressed, I believe that a broad community will find the paper useful.
The paper extends the weak-to-strong training to multi-weak-to-strong or weaks-to-strong. In a weak-to-strong setting, a smaller model provides a training signal for a larger model. This paper proposes to use multiple small models to provide the training signal. To squize out the best from the ensemble of small models, the paper proposes a Bayesian strategy based on evidential deep learning. The paper also performs sequence generation training, which was not done previously in the weak-to-strong setting. The results demonstrate that weaks-to-strong provides produces better results compared to weak-to-strong in both classification as well as generation.
1. Some of the design choices are not well justified. For example, for the sequence-level training using soft labels, it is not clear why the strategy proposed in section 4.1 should work or is optimal. 
2. The datasets selected for the classification and generation tasks seem very specific. Is there a particular reason for selecting only these? The impact of the paper could be much better if model general datasets were included, at least for the generation task.
Yes
No ethics review needed.
- In Equation 3, are $\alpha_k$s are trainable parameters?
- Is the Equation 3 the loss for each sample?
- The proposal is simple but works.
The paper introduces a Bayesian-based method inspired by the weak-to-strong approach for the classification task of language models. The author also extends the algorithm for the generation task by some simple transformations. To improve the performance of the generation task further, the author utilized DPO to finetune the model at the tail of the framework.
- The motivation for using Dirichlet distribution to model the prior is ambiguous.
- The writing for the methodology description is unclear and hard to follow
- The transformations in Equation 3 should be clarified.
- What is the definition of $p_k$ in Equation 4?
- In Section 4.1, the probability of the target token of the strong model changing after each update of network parameters because $C_s(s_1)$ and $C_s(s_2)$ shift. This means that the targets vary along the training, likely creating instability or leading to divergence. The time consumed for recomputing these terms is also one of my concerns.
- All baselines that are naive and very simple algorithms are weak for both classification and generation tasks. It is suggested that stronger baselines be included for both tasks.
Yes
No ethics review needed.
1. I didn't get the idea behind "The description about complementary of weak models"(Section 6.3). I can tell that classification enjoy better agreement than generation(as generally expected), but I didn't get "This suggests that the consistency among different weak models is low, thus they can complement each other well as the faults made by different weak models are not the same"
1. They propose a Bayesian WeakS-to-Strong framework that improves weak supervison.(Although I tend to say this improvement is not well-proved). 
2. Extention of Weak-to-Strong framework from text classification to generation tasks.
3. A method for estimating token-level probabilities is introduced to create soft labels for training robust models.
This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS- to-Strong generalization.

Furthermore, they broaden the use of the WeakS-to-Strong approach from its original focus on text classification to encompass text generation tasks(this point causes confusion for me, see in the weakness below), exploring more sophisticated supervision strategies. Additionally, direct preference optimization is employed to enhance the student model's ability to learn preferences, moving beyond the traditional teacher-forcing training framework(I think this contribution is not clear and verified by comprehensive experiments). 

I am not an expert in this domain, but I read this paper twice to get the paper's idea. I am open to other reviewers' opinion.
I have the following concerns regarding to this paper: 

1. The experiments is not comprehensive: the experiment they conduct are mainly on the ~1B weak models and the strong model are ~7B. I think from the experiment results conducted now, the claim on WeakS-to-Strong in the paper is slightly overclaimed. (**The major weakness in my opinion**)
2. The improvement shown in the experiment is not observable: for instance, for accuracy: 0.843(baseline) vs 0.850(WeakS-to-Strong-3). The improvement is negligible.
@inproceedings{
anonymous2024bayesian,
title={Bayesian WeakS-to-Strong from Text Classification to Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=pHe4P1IVnb},
note={under review}
}
Advances in large language models raise the question of how alignment techniques will adapt as models become increasingly complex and humans will only be able to supervise them weakly. Weak-to-Strong mimics such a scenario where weak model supervision attempts to harness the full capabilities of a much stronger model. This work extends Weak-to-Strong to WeakS-to-Strong by exploring an ensemble of weak models which simulate the variability in human opinions. Confidence scores are estimated using a Bayesian approach to guide the WeakS-to-Strong generalization. Furthermore, we extend the application of WeakS-to-Strong from text classification tasks to text generation tasks where more advanced strategies are investigated for supervision. Moreover, direct preference optimization is applied to advance the student model's preference learning, beyond the basic learning framework of teacher forcing. Results demonstrate the effectiveness of the proposed approach for the reliability of a strong student model, showing potential for superalignment.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
weaks-to-strong bayesian generation per token
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/09e2097ec1565fab9cd3546d76cd55aebf7930a1.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Bayesian WeakS-to-Strong from Text Classification to Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission