Weak-to-Strong Trustworthiness: Eliciting Trustworthiness with Weak Supervision
Weak-to-strong learning;Fairness;Adversarial Robustness;OOD Robustness;Privacy
Yes
No ethics review needed.
* Am i understanding correctly that in Weak+WTS TFT the strong model gets trustworthiness regularization + the normal weak-to-strong loss?  does the trustworthiness regularizer make use of the weak model?
* How is the Enron data used?
* Is all fine-tuning only on the trustworhtiness datasets? What is the vanilla form of each task that doesn't apply the trustworthiness loss?
* Is it a reasonable assumption that we have these curated datasets? It would seem that even having some of the datasets makes the WTS problem secondary (eg once I know demographic attributes and have decided on demogrpahic parity, or differential privacy)

nits:
* "modelss" (extra s)
* While weak-to-strong generalization has been studied as a general property, it has not been sufficiently examined for safety and trustworthiness properties.

* The paper is fairly clear and thorough in its experiments, making it relatively easy to interpret.
In this paper the authors studied the abiilty for a weak model to teach a more powerful model to be fair, robust (both OOD and adversarial), and privacy sensitive.  The paper explores two appraoches: (1) traditional weak-to-strong generalization after training the weak model to have good trustworthiness and (2) additional trustworthiness regularization of the strong model. They find that traditional weak-to-strong generalization does not hold but that their second method works better.
* Some of the tasks chosen I don't believe are a good fit for the study.  As an example, classification over discrete features (as in Adult) is an odd fit for testing LLMs (in contrast to something like BBQ) and demographic parity as an objective is particularly easy for weak models to learn such that teaching the concept to stronger models doesn't require much. I'd expect the weak-to-strong hypothesis is more interesting in tasks where it is hard for weak models and larger models we think can be more capable but giving them good supervision is hard.  I'd expect this is true in more nuanced definitions of fairness, stereotyping, and safety as an example.  (Similar statements could be said for newer ways of evaluating privacy and robustness to jailbreaks)

* There is some lack of clarity in the method description here so I could be wrong but it seems like the Weak+WTS TFT method is largely succeeding by adding the trustworthiness regularization directly and maybe not needing the weak model at all?  My current reading as a result is that the paper is largely presenting instead a negative result for WTS genrealiztaion on trustworthiness.

* There are some missing details that I think significantly shape the results, eg what is the actual loss for Weak+WTS TFT, what data is used for every method, and what is the vanilla form of using each type of data.
Yes
No ethics review needed.
See weakness 1, 2, 3
1. The authors propose two novel methods (Weak TFT and Weak+WTS TFT) to enhance trustworthiness transfer.

2. The paper is well organized and easy to follow. 

3. The authors provide a comprehensive study about the related works.

4. The authors propose three-phase experimental design provides a thorough analysis of trustworthiness transfer.
In this paper, the authors explore the challenge of whether the fairness, robustness, and privacy can be transferred from weak to strong models through weak-to-strong generation. This is an important research topic, because the ability to transfer trustworthiness properties is crucial for preventing harmful outcomes, complying with regulations, and maintaining public trust in AI technologies. In this paper, the authors propose two novel approaches to enhance the transfer of trustworthiness properties between weak and strong models.
1. The size of the strong models included in the paper is small. How about the results on the models around 7B?

2. Lack of generazation towards fairness. 
	In this paper, the authors adopt the Demographic Parity as the representative definition. In fairness research domain, there are a series of fairness definition, such as Equal False Positive/Negative Rates, Calibration/Predictive Parity, etc. How the proposed method work with different fairness definition.


3. Lack the in-depth discussion about why privacy do not exhibit signs of weak-to-strong trustworthiness.


Minor: 

1. Formatting issue: The subfigures in Figure 3 have varying sizes.
Yes
No ethics review needed.
- Can you expand on the experimental setting? E.g. hyperparam search. This should be contained at least in the appendix.
- How do you connect the rate of improvement of trustworthiness capabilities with the previous results in weak-to-strong generalization?
Overall, the problem is important and well-motivated, the paper is well-written, and the experimental section had an interesting conclusion, matching the claims in abstract in intro.
This work investigates this critical studies the transfer of trustworthiness properties from weak to strong models. This is an empirical work showing how some trustworthiness properties, such as fairness, adversarial, and OOD robustness, show significant improvement in transfer when both models were regularized towards trustworthiness, others like privacy do not exhibit signs of weak-to-strong trustworthiness.
This paper is an empirical evaluation extending a previously known result (weak-to-strong generalization) to a new set of capabilities, ie trustworthiness. Since there is no new methodology, I was hoping to see a stronger experimental section, containing a wider variety of models and datasets, ie >1 per task. I don't think the paper has any deep technical issues. In my opinion it's incomplete due to a few things, like only evaluating one definition of fairness/privacy/ood,. and using one dataset per setting. I understand using more models would be hard, due to accessibility, but if possible this would also improve the empirical evaluation.
Yes
No ethics review needed.
1. In figure 2 (d), why does the Weak+WTS TFT perform better than the strong ceiling?
- This work considers 4 different aspects of trustworthiness.
- It performs a relatively comprehensive study of parameter sensitivity.
This paper conducts an empirical study of weak-to-strong generalization for trustworthiness (fairness, OOD generalization, adversarial robustness and privacy). They compared three ways to perform weak-to-strong generalization, where the difference is whether the trustworthiness regularization is added to the loss of the weak/strong models. Experiments show the Weak+WTS TFT approach is the most effective way of transferring trustworthiness.
- The motivation of the weak-to-strong transfer in this work is questionable. It can be meaningful to study this problem in some special cases, such as when the labeled dataset is small while there exists a large unlabeled dataset. If the strong model is only trained with the small labeled data, it can suffer from underfitting, thus making it more meaningful to train the weak model first and train the strong model with weak labels generated by the weak model. However, this work does not consider such cases.
- This work is benchmarking three different ways to fine-tune the strong model, there is not much technical novelty.
@inproceedings{
anonymous2024weaktostrong,
title={Weak-to-Strong Trustworthiness: Eliciting Trustworthiness with Weak Supervision},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=OwntMF6McA},
note={under review}
}
The rapid proliferation of generative AI, especially large language models (LLMs), has led to their integration into a variety of applications. An emergent behavior known as weak-to-strong generalization—where a strong model trained on a weak model's outputs surpasses the weak model in task performance—has gained significant attention. However, whether trustworthiness properties such as robustness, fairness and privacy can be transferred in a similar manner remains an open question. In this work, we investigate this critical question by exploring the transfer of trustworthiness properties from weak to strong models via weak-to-strong generalization. Specifically, we examine whether a strong model can inherit or even enhance trustworthiness attributes when fine-tuned on a weak model's outputs. We refer to this as weak-to-strong trustworthiness. To this end, we introduce two novel approaches aimed at improving trustworthiness transfer between weak and strong models: 1) Weak Trustworthiness Finetuning (Weak TFT), which applies trustworthiness regularization during the fine-tuning of the weak model, and 2) Weak and Weak-to-Strong Trustworthiness Finetuning (Weak+WTS TFT), which extends trustworthiness regularization to both the weak model and the strong model during fine-tuning. Our experimental evaluation on real-world datasets (Adult, OOD Style Transfer, AdvGLUE++, and Enron Emails) reveals that while some trustworthiness properties, such as fairness, adversarial, and OOD robustness, show significant improvement in transfer when both models were regularized, others like privacy do not exhibit signs of weak-to-strong trustworthiness. As the first study to explore the transfer of trustworthiness properties via weak-to-strong generalization, our work provides valuable insights into the potential and limitations of this method. Our findings highlight the importance of systematically studying trustworthiness transfer to develop AI systems that are not only accurate but also ethically aligned and reliable in critical applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Weak-to-strong learning Fairness Adversarial Robustness OOD Robustness Privacy
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/68618ef5454fd57b7b866432ad2464ef94b7ed53.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Weak-to-Strong Trustworthiness: Eliciting Trustworthiness with Weak Supervision
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission