Quantitative Certification of Bias in Large Language Models
Large Language Models;Bias;Certification
Yes
No ethics review needed.
In addition to the points listed above in the weaknesses section:
- The confidence intervals in Table 1 are fairly wide. Is it possible to run with more samples in order to obtain more conclusive results?
- Would it be possible to sample multiple prefixes for a given prompt in order to obtain more unbiasedness observations when constructing the confidence intervals? If so, how would this (if at all) affect the construction of the confidence intervals?
- Would it be valuable to consider other sorts of prefix distributions beyond those explored in Section 4? If so, what sorts of prefix distributions would be useful for what contexts?
Strengths of this paper include:
- Quantification of bias in LLMs is an important issue that is of interest to the community.
- The strategy of encoding information about counterfactual prompts into a prompt distribution is interesting.
- The experiments raise valuable insights about the sort of prompts for which current LLMs are likely to generate biased responses.
This paper proposes a method to certify the amount of bias in responses generated by large language models (LLMs). There are three main contributions. First, the paper introduces a distributional definition of bias that samples prompt prefixes from a distribution and passes the response generated by the LLM into an unbiasedness detector. Second, it proposes a method for quantifying the level of bias in which Clopper-Pearson confidence intervals are constructed based on the output of the unbiasedness detector. Third, experiments are conducted on contemporary LLMs to measure and analyze the amount and type of bias that they exhibit.
Weaknesses of this paper include:
- From a statistical methodology standpoint, there is not much novelty since the method is a direct application of Clopper-Pearson intervals.
- The writing in Section 3 could be improved in several ways.
  1. Specifying what spaces the different quantities (e.g. $\mathcal{G}$, $\mathcal{A}$) live in, and what operations (e.g. string concatenation) can be applied to them.
  2. Clarifying Definition 1, especially part (3) as it is not directly obvious how an unbiased text generator is relevant for the definition.
  3. Discussing briefly the meaning of a certificate. Equation (1) defines a probability, but intuitively a certificate would be some statement (such as an upper bound) about a probability.
- The presentation in Section 4 is disjointed. It would be helpful to have a single algorithm box that consolidates the procedure, and then discusses the different design choices (such as prefix distribution).
Yes
No ethics review needed.
- Have the authors thought about the impact of model temperature $T$ (i.e. the sharpening/softening of the probability distribution over tokens at inference time), or top-$k$/top-$p$ selection mechanisms, on the certificate $\mathcal{C}$? As far as I understand the formulation in the paper, $T$ and top-$k$ (or top-$p$) selection will induce a different $\Delta$. I don’t think this undermines the main point of the paper, but it’s an important detail that  $\mathcal{C}$ is for a specific parametrization of the model.

- Can you please clarify the utility of “random prefixes”? The works cited (Wei et al. 2023, Zou et al., 2023) might be misunderstood: those works are not about generating random prefixes, but instead, very intentional, learned sequences of tokens that lead to jailbreaks (they just happen to look like random characters). And in fact, in Table 1, you can see that random prefixes had very little effect.

- Re: Table 1: do the authors have a hypothesis for why the main jailbreak was so effective against Mistral-7B? In my experience, the Mistral-7B model (particularly the v0.3-Instruct model) performs very well against jailbreaks. As a suggestion to the authors, adding which version of the models was used (i.e. instruct, guard, version number etc.) would be helpful.
- The paper is very well-written, organized, and clear

- As I understood the paper, the approach is straightforward: for a certain set of prompts that may be susceptible to biased model responses, the authors ask, "can we provide confidence intervals bounding how often the model will produce those unwanted/biased responses by essentially generating lots of responses under pertubations/different conditions?" I wouldn’t be surprised if others called this simplicity out as a weakness — but in fact, I quite liked the simplicity and saw it as a strength. One framing/application of this work I could envision is a way to move away from benchmarks (e.g. 65% on AdvBench) towards confidence intervals (e.g. 40-80% on AdvBench), which may ultimately more meaningful (especially the lower bound).

- Open-source implementation
The work proposes a framework/approach that certifies LLMs for bias on a set of prompts. Certificates provide confidence interval for the likelihood of finding biased responses for a model, a given set of prompts, and a bias detection function. The authors operationalize this framework on the following question: “what is the probability of LLM responses being unbiased for any random set of prompts each mentioning different demographic groups?”
- I felt the use of “certificates” was not well-motivated, and even after reading the paper, I’m not sure why one would prefer certificates over just “confidence intervals.” I am unfamiliar with the literature on certificates, and I would recommend the authors better motivate this beyond talking about how others have restricted certificates to local specifications (Lines 246-255).

- The authors might suggest adding a section to connect with practitioners who want to deploy their tool, i.e. how should the open-source toolkit best be used in practice, and in what circumstances?

- A random suggestion to the authors: you might consider releasing multiple certificates for popular models (something like the Hugging Face leaderboard)
Yes
No ethics review needed.
The proposed distribution of jailbreaks seems somewhat weak as the % unbiased does not decrease significantly except in the instance of GPT-3.5. Why do the authors not consider using some of the learned jailbreaks, such as in [1], which are demonstrated to be more effective in influencing the outputs of LLMs?

Can a synthetic study be generated to verify that these confidence intervals are truly valid?

[1] Zou, et. al. Universal and transferable adversarial attacks on aligned language models.
The authors propose a new approach to provide certifications of unbiasedness (with response to counterfactual prompts), which previously has not been studied. This is a well-motivated problem setting, especially in the case of black-box models.
This paper proposes a framework to detect instances of bias in the outputs of LLMs. The authors propose a notion of a “certificate”, which is a high-confidence bound on the unbiasedness of LLM responses as a result of prompts that mention specific demographic groups. 

The authors consider prefix distributions of random sequences, manual jailbreaks (i.e., interleaving and mutation), and jailbreaks in the embedding space to certify bias.

This is formalized via the notion of counterfactual prompt sets, where a set of prompts only differs in the sensitive attribute (e.g., gender).

Using these counterfactual prompt sets, the authors measure the bias in LLM generations (conditioned on different prompt distributions), where their notion of bias is captured by a BOLD classifier that gives a score for different potential completions.

The authors empirically demonstrate that many LLMs can be influenced to produce harmful or biased outputs by changing the prefix distribution (which is inexpensive to perform). They also quantitatively measure the lack of bias in LLMs (with corresponding high-confidence bounds), showing no trends concerning model scale.
The bias metric seems to match human judgments 76% of the time (as noted in the Appendix). This seems relatively low and brings into question the usability of this metric. I also believe that this information in the Appendix is quite important and should be included in the main text of the paper.
Yes
N/A
No ethics review needed.
Please see weaknesses section for specific questions.  

Most importantly, I wonder how one should understand this work with respect to other related frameworks for producing LLM guarantees, both conceptually and empirically?
This paper addresses multiple important and open questions in LLM deployment.  First, it aims to supplement available tools for evaluating how likely an LLM is to reproduce harmful stereotypes, an issue of paramount importance as these models continue to permeate society.  Second, this work goes beyond typical, noisy benchmarking frameworks, with the goal of providing high-probability guarantees on behavior across prompt distributions of interest.  Their experiments are extensive, offering results across 3 prompt distributions and 9 LLMs.
This work addresses the problem of bias, in particular representational harms, in the outputs of LLMs.  Their approach moves beyond standard, finite-sample benchmarking to providing high-probability guarantees on the probability of bias in some language model responses.  Their method involves certifying the probability of biased responses with respect to counterfactual prompts, measuring how an LLM might produce stereotyped responses, for example with respect to inputs regarding gender and professions.  In order to produce their high-probability bounds, the algorithm involves specifying some prompt prefix distribution of interest from which samples can be generated.  Their experiments offer results across 3 prefix distributions applied to a set of open- and closed-source LLMs.
I think this paper has significant weaknesses that outweigh its strengths, detailed below.  In summary, I find the paper to lack comparison to some highly relevant related work, and its claimed contributions to be largely unsupported.  Also, issues of bias are not well-motivated, or thoroughly explained, in the writing.

-There is a growing body of work in providing high-probability guarantees on LLM behavior and performance.  For just a few examples, including multiple from the last iteration of this conference:

[1] Conformal Language Modeling (https://arxiv.org/abs/2306.10193)
[2] Conformal Autoregressive Generation: Beam Search with Coverage Guarantees (https://arxiv.org/abs/2309.03797)
[3] Mitigating LLM Hallucinations via Conformal Abstention (https://arxiv.org/abs/2405.01563)
[4] Language Models with Conformal Factuality Guarantees (https://arxiv.org/abs/2402.10978)
[5] Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Models (https://arxiv.org/abs/2311.13628)

While the exact nature and uses of these and other existing frameworks may differ from QCB, this would have to be made clear by acknowledging some of this work, and explaining how QCB is different.  The introduction ignores all previous work on LLM guarantees, including those that give high-probability bounds on bias (see next point), and the "Guarantees for LLMs" paragraph in the related works section gives an insufficient treatment.

-[5], published in 2023 and presented at ICLR 2024, offers a framework for producing high-probability bounds on LLM behavior, including examples of applying such a framework to measure bias.  It also seems straightforward to apply this Prompt Risk Control framework to produce bounds on the settings in the experiments.  As such, I find the claims to novelty and "first" (e.g. in the abstract, line 105, and conclusion) to be incorrect.  Also, this stands as a missing baseline in experiments.

-Throughout the paper, I find the treatment given to the issues at hand of bias and representational harm to be lacking in detail.  For example, in the first paragraph, what is an allocative harm, how can it be caused by a representational harm, and when is it illegal?  How can LLMs cause new social hierarchies?  Also, since there is only one set of counterfactual prompts considered in the paper, it would be good to give more details on what exactly the concern is here.  Overall, it seems like bias is just being shoehorned in to increase the importance of the quantitative framework; I think a better approach would be to either present a general framework, with bias as a use case, or give the specific issue(s) being addressed a more thorough treatment.

-The comparison to existing evaluation frameworks in the second paragraph is unclear (as well as lacking the background noted earlier).  For example, how does QCB ameliorate concerns with respect to test set leakage?  Also, given that QCB is ultimately performed on a finite set of prompts, and a seemingly small set of pivot prompts, I am not sure how it resolves "(1) Limited Test Cases".

-The third contribution on "novel insights into the biases of popular LLMs" is unsupported by the rest of paper.

-The definition of bias in lines 177-179 should be sharpened.

-The need to be able to sample from the distribution of interest seems like a weakness compared to a framework like Prompt Risk Control, which can handle the settings tested here as well as the finite-sample setting.

-How are we supposed to understand experiment results?  Can we compare the bounds to some empirical rate, or other bounding methods?  Are these bounds tight?  I am struggling to draw any conclusions in this section.
We present the first framework to formally certify the bias in the responses of LLMs.
@inproceedings{
anonymous2024quantitative,
title={Quantitative Certification of Bias in Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=HQHnhVQznF},
note={under review}
}
Large Language Models (LLMs) can produce biased responses that can cause representational harms. However, conventional studies are insufficient to thoroughly
evaluate LLM bias, as they can not scale to large number of inputs and provide
no guarantees. Therefore, we propose the first framework, QCB (Quantitative
Certification of Bias) that certifies LLMs for bias on distributions of prompts.
A certificate consists of high-confidence bounds on the probability of unbiased
LLM responses for any set of prompts mentioning various demographic groups,
sampled from a distribution. We illustrate the bias certification for distributions of
prompts created by applying varying prefixes drawn from a prefix distributions, to a
given set of prompts. We consider prefix distributions for random token sequences,
mixtures of manual jailbreaks, and jailbreaks in the LLM’s embedding space to
certify bias. We obtain non-trivial certified bounds on the probability of unbiased
responses of SOTA LLMs, exposing their vulnerabilities over distributions
of prompts generated from computationally inexpensive distributions of prefixes.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models Bias Certification
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c89ea38ce7075369587e924ee7b629d44d426563.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/41fe2093747675556c6c7676fae2c6047aab98f6.zip
Quantitative Certification of Bias in Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission