Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach
Linyu Liu Yu Pan Xiaocheng Li Guanting Chen

Spin Supersolid Phase and Double Magnon-Roton Excitations in a Cobalt-based Triangular Lattice
    Wei Li
    April 29, 2024
==============================================================================================



*Equal contribution.
Corresponding to: xiaocheng.li@imperial.ac.uk, guanting@unc.edu.


Large language models (LLMs) are highly capable of many tasks but they can sometimes generate unreliable or inaccurate outputs. To tackle this issue, this paper studies the problem of uncertainty estimation and calibration for LLMs. We begin by formulating the uncertainty estimation problem for LLMs and then propose a supervised approach that takes advantage of the labeled datasets and estimates the uncertainty of the LLMs' responses. Based on the formulation, we illustrate the difference between the uncertainty estimation for LLMs and that for standard ML models and explain why the hidden activations of the LLMs contain uncertainty information. Our designed approach effectively demonstrates the benefits of utilizing hidden activations for enhanced uncertainty estimation across various tasks and shows robust transferability in out-of-distribution settings. Moreover, we distinguish the uncertainty estimation task from the uncertainty calibration task and show that a better uncertainty estimation mode leads to a better calibration performance. In practice, our method is easy to implement and is adaptable to different levels of model transparency including black box, grey box, and white box, each demonstrating strong performance based on the accessibility of the LLM's internal mechanisms.






§ INTRODUCTION




Large language models (LLMs) have marked a significant milestone in the advancement of natural language processing <cit.>, showcasing remarkable capabilities in understanding and generating human-like text. However, one pressing issue for the LLMs is their propensity to hallucinate <cit.> and generate misleading or entirely fabricated information that can significantly undermine their trustworthiness and reliability. The task of uncertainty estimation has then emerged to be an important problem, where an uncertainty estimation model can be used to determine the confidence levels of LLMs' outputs. While the problem of uncertainty estimation and calibration has seen considerable development within the general machine learning and deep learning domains <cit.>, we see less development in the domain of LLMs. One of the major challenges is the difference in the format of the output: while machine learning and deep learning typically involve fixed-dimensional outputs, natural language generation (NLG) tasks central to LLM applications require handling variable outputs that carry semantic meanings, and it is unclear whether the uncertainty estimation for NLG should target for (fixed-dimensional) token level or (variable-dimensional) sentence/semantic-level uncertainty.

Existing uncertainty estimation approaches for LLMs usually involve designing uncertainty metrics for their outputs. For black-box LLMs, these metrics are computed by examining aspects like the generated outputs' consistency, similarity, entropy, and other relevant characteristics <cit.>. Given the complexity of LLMs' underlying architectures, semantic information may be diluted when processing through self-attention mechanisms and during token encoding/decoding. To address this issue, a growing stream of literature argues that hidden layers' activation values within the LLMs offer insights into the LLMs' knowledge and confidence <cit.>. Based on this argument, white-box LLMs, which allow access to more of LLMs' inner values, such as logits and hidden layers, are believed to have the capacity to offer a more nuanced understanding and improved uncertainty estimation results <cit.>.

Most state-of-the-art uncertainty estimation methods in NLG-related tasks are developed in an unsupervised manner <cit.>. However, in the realm of LLMs, there is increasing evidence suggesting the benefits of pursuing supervised approaches. For instance, LLMs' outputs and their internal states can offer conflicting information about truthfulness <cit.>, and determining whether outputs or internal states are more reliable sources of information often varies from one scenario to another. This phenomenon underscores the potential advantages of a supervised learning approach, which can adaptively leverage both types of information for better uncertainty estimation. Meanwhile, employing a supervised learning approach to utilize hidden information within LLMs for uncertainty estimation remains relatively uncharted territory. Prior to the adventure of LLM, a line of literature has studied incorporating supervised learning approaches for uncertainty estimation and calibration <cit.> for white-box <cit.> and black-box <cit.> natural language processing (NLP) models. However, the LLMs distinguish themselves from earlier NLP models through their architectures, training methodologies, and the datasets used. Consequently, there is a need to study and investigate supervised uncertainty estimation approaches specifically tailored for LLMs. There is also a recently expanding body of literature that employs supervised approaches that leverage hidden layers' information for hallucination detection within LLMs <cit.>. However, there is not so much of a consensus on the definition of hallucination while the uncertainty estimation problem is more well defined. Consequently, questions regarding the practicality, effectiveness, and potential of these supervised approaches to improve uncertainty estimation practices in LLM remain unanswered.

Our study embarks on a systematic evaluation of the integration of LLMs' hidden states within the uncertainty estimation framework in a supervised manner. We aim to assess the tangible benefits this approach may offer in improving the confidence, reliability, transferability, and practicality of LLMs' uncertainty estimation. By examining the impact of incorporating the knowledge of these hidden states into the uncertainty estimation methods, our results seek to clarify whether and how the internal states of LLMs can contribute to uncertainty estimation, paving the way for more trustworthy and dependable LLMs.

Our contributions are three-fold: 


  * We formulate the problem of uncertainty estimation for LLMs. Building on this formulation, we explore the nuanced distinctions between uncertainty estimation and calibration for LLMs, and theoretically demonstrate how uncertainty estimation for LLMs differs from that in traditional ML models. Moreover, our theoretical analysis suggests that the existing architecture and training procedures of LLMs require additional features from their input and output for improved uncertainty estimation. 


  * Motivated by the above findings, we propose a supervised method for the uncertainty estimation problem. Specifically, the method aims to train an uncertainty estimation function that maps the hidden activations of the LLMs, and the probability-related information of the LLMs (when generating the response) to an uncertainty score that captures the LLM's confidence about its response. This supervised approach is systematically designed for straightforward implementation and broad applicability, suitable for black-box, grey-box, and white-box LLMs.



  * We conduct numerical experiments on NLP tasks including question answering, machine translation, and multiple choice to evaluate the performance of our proposed approach against existing benchmarks. When designing these experiments, we measure the performance of our approach across LLMs with different accessibility mode (black-box, grey-box, and white-box), and evaluate them in both in-distribution and out-of-distribution test datasets. The results demonstrate that by leveraging hidden activations of LLMs, our method consistently extracts additional knowledge from these models to enhance uncertainty estimation across various NLP tasks. These findings provide insights into the working mechanism of the uncertainty estimation method and its robustness and transferability. 

    
 






 §.§ Related literature




  
Uncertainty estimation for natural language generalization.
The uncertainty estimation and calibration for traditional machine learning is relatively well-studied <cit.>. However, with the rapid development of LLMs, there is a pressing need to better understand the uncertainty for LLMs' responses, and measuring the uncertainty from sentences instead of a fixed-dimension output is more challenging. One stream of work has been focusing on unsupervised methods that leverage entropy <cit.>, similarity <cit.>, semantic <cit.>, logit or hidden states' information <cit.> to craft a uncertainty metric that helps to quantify uncertainty. For black-box models, some of the metrics can be computed based on multiple sampled output of the LLMs <cit.>; while for white-box models, more information such as the output's distribution, the value of the logit and hidden layers make computing the uncertainty metric easier. We also refer to <cit.> for other related uncertainty estimation methods such as calibration and conformal prediction.



  
Hallucination detection. Recently, there is a trend of adopting uncertainty estimation approaches for hallucination detection. The rationale is that the information of the value of logits and the hidden states contain some of the LLMs' beliefs about the trustworthiness of its generated output. By taking the activations of hidden layers as input, <cit.> train a classifier to predict hallucinations, and <cit.> develop epistemic neural networks aimed at reducing hallucinations. <cit.> demonstrate that the information from hidden layers of LLMs' output can indicate the answerability of an input query, providing indirect insights into hallucination occurrences. <cit.> develop an unsupervised metric that leverages the internal states of LLMs to perform hallucination detection. More related works on hallucination detection can be found in <cit.>. While there is a lack of a rigorous definition of hallucination, and its definition varies in the above-mentioned literature, the uncertainty estimation problem can be well defined, and our results on uncertainty estimation can also help the task of hallucination detection. 



  
Leveraging LLMs' hidden activation.
The exploration of hidden states within LLMs has been studied to better understand LLMs' behavior. <cit.> leverage the language model's hidden states to train a calibrator that predicts the likelihood of outputs' correctness. With an unsupervised approach, <cit.> utilizes hidden activations in language models to represent knowledge about the trustfulness of their outputs. <cit.> show that LLMs' outputs and their internal states can offer conflicting information about truthfulness, and determining whether outputs or internal states are more reliable sources of information often varies from one scenario to another. By taking the activations of hidden layers as input, <cit.> employ a linear probe to show that hidden layers' information from LLMs can be used to differentiate between epistemic and aleatoric uncertainty. <cit.> experimentally reveal the variations in hidden layers' activations when LLMs generate true versus false responses. Lastly, <cit.> enhance the truthfulness of LLMs during inference time by adjusting the hidden activations' values in specific directions.




    <cit.> <cit.> Probe for hallucination. 


Transferability. 


<cit.> 

 
Middle layer choice. <cit.>





















§ PROBLEM SETUP


Consider the following environment where one interacts with LLMs through prompts and responses: An LLM is given with an input prompt x=(x_1,x_2,...,x_k)∈𝒳 with x_i∈𝒱 representing the i-th token of the prompt. Here 𝒱 denotes the vocabulary for all the tokens. Then the LLM generates its response y = (y_1,y_2,...,y_m)∈𝒴 (randomly) following the probability distribution 

    y_j ∼  p_θ(·|x,y_1,y_2,...,y_j-1).

Here the probability distribution p_θ denotes the distribution (over vocabulary 𝒱) as the LLM's output, and θ encapsulates all the parameters of the LLM. The conditional part includes the prompt x and all the tokens y_1,y_2,...,y_j-1 generated preceding the current position. We allow both the prompt and the response to have a variable length of k and m.

We consider using the LLM for some downstream NLP tasks such as question answering, multiple choice, and machine translation. Such a task usually comes with an evaluation/scoring function that evaluates the quality of the generated response 

    s(·, ·): 𝒴×𝒴→ [0,1].

For each pair of (x,y), the evaluation function rates the response y with the score z s(y_true, y) where y_true is the true response for the prompt x. The true response y_true is usually decided by factual truth, humans, or domain experts. It does not hurt to assume a larger score represents a better answer; z=1 indicates a perfect answer, while z=0 says the response y is off the target. 

We define the task of uncertainty estimation for LLMs as the learning of a function g that predicts the score

    g(x, y) ≈𝔼[s(y, y_true)|x, y]

where the expectation on the right-hand side is taken with respect to the (possible) randomness of the true response y_true. We emphasize two points on this task definition: 


  * The uncertainty function g takes the prompt x and y as its inputs, which has the following implications. First, the true and predicted uncertainty score can and should depend on the specific realization of the response y. Second, the uncertainty function g does not require the true response y_true as the input. Since true response data are often limited and typically only available from labeled datasets or human experts, this design broadens the applicability of the function across various settings, enhancing its practical usage.

  * The uncertainty score function g, in the language of uncertainty calibration of ML models <cit.>, is defined in an individual (conditional) sense. That is, the predicted score g(x, y) is hopefully precise and matches the true score on an individual level (for each prompt-response pair) but not on the population level (for a distribution of prompt-response pairs). With such a definition of the uncertainty quantification task, the score can be useful in that a well-calibrated score guides the extent to which the users should trust the response. 
 

In the following sections, we will first describe our method as a generic framework of uncertainty estimation that formulates the problem as a supervised task and utilizes the available labeled dataset. Then we discuss how this approach can be effectively integrated in the state-of-the-art LLMs, and present the empirical experiments and findings. Finally, we discuss the relationship between our method and the existing literature on fine-tuning LLM and hallucination detection.












§ UNCERTAINTY ESTIMATION VIA SUPERVISED CALIBRATION



In Section <ref>, we present our method of supervised calibration as a post hoc procedure to estimate the uncertainty of the LLM's responses.





 §.§ Supervised calibration

We consider a supervised approach of learning the uncertainty function g: 𝒳×𝒴→ [0,1], which is similar to the standard setting of uncertainty quantification for ML/deep learning models. First, we start with a dataset of n samples 

    𝒟_raw = {(x_i, y_i, y_i, true, s(y_i, y_i, true))}_i=1^n.

𝒟_raw can be generated based on a labeled dataset for the tasks we consider. Here x_i=(x_i,1,...,x_i,k_i) and y_i=(y_i,1,...,y_i,m_i) denote the prompt and the corresponding LLM's response, respectively. y_i, true denotes the true response (that comes from the labeled dataset) of x_i, and s(y_i, y_i, true) assigns a score for the response y_i based on the true answer y_i, true. We remark that when generating the dataset 𝒟_raw with the LLM, there can be multiple samples with the same prompt for tasks such as question answering and machine translation. That is, we can generate multiple samples with x_i ≡x, but these samples may have different responses y_i's due to the probabilistic nature of the LLM. Meanwhile, because of the same x_i, these samples have the same true response y_i, true. Each different realization of y_i constitutes one meaningful sample for the uncertainty estimation task.

The next step is to formulate a supervised learning task based on the dataset 𝒟_raw. Specifically, we construct the following dataset 

    𝒟_un = {(v_i, z_i)}_i=1^n

where z_i  s(y_i, y_i, true)∈ [0,1] denotes the target score to be predicted. The vector v_i summarizes useful features for the i-th sample based on (x_i,y_i) for the prediction task. In this light, a supervised learning task on the dataset 𝒟_un corresponds exactly to the definition of the uncertainty estimation task in (<ref>).

Now we discuss how we construct the feature vector v_i based on (x_i, y_i). We mainly consider features from two sources. 


  * White-box features: LLM's hidden-layer activations. We feed (x_i, y_i) as input into a LLM, and extract the corresponding hidden layers' activations of the LLM. Given the architecture of the decoder-based transformer, the consensus is that the activations of the last token (of an input sequence) should contain the most information <cit.>. Thus we use the hidden layers' activation of the last token of (x_i, y_i) as part of v_i. We defer more discussions on the choice of the layer to Section <ref>.


  * Grey-box features: Entropy- or probability-related outputs. The entropy of a discrete distribution p over the vocabulary 𝒱 is defined by 

    H(p) -∑_v∈𝒱 p(v)log(p(v)).

For a prompt-response pair (x,y)=(x_1,...,x_k,y_1,...,y_m), we consider as the features the entropy at each token such as H(p_θ(·|x_1,...,x_j-1)) and H(p_θ(·|x,y_1,...,y_j-1)) where p_θ denotes the LLM. 

We defer more discussions on feature construction to Appendix <ref>. Also, we note that another direct feature for predicting z_i is to ask the LLM “how certain it is about the response” and incorporate its response to this question as a feature for predicting z_i <cit.>. This is naturally a viable option, and there are also other sources of features that can be incorporated into v_i such as the hidden activations for the second last token. We do not try to exhaust all the possible features, and the aim of our paper is more about formulating the uncertainty estimation for the LLMs as a supervised task and understanding how the internal states of the LLM encode uncertainty. To the best of our knowledge, our paper is the first one to do so. Specifically, the above formulation aims for the following two outcomes:



  * We train an uncertainty model ĝ(v_i) that predicts z_i. Based on the dataset 𝒟_un = {(v_i, z_i)}_i=1^n, one can learn a model ĝ that predicts the uncertainty z_i with summarized features v_i. Intuitively, this approach will give better performance (because of its supervised nature and the reason presented in Section <ref>) than the ad-hoc/black-box methods such as the entropy-based method <cit.>, similarity-based method <cit.>, etc., which is indeed verified by the numerical experiments in the next section. Thus this result underscores the usefulness of the supervised dataset for predicting uncertainty which is more aligned with the canonical uncertainty quantification methods. 

  * We aim to answer the question of whether the hidden layers carry the uncertainty information. We note that using hidden layer information is not a mainstream approach for the existing uncertainty estimation literature on ML models. This highlights the difference between the uncertainty estimation for classic deep learning models and that for LLMs. We provide some theoretical insights in the following sections and show by theoretical insights and empirical experiments why the supervised approach can extract additional knowledge of LLMs to enhance uncertainty estimation.





 §.§ Uncertainty estimation v.s. uncertainty calibration




So far in this paper, we focus on the uncertainty estimation task which aims to predict whether the LLM makes mistakes in its response or not. There is a different but related task known as the uncertainty calibration problem. In comparison, the uncertainty calibration aims to ensure that the output from the uncertainty estimation model such as ĝ in above conveys a probabilistic meaning. In this sense, our paper is more focused on addressing the predictability of the LLM's uncertainty, and we find that our supervised formulation and the LLM's hidden activations are helpful in such a prediction task. In terms of the uncertainty calibration aspect, our developed uncertainty estimation model is compatible with all the recalibration methods for ML models in the literature of uncertainty calibration. And intuitively, a better uncertainty estimation/prediction will lead to a better-calibrated uncertainty model, which is also verified in our numerical experiments in Section <ref>. 





 §.§ Why hidden layers as  features?


In this subsection, we provide a simple theoretical explanation for why the hidden activations of the LLM can be useful in uncertainty estimation. Consider a binary classification task where the features X∈ℝ^d and the label Y∈{0,1} are drawn from a distribution 𝒫. We aim to learn a model f: ℝ^d→ [0,1] that predicts the label Y from the feature vector X, and the learning of the model employs a loss function l(·, ·):[0,1]× [0,1]→ℝ.


Let ℱ be the class of measurable function that maps from ℝ^d to [0,1]. Under the cross-entropy loss l(y, ŷ)= ylog (ŷ)+(1-y)log (1-ŷ), the function f^* that minimizes the loss

    f^* = _f∈ℱ𝔼[l(Y, f(X))]

is the Bayes optimal classifier

    f^*(x) = ℙ(Y=1|X=x)

where the expectation and the probability are taken with respect to (X,Y)∼𝒫. Moreover, the following conditional independence holds 

    Y⊥X |  f^*(X).




The proposition is not technical and it can be easily proved by using the structure of f^*(X). It states a nice (probably well-known) property of the cross-entropy loss. Specifically, the function learned under the cross-entropy loss coincides with the Bayes optimal classifier. Note that this is contingent on two requirements. First, the function class ℱ is the measurable function class. Essentially, it suffices to require the function class ℱ to be rich enough to cover the Bayesian optimal classifier, also known as the realizability condition. With the capacity of the large vision and NLP models, we can think this requirement to be (at least approximately) met. Second, it requires the function f^* learned through the population loss rather than the empirical loss/risk consisting of the training samples. With a large size of training data, we can think this requirement also to be (approximately) met.
 
The proposition also states one step further on conditional independence Y⊥X |  f^*(X). This means all the information related to the label Y that is contained in X is summarized in the prediction function f^*. While the numeric situation cannot be captured by such a simple theory (due to these two requirements), it provides insights into why the existing uncertainty quantification/calibration/recalibration methods do not utilize much of the original feature X or the hidden-layer activations. Specifically, when a prediction model f̂:ℝ^d → [0,1] is well-trained, the predicted score f̂(X) should capture all the information about the true label Y contained in the features X, and there is no need to get the features X re-involved in the recalibration procedure when one adjusts the prediction model f̂ to meet some calibration objectives. This indeed explains why the classic uncertainty quantification and calibration methods only work with the predicted score f̂(X) for re-calibration, including Platt
scaling <cit.>, isotonic regression <cit.>, temperature scaling <cit.>, etc.  

When it comes to LLMs, we will no longer have conditional independence, and that requires additional procedures to retrieve more information on Y. The following corollary states that when the underlying loss function l̃ does not possess this nice property (the Bayes classifier minimizes the loss point-wise) of the cross-entropy loss, the conditional independence will collapse. 


Suppose the loss function l̃ satisfies 

    ℙ(f^*(x) ≠_ỹ∈[0,1]𝔼[l̃(Y,ỹ)|X=x] )>0,

where f^* is defined as Proposition <ref>, then for the function

    f̃ = _f∈ℱ𝔼[l̃(Y, f(X))],

where the expectation is with respect to (X,Y)∼𝒫, there exists a distribution 𝒫 such that the conditional independence no longer holds

    Y ⊥̸X |  f̃(X).




Proposition <ref> and Corollary <ref> together illustrate the difference between uncertainty estimation for a traditional ML model and that for LLMs. For the traditional ML models, the cross-entropy loss which is commonly used for training the model is aligned toward the uncertainty calibration objective. When it comes to uncertainty estimation for LLMs, the label Y can be viewed as the binary variable for whether the LLM's response is correct, and the features X represent some features extracted from the prompt-response pair. Meanwhile, the LLMs are often pre-trained with some other loss functions (for example, the negative log-likelihood loss for next-token prediction), and this causes a misalignment between the model pre-training and the uncertainty quantification task. The consequence is the collapse of conditional independence in Corollary <ref>. The collapse is of a deeper extent than the violation of the two requirements as discussed above. Consequently, the original features, the prompt-response pair, may and should (in theory) contain information about the uncertainty score Y that cannot be fully captured by f̃(X). This justifies why we formulate the uncertainty estimation task as the previous subsection and take the hidden-layer activations as features to predict the uncertainty score; it also explains why we do not see much similar treatment in the mainstream uncertainty quantification literature. 





 §.§ Three regimes of supervised uncertainty estimation



In Section <ref>, we present the method of supervised uncertainty estimation under the assumption that we know the parameters of the LLMs including both the hidden activation and the output probabilities. Now we categorize the application of our method into three regimes to solve the case where the LLM is black-box and we do not have access to the parameter information.  


A natural implementation of our supervised learning approach involves using an LLM to generate the response y for input x, and extracting insights on confidence from the same LLM's hidden layers' activations. This method functions effectively with white-box LLMs where hidden activations are accessible, but with black-box LLMs, which restrict access to hidden activations, alternative black-box uncertainty estimation methods become necessary.

We observe that obtaining hidden layers' activations merely requires an LLM and the prompt-response pair (x, y). Therefore, it is not mandatory for (x, y) to be generated by the LLM that provides the hidden layers' activations. Based on this observation, we note that the extra knowledge of uncertainty can come from the hidden layers of any white-box LLM that takes as input the (x, y) pair, not necessarily from the LLM that generates (x, y). It is natural to start with the belief that the LLM which generates (x,y) should have more information about the uncertainty of y. However, any white-box LLM can output the hidden activations corresponding to (x, y). 

To clarify further, once (x,y) is generated by the LLM p_θ, all of p_θ's knowledge about the uncertainty of y is encoded within its internal representations. However, another LLM q_θ can also evaluate (x,y), with its understanding of uncertainty reflected in q_θ's internal representations as well. 





Next, we formally present our supervised uncertainty calibration method for white-box, grey-box, and black-box LLMs.

White-box supervised uncertainty estimation (Wb-S): This Wb-S approach implements the method discussed in Section <ref>. It first constructs the features from both two sources and trains a supervised model to predict the uncertainty of the LLM's responses. 

Grey-box supervised uncertainty estimation (Gb-S): This Gb-S regime constructs the features only from the grey-box source, that is, those features relying on the probability and the entropy (such as those in Table <ref> in Appendix <ref>) but it ignores the hidden-layer activations. 

Both the above two regimes consider one single LLM throughout the whole procedure. Specifically, the dataset 𝒟_raw is generated based on the LLM p_θ, and then features are extracted from 𝒟_raw to construct 𝒟_un. In particular, the feature extraction is based on the same LLM p_θ as well and this assumes the full knowledge of p_θ as the following diagram

    𝒟_rawp_θ⟶𝒟_un.
 Then a supervised uncertainty estimation model is trained upon 𝒟_un.


Black-box supervised uncertainty estimation (Bb-S): The Bb-S regime does not assume the knowledge of the parameters of p_θ but still aims to estimate its uncertainty. To achieve this, it considers another open-source LLM denoted by q_θ. The original data 𝒟_raw is generated by p_θ but then the uncertainty estimation data 𝒟_un is constructed based on q_θ from 𝒟_raw as illustrated in the following diagram

    𝒟_rawq_θ⟶𝒟_un.

For example, for a prompt x, a black-box LLM p_θ generates the response y. We utilize the open-source LLM q_θ to treat (x, y) jointly as a sequence of (prompt) tokens and extract the features of hidden activations and entropy as in Section <ref>. In this way, we use q_θ together with the learned uncertainty model from 𝒟_un to estimate the uncertainty of responses generated from p_θ which we do not have any knowledge about. 


A natural way of conducting our supervised learning approach is that we employ a LLM to generate the response y for the input x, and use the hidden layers' activation of the same LLM to extract its insights on confidence. This strategy, however, has a significant limitation. It works well when the LLM we are using is a white-box one such that we can access to its internal workings. However, when employing black-box LLMs, where internal details are inaccessible, we must resort to alternative black-box UQ methods.

To address this issue, we start with a key observation. Obtaining hidden layers' activations merely requires an LLM and the prompt-response pair (x, y). Therefore, it is not mandatory for (x, y) to be generated by the LLM that provides the hidden layers' activations. 

Based on this observation, we emphasize that the extra knowledge of uncertainty comes from the LLM that evaluates the (x, y) pair, not necessarily from the LLM that generates (x, y) (As shown in figure X). It is natural to start with the belief that the LLM which generates (x,y) should have more information about the uncertainty of y. However, any LLM can assess the uncertainty of a given input of (x, y). In other words, once (x,y) is produced by, for instance, LLM A, all of LLM A's knowledge about uncertainty of y is encoded within its internal representations. However, LLM B can also evaluate (x,y), with its understanding of y's uncertainty reflected in its own internal representations as well. Therefore, an LLM with better capacity to evaluate uncertainty should be chosen to analyze the hidden layers' activations. We hypothesize that larger LLMs have a better understanding of the uncertainty related to (x,y) compared to smaller models, and we verify this hypothesis in section X.

To sum up, by choosing a white-box LLM to evaluate the prompt-response pair (as shown in figure X), we can use the supervised learning approach for black-box LLMs.




Algorithm <ref> summarizes our discussions so far on the supervised approach for uncertainty estimation. When the target LLM p_θ=q_θ, it corresponds to the first two regimes (white-box and grey-box). When the target LLM p_θ≠ q_θ, it corresponds to the third regime (black-box). 




§ NUMERIAL EXPERIMENTS AND FINDINGS


In this section, we provide a systematic evaluation of the proposed supervised approach for estimating the uncertainty of the LLMs. All code used in our experiments is available at https://github.com/LoveCatc/supervised-llm-uncertainty-estimationhttps://github.com/LoveCatc/supervised-llm-uncertainty-estimation.

In this section, we present our systematic evaluations of the supervised learning approach for LLMs. 

We try to investigate questions that we believe are important for the LLM community:

    
  * Effectiveness: We evaluate the effectiveness of incorporating LLMs' hidden states into UQ across various datasets. Considering LLMs' wide applicability in numerous downstream tasks, we also examine the robustness of our supervised UQ approach in handling OOD data. Furthermore, we evaluate strategies we mentioned in section <ref>. By utilizing a white-box LLM's hidden layer to perform UQ, we experiment with outputs generated by both black-box and white-box LLMs, aiming to optimize UQ across various model types and application scenarios. These efforts seek to validate the effectiveness and adaptability of leveraging hidden states to enhance UQ in diverse contexts.
    
  * Commonality: Within the context of LLMs' transformer-based architecture, we explore strategies of extracting the most valuable information from transformer's hidden layers. We investigate if these findings and properties generalize across different LLMs (For example, the consistency of the effectiveness when incorporating hidden layers across different LLMs). This exploration aims to understand the universality and adaptability of our approach within the diverse landscape of LLM applications.
    
    
    
    






 §.§ LLMs, tasks, benchmarks, and performance metrics


Here we outline the general setup of the numerical experiments. Certain tasks may deviate from the general setup, and we will detail the specific adjustments as needed. 

LLMs. For our numerical experiments, we mainly consider two open-source LLMs, LLaMA2-7B <cit.> and Gemma-7B <cit.> as p_θ defined in Section <ref>. For certain experiments, we also employ the models of LLaMA2-13B and Gemma-2B. We also use their respective tokenizers as provided by Hugging Face. We do not change the parameters/weights θ of these LLMs.

Tasks and Datasets. We mainly consider three tasks for uncertainty estimation, question answering, multiple choice, and machine translation. All the labeled datasets for these tasks are in the form of {(x_i, y_i,true)}_i=1^n where x_i can be viewed as the prompt for the i-th sample and y_i,true the true response. We adopt the few-shot prompting when generating the LLM's response y_i, and we use 5 examples in the prompt of the multiple-choice task and 3 examples for the remaining natural language generation tasks. This enables the LLM's in-context learning ability <cit.> and ensures the LLM's responses are in a desirable format. We defer more details of the few-shot prompting to Appendix <ref>. The three tasks are:



  * Question answering. We follow <cit.> and use the CoQA and TriviaQA <cit.> datasets. The CoQA task requires the LLM to answer questions by understanding the provided text, and the TriviaQA requires the LLM to answer questions based on its pre-training knowledge. We adopt the scoring function s(·, ·) as Rouge-1 <cit.> and label a response y_i as correct if s(y_i, y_i,true)≥ 0.3 and incorrect otherwise.

  * Multiple choice. We consider the Massive Multitask Language Understanding (MMLU) dataset <cit.>, a collection of 15,858 questions covering 57 subjects across STEM. Due to the special structure of the dataset, the generated output y_i and the correct answer y_true, i∈{A, B, C, D}. Therefore, this task can also be regarded as a classification problem for the LLM by answering the question with one of the four candidate choices. 

  * Machine translation. We consider the WMT 2014 dataset <cit.> for estimating LLM's uncertainty on the machine translation task. The scoring function s(·, ·) is chosen to be the BLEU score <cit.> and the generated answer y_i is labeled as correct if s(y_i, y_i,true)>0.3 and incorrect otherwise.





Benchmarks. We compare our approach with a number of the state-of-the-art benchmarks for the problem. <cit.> give a comprehensive survey of the existing methods and compare four distinct measures for predicting sentence generation uncertainty. The measures are based on either the maximum or average values of entropy or probability across the sentence, including Max Likelihood, Avg Likelihood, Max Ent, and Avg Ent defined in Table <ref>. We note that each of these measures can be applied as a single uncertainty estimator, and they are all applied in an unsupervised manner that does not require additional supervised training. In particular, in applying these measures for the MMLU dataset, since the answer only contains one token from {A, B, C, D}, we use the probabilities and the entropy (over these four tokens) as the benchmarks which represent the probability of the most likely choice and the entropy of all choices, respectively. <cit.>  generate multiple answers, compute their entropy in a semantic sense, and define the quantity as semantic entropy. This semantic-entropy uncertainty (SU) thus can be used as an uncertainty estimator for the LLM's responses. <cit.> propose the approach of asking the LLM for its confidence (denoted as A4U) which directly obtains the uncertainty score from the LLM itself. 


Our methods. We follow the discussions in Section <ref> and implement three versions of our proposed supervised approach: black-box supervised (Bb-S), grey-box supervised (Gb-S), and white-box supervised (Wb-S). These models have the same pipeline of training the uncertainty estimation model and the difference is only on the availability of the LLM. For the Bb-S method, we use the Gemma-7B as the model q_θ to evaluate the uncertainty of LLaMA-7B p_θ (treated as a black-box), and reversely, use LLaMA-7B to evaluate Gemma-7B. The supervised uncertainty model ĝ is trained based on the random forest model <cit.>. Details on the feature construction and the training of the random forest model are deferred to Appendix <ref>.









Performance metrics. For the model evaluation, we follow <cit.> and compare the performance of our methods against the benchmark using the generated uncertainty score to predict whether the answer is correct. The area under the receiver operator characteristic curve (AUROC) metric is employed to measure the performance of the uncertainty estimation. As discussed in Section <ref>, AUROC works as a good metric for the uncertainty estimation task whereas for the uncertainty calibration task, we follow the more standard calibration metrics and present the results in Section <ref>.




 §.§ Performance of uncertainty estimation



Now we present the performance on the uncertainty estimation task.






  §.§.§ Question answering and machine translation


The question answering and machine translation tasks can all be viewed as natural language generation tasks so we present their results together. Table <ref> summarizes the three versions of our proposed supervised method against the existing benchmarks in terms of AUROC. 

























We make several remarks on the numerical results. First, our methods generally have a better performance than the existing benchmarks. Note that the existing benchmarks are mainly unsupervised and based on one single score, and also that our method proceeds with the most standard pipeline for supervised training of an uncertainty estimation model. The advantage of our method should be attributed to the supervised nature and the labeled dataset. While these unsupervised benchmark methods can work in a larger scope than these NLP tasks (though they have not been extensively tested on open questions yet), our methods rely on the labeled dataset. But in addition to these better numbers, the experiment results show the potential of labeled datasets for understanding the uncertainty in LLM's responses. In particular, our method Gb-S uses the exactly same features as the benchmark methods, and it shows that some minor supervised training can improve a lot upon the ad-hoc uncertainty estimation based on one single score such as Max Pro or Max Ent.

Second, our method Wb-S has a clear advantage over our method Gb-S. Note that these two methods differ in that the Wb-S uses the hidden activations while the Gb-S only uses probability-related (and entropy-related) features. This implies that the hidden activations do contain uncertainty information which we will investigate more in the next subsection. Also, we note from the table that there is no single unsupervised grey-box method (under the Benchmarks columns) that consistently surpasses others across different datasets/NLP tasks. For example, among all these unsupervised benchmark methods, Avg Ent emerges as a top-performing one for the Gemma-7B model when applied to the machine translation task, but it shows the poorest performance for the same Gemma-7B model when tested on the question-answering TriviaQA dataset. This inconsistency highlights some caveats when using the unsupervised approach for uncertainty estimation of LLMs.

Lastly, we note that the Bb-S method has a similar or even better performance as the Wb-S method. As discussed in Section <ref>, the performance of uncertainty estimation relies on the LLM that we use to evaluate the prompt-response pair. Therefore, it is not surprising to see that in the TriviaQA task, for answers generated by Gemma-7B, Bb-S features better uncertainty estimation than Wb-S, possibly because LLaMA2-7B, the LLM that is used as the “tool LLM” in Algorithm <ref>, encodes better knowledge about the uncertainty of the answers than Gemma-7B. We also note that the performance of Bb-S is not always as good as Wb-S, and we hypothesize that it is because LLMs' output distribution differs, which could result in evaluating the uncertainty of different answers. Despite these inconsistencies, the performance of Bb-S is still strong, and these results point to a potential future avenue for estimating the uncertainty of closed-source LLMs. 













  §.§.§ Multiple choice (MMLU)


Table <ref> presents the performance of our methods against the benchmark methods on the MMLU dataset. For this multiple choice task, the output is from {A,B,C,D} which bears no semantic meaning, and therefore we do not include the Semantic Uncertainty (SU) as Table <ref>. The results show the advantage of our proposed supervised approach, consistent with the previous findings in Table <ref>. 












 §.§ Interpreting the results



Now we use some visualizations to provide insights into the working mechanism of the uncertainty estimation procedure for LLMs and to better understand the experiment results in the previous subsection. 









  §.§.§ Layer comparison


For general LLMs, each token is associated with a relatively large number of hidden layers (32 layers for LLaMA2-7B for example), each of which is represented by high-dimensional vectors (4096 for LLaMA2-7B). Thus it is generally not a good practice to incorporate all hidden layers as features for the uncertainty estimation due to this dimensionality. Previous works find that the middle layer and the last layer activations of the LLM's last token contain the most useful features for supervised learning <cit.>. To investigate the layer-wise effect for uncertainty estimation, we implement our Wb-S method with features different in two aspects: (i) different layers within the LLM architecture, specifically focusing on the middle and last layers (e.g., LLaMA2-7B: 16th and 32nd layers out of 32 layers with 4096 dimensions; Gemma-7B: 14th and 28th layers out of 28 layers with 3072 dimensions); and (ii) position of token activations, including averaging hidden activations over all the prompt/answer tokens or utilizing the hidden activation of the last token. The second aspect makes sense when the output contains more than one token, so we conduct this experiment on the natural language generation tasks only. Figure <ref> gives a visualization of the comparison result. While the performances of these different feature extraction ways are quite similar in terms of performance across different tasks and LLMs, activation features from the middle layer generally perform better than the last layer. This may come from the fact that the last layer focuses more on the generation of the next token instead of summarizing information of the whole sentence, as has been discussed by <cit.>.





  §.§.§ Scaling effect

In Figure <ref>, we investigate whether larger LLMs' hidden activations enhance our uncertainty estimation method. For a fair comparison, we fix the target LLM that generates the output in Algorithm <ref> and vary the tool LLM used for analysis. For example, in the left plot of Figure <ref>, we use Gemma-7B to generate the outputs, and LLaMA2-7B, LLaMA2-13B, and Gemma-7B to perform uncertainty estimation.



We find that there is no significant performance difference between LLaMA2-7B and LLaMA2-13B, nor between Gemma-2B and Gemma-7B (Gemma does not offer a 13B model). Our result suggests that larger LLM do not necessarily encode better knowledge about uncertainty. Furthermore, based on the performance of Wb-S in Figure <ref>, we suggest use the same LLM to generate the output and evaluate the uncertainty.




  §.§.§ Histogram of correlations


Figure <ref> plots the histograms of the pairwise correlations between the neuron activations and the labels
(whether the LLM’s response is correct). We make two observations here: First, for both LLMs, some neurons have a significantly positive (or negative) correlation with the label. We can interpret these neurons as the uncertainty neuron for the corresponding task. When these neurons are activated, the LLMs are uncertain about their responses. Second, Gemma-7B has more significant neurons than LLaMA2-7B, and this is consistent with the better performance of Gemma-7B in Table <ref> and Table <ref>. Also, this reinforces that the hidden activations of the LLMs contain uncertainty information about the LLM's output.



Figure <ref> plots some example neurons' activation by selecting the neurons with the largest and smallest correlations in Figure <ref>. More such neurons can be found in 
Figure <ref> in the Appendix. These neurons as an individual indicator exhibit different distributional patterns when the response is correct compared to when the response is incorrect, and thus reflect the uncertainty of the LLM's responses. 







































 §.§ Calibration performance




In Section <ref>, we distinguish the two tasks of uncertainty estimation and uncertainty calibration. Throughout the paper, we have been focused on improving the performance on the task of uncertainty estimation – to predict when the LLM is uncertain about its response. Generally, a better uncertainty estimation model leads to one with better calibration performance. The calibration (or recalibration) of the uncertainty estimation model can be indeed reduced to the classic ML setting which does not involve the LLM. Table <ref> gives the calibration performance and we see an advantage of our supervised methods over benchmark methods consistent with the AUROC performance in Table <ref>. We adopt the histogram binning method here because we find that the temperature scaling method and the Platt scaling method will give all predicted scores concentrated within a small range such as [0.2,0.6]. We also do not exclude the possibility that the other calibration methods can give even better performance. The point to make here is that uncertainty estimation and uncertainty calibration are two closely related tasks. Note that (i) a better uncertainty estimation model leads to a better calibration performance and (ii) the LLMs are pre-trained and not designed for these NLP tasks in the first place (see Section <ref>) so that there is no uncertainty score readily available (as the predicted probabilities for the image classifiers); we emphasize the importance of an extra uncertainty estimation procedure as our supervised one so to extract the uncertainty information from the inside of the LLMs.











    




 §.§ Transferability

In this subsection, we evaluate the robustness of our methods under the out-of-distribution (OOD) setting for both question-answering and multiple-choice tasks.








Setup for the OOD multiple-choice task. We split the MMLU datasets into two groups based on the subjects: Group 1 contains questions from the first 40 subjects while Group 2 contains the remaining 17 subjects, such that the test dataset size of each group is similar (around 600 questions). Note that these 57 subjects span a diverse range of topics, and this means the training and test set can be very different. To test the OOD robustness, we train the proposed methods on one group and evaluate the performance on the other group. 

Setup for the OOD question-answering task. For the QA task, since we have two datasets (CoQA and TriviaQA), we train the supervised model on either the TriviaQA or CoQA dataset and then evaluate its performance on the other dataset. While both datasets are for question-answering purposes, they diverge notably in two key aspects: (i) CoQA prioritizes assessing the LLM's comprehension through the discernment of correct responses within extensive contextual passages, while TriviaQA focuses on evaluating the model's recall of factual knowledge. (ii) TriviaQA typically contains answers comprising single words or short phrases, while CoQA includes responses of varying lengths, ranging from shorter to more extensive answers.































Table <ref> summarizes the performance of these OOD experiments. As expected, for all the methods, there is a slight drop in terms of performance compared to the in-distribution setting (reported by the numbers in the parentheses in the table). We make the following observations based on the experiment results. First, based on the performance gap between in-distribution and OOD evaluation, it is evident that although incorporating white-box features such as hidden activations makes the model more susceptible to performance decreases on OOD tasks, these features also enhance the uncertainty estimation model's overall capacity, and the benefits outweigh the drawbacks. It is also noteworthy that even in these scenarios of OOD, our Wb-S and Bb-S method almost consistently outperform baseline approaches. Overall, the robustness of our methods shows that the hidden layers' activations within the LLM exhibit similar patterns in encoding uncertainty information to some extent. The performance drop (from in-distribution to OOD) observed in the MMLU dataset is notably less than that in the question-answering dataset, which may stem from the larger disparity between the CoQA and TriviaQA datasets compared to that between two distinct groups of subjects within the same MMLU dataset. This suggests that in cases of significant distributional shifts, re-training or re-calibrating the uncertainty estimation model using test data may be helpful.




Alignment data. 







§ CONCLUSIONS


In this paper, we study the problem of uncertainty estimation and calibration for LLMs. We follow a simple and standard supervised idea and use the labeled NLP datasets to train an uncertainty estimation model for LLMs. Our finding is that, first, the proposed supervised methods have better performances than the existing unsupervised methods. Second, the hidden activations of the LLMs contain uncertainty information about the LLMs' responses. Third, the black-box regime of our approach (Bb-S) provides a new approach to estimating the uncertainty of closed-source LLMs. Lastly, we distinguish the task of uncertainty estimation from uncertainty calibration and show that a better uncertainty estimation model leads to better calibration performance. We also remark on the following two aspects:



  * Fine-tuning: For all the numerical experiments in this paper, we do not perform any fine-tuning with respect to the underlying LLMs. While the fine-tuning procedure generally boosts the LLMs' performance on a downstream task, our methods can still be applied for a fine-tuned LLM, which we leave as future work. 

  * Hallucination: The hallucination problem has been widely studied in the LLM literature. Yet, as mentioned earlier, it seems there is no consensus on a rigorous definition of what hallucination refers to in the context of LLMs. For example, when an image classifier wrongly classifies a cat image as a dog, we do not say the image classifier hallucinates, then why or when we should say the LLMs hallucinate when they make a mistake? Comparatively, the uncertainty estimation problem is more well-defined, and we provide a mathematical formulation for the uncertainty estimation task for LLMs. Also, we believe our results on uncertainty estimation can also help with a better understanding of the hallucination phenomenon and tasks such as hallucination detection. 

One limitation of our proposed supervised method is that it critically relies on the labeled data. For the scope of our paper, we restrict the discussion to the NLP tasks and datasets. One future direction is to utilize the human-annotated data for LLMs' responses to train a supervised uncertainty estimation model for open-question prompts. We believe the findings that the supervised method gives a better performance and the hidden activations contain the uncertainty information will persist. 




informs2014







§ DETAILS FOR THE NUMERICAL EXPERIMENTS




 §.§ Details for used features



In Section <ref>, we briefly discussed how to construct features v_i for the uncertainty prediction based on the original prompt response pair (x_i,y_i). In Table <ref>, we provide a full list of the probability- and entropy-related features used in our uncertainty estimation model where p_θ denotes the LLM as previous.







 §.§ Dataset preparation



Prompt dataset generation. For all the tasks studied in this paper, we adopt the few-shot prompting for the LLM. Specifically, in the prompt, we provide r examples to make the LLM learn the format of the response, as illustrated in the following. For the question-answering task, we construct the prompt without using any question-answering sample repeatedly in the original dataset. For example, Prompt 1 includes the 1st to r-th question-answering samples in the original dataset as the examples and the (r+1)-th sample as the target question-answering pair for the LLM; next, Prompt 2 uses the (r+2)-th to (2r+1)-th samples as the examples and the (2r+2)-th sample as the target question-answering pair. However, as the test datasets of MMLU and WMT used for evaluation are not sufficiently large, we generate the prompt in a convolution-like manner: Prompt 2 includes the 2nd to (r+1)-th question-answering samples as the examples and the (r+2)-th sample as the target question-answering pair.



Dataset split. After generating the prompt-answering dataset, we split this dataset into two parts for training the calibration model and evaluation/test. For the MMLU and WMT datasets, we take the dataset generated from the original validation/test dataset. For the question-answering task, as the answer of TriviaQA in the original test dataset is vacant, we take the first 2000 generated prompt-answering pairs from the training dataset as the test dataset, and the remaining for training. 


Prompting format. Here we give the different prompting templates used for different tasks. We use few-shot prompting and the templates can always be roughly divided into four parts: introduction (empty only for WMT), examples, question, and answer, where examples are just r distinct question-answer pairs in the same form as the question and answer parts. We feed the model with the template string except for the reference answer as inputs. 



.9
COQA


















.9
TriviaQA
















.9
MMLU

















 






.9
WMT





 






 §.§ Details of the training procedure



For the three regimes of our supervised approach presented in Section <ref>, the details of the supervised training procedure are as below: 


Gb-S. For the natural language generation tasks (question-answering and machine-translation), we train a random forest model with the input features listed in Table <ref> (20 features in total). For the multiple-choice task, as the answer has only one token from {A, B, C, D}, we take the output logits of these 4 tokens (denoted as α_A, α_B, α_C, and α_D) after inputting the question prompt x to the LLM. Then, we get the probability of each choice as follows:

    p_θ(y|x)=exp(α_y)/∑_y'∈{A,B,C,D}exp(α_y'), ∀ y∈{A, B, C, D}.

Then we use 5 features as the input to Gb-S: the entropy of this distribution, and the sorted probability values in descending order.

Wb-S. The dimension of a hidden layer from LM is typically high (e.g., 4096 for LLaMA2-7B), which may prevent the calibration model from capturing the effective uncertainty information revealed from the activations, especially with limited training samples. Thus, before training a model, we do the feature selection first. We maintain all the features used in the Gb-S and select another 300 features (neural nodes): (i) We use all the features to train a Lasso model and select 100 neural nodes with the highest absolute coefficient values; (ii) By calculating the mutual information between any neural node and the label (correct or not), we select another 100 features possessing top absolute mutual information; (iii) We select another 100 features with top absolute Pearson correlation coefficient. After the feature selection, we train a random forest model to predict whether the response is correct based on the selected features. 

 In the experiment section of the main text, the features in the Wb-S for natural language generation tasks include (i) all the features used in the Gb-S, (ii) the hidden activations of the last token of the question from the middle layer (LLaMA2-7B: 16th layer; Gemma-7B: 14th layer), and (iii) the hidden activations of the last token of the answer from the middle layer. Therefore, in these natural language generation tasks, the dimension is 8212 for LLaMA2-7B and 6164 for Gemma-7B.
 
 The features in the Wb-S for the multiple-choice task include (i) all the features used in the Gb-S and (ii) the hidden activations of the last token of the answer (letter A, B, C, or D) from the middle layer. The dimension is 4101 for LLaMA2-7B and 3077 for Gemma-7B.

 Notably, there are many choices of the hidden activations employed in the Wb-S. Besides what has been shown in Section <ref>, we provide further discussion in Section <ref>.

Bb-S. The idea of building a supervised calibration model for a black-box LLM is to use the hidden layers and output distributions from another open-source LLM model by feeding it with the question and the provided response. Therefore, the features available for the Wb-S are also available for the open-source LLM, so we just take the corresponding features from the open-source LLM in the Bb-S. Hence, in the natural language generation tasks, the input dimension of the calibration model is 4196 (including hidden activations of the question and answer and 20 entropy and likelihood-related features, 2× 2048+20) for Gemma-2B, 6164 for Gemma-7B, 8212 for LLaMA2-7B, and 10260 for LLaMA2-13B. In the multiple-choice task, the dimension is 2053 for Gemma-2B (including the hidden activations of the answer and 5 entropy- and probability-related features used in the Gb-S), 3077 for Gemma-7B, 4101 for LLaMA2-7B, and 5125
for LLaMA2-13B.



§ ADDITIONAL RESULTS AND VISUALIZATIONS


In Section <ref>, we show the advantage of utilizing the hidden activations of the answer from the middle layer of the LLM to estimate the uncertainty in Wb-S. In this section, we further discuss the impact of employing the hidden activations from the question in the Wb-S. 

The motivation stems from the observation that within the transformer architecture, although the hidden activation of a question's last token (referred to as the question's activation) is forwarded to obtain the hidden activation of the answer's last token (referred to as the answer's activation), implying that the answer's activation incorporates the question's activation information, it has been discovered that concatenating the question's activation with the answer's activation offers additional insights into the answer's uncertainty <cit.>. We would like to further investigate the effectiveness of incorporating the question's activation along with the answer's activation into the supervised setting.


We experiment with three feature combinations in our supervised setting: (i) Question: we use the hidden activation of the last token of the question from the middle layer, incorporated with the entropy- or probability-related features of the question (10 features in total listed in the right column of Table <ref>) if it is a natural language generation task, otherwise incorporated with all the features in Gb-S; (ii) Answer: we use the hidden activation of the last token of the answer from the middle layer incorporated with all the features used in Gb-S; (iii) Question-Answer: we use the last-token hidden activation of both the question and answer from the middle layer and all the features in Gb-S. We compare their performance with Gb-S in Figure <ref> and present the following observations.













Question itself cannot capture enough uncertainty information. From Figure <ref>, we observe that the method Bb-S consistently outperforms Question across all these tasks. This implies that incorporating the features relating to the question only cannot provide enough information about the uncertainty of the answer. This aligns with the inferior performance of the sample-based method <cit.> we tested in the earlier sections. In these methods, the uncertainty score is used to estimate the language model's uncertainty about the question. This result implies that uncertainty cannot be captured in the question by the language model without generating the answer. 

Question's hidden activation cannot help to generate more uncertainty information Again from Figure <ref>, by comparing the performance of Answer and Question-Answer, we find that the inclusion of question's activation has little impact on improving the performance. This shows that the uncertainty from the question might have already been well encoded in the last token activation of the answer. In other words, it justifies our approach to employ Wb-S, which includes the answer's hidden activation and the entropy- and probability-related features.























The middle layer is still better than the last layer. In Section <ref>, Figure <ref> shows that when using the hidden activation of the answer in the Wb-S, the middle layer of the LLM is a better choice than the last layer. The next question is: Does this conclusion still hold for using the concatenated hidden activations of the question and answer? We depict the experiment result in Figure <ref>, which is consistent with the conclusion drawn from Figure <ref>.



Our method better characterizes the uncertainty.
We find that the grey-box and white-box features enhance the ability to characterize the dataset so that the distribution of the generated output's uncertainty score is better correlated with the output's correctness. According to Figure <ref>, we observe that with black-box features, the distributions of the uncertainty score for true and false answers are not very distinguishable, and the true answer's distribution is even similar to a uniform distribution. With grey-box and white-box features, the distributions of the uncertainty scores are more separated between the true and false answers. The results show the supervised learning approach not only achieves better AUROC but also learns to better separate the distribution of the uncertainty scores.

























§ EXAMPLES

In this section, we show some examples of the wrong answers the LLM generated and explore how different methods understand the LLM's uncertainty. The wrong answers are selected from those samples where the LLM makes wrong predictions. 

Since we let the LLM output the greedy answer, which could be wrong, we expect an ideal uncertainty estimation model to output a high confidence score when the LLM generates the correct answer, and give a low confidence score when the LLM outputs the wrong answer. By looking at different wrong answers generated by the LLM, we note that although our approach sometimes gives a high confidence score on a wrong answer generated by the LLM, at other times it shows desirable properties such as giving higher uncertainty scores to better answers, and giving low confidence score when LLM does not know the answer.

Our illustrative examples are generated as follows: For questions where the LLM's greedy response is incorrect, we also extract the correct answer from the dataset and additional answers randomly generated by the LLM with lower probabilities than the greedy answer. Along with these answers, we also compute the answers' corresponding metrics and features so that we can observe how they behave with different outputs. We conduct this experiment in the test dataset of TriviaQA, in which both the question and answer are short. We summarize the ways that our uncertainty estimation model behaves as follows:


  * Confidently support a wrong answer. The LLMs are confident that the wrong greedy answer is true and assign a high confidence score. Moreover, the LLMs give low uncertainty scores to the correct answers, suggesting a lack of knowledge about these questions. We give an example of LLaMA2-7B and Gemma-7B in Figure <ref> and <ref>. Note that in both examples, our method assigns a low uncertainty score to the correct answer and a much higher uncertainty score to the wrong answer. In contrast, the unsupervised grey-box methods assign higher uncertainty scores to the correct answer.

  * Confidently reject a wrong answer. We give examples from LLaMA2-7B and Gemma-7B in Figure <ref> and <ref>. The uncertainty estimation model gives a higher score to the true answer or answers that are better than the wrong answer. This means that for these questions, our model actually knows which answer is better and can assign uncertainty scores accordingly. In contrast, the unsupervised methods tend to assign much higher uncertainty scores to the greedy (wrong) answer. 

  * Unconfident about any answer. Due to the lack of knowledge, the LLM may not know the true answer. We show the examples in Figure <ref> and <ref>. From these examples, we can see that the model assigns almost the same uncertainty scores to these generated answers, including the true answer. In this scenario, the uncertainty estimation model is uncertain about the correctness of any answer. Furthermore, it is interesting to note that the unsupervised methods exhibit similar behavior, assigning almost similar scores to other answers as well, albeit with much higher uncertainty scores. 
This differs from the previous two cases, where the unsupervised method behaved differently from our uncertainty estimation model.




























