Studying Large Language Model Behaviors Under Realistic Knowledge Conflicts
Evgenii Kortukov Alexander Rubinstein Elisa Nguyen Seong Joon Oh




Spectroscopic Confirmation of an Ultra-Massive Galaxy in a Protocluster at z∼ 4.9^*

    A. G. Green
    April 29, 2024
=====================================================================================





Retrieval-augmented generation (RAG) mitigates many problems of fully parametric language models, such as temporal degradation, hallucinations, and lack of grounding.
In RAG, the model's knowledge can be updated from documents provided in context.
This leads to cases of conflict between the model's parametric knowledge and the contextual information, where the model may not always update its knowledge.
Previous work studied knowledge conflicts by creating synthetic documents that contradict the model's correct parametric answers.
We present a framework for studying knowledge conflicts in a realistic setup.
We update incorrect parametric knowledge using real conflicting documents.
This reflects how knowledge conflicts arise in practice.
In this realistic scenario, we find that knowledge updates fail less often than previously reported.
In cases where the models still fail to update their answers, we find a parametric bias:
the incorrect parametric answer appearing in context makes the knowledge update likelier to fail.
These results suggest that the factual parametric knowledge of LLMs can negatively influence their reading abilities and behaviors.
Our code is available at: <https://github.com/kortukov/realistic_knowledge_conflicts/>.






§ INTRODUCTION


Retrieval-augmented generation systems (RAG)
combine a generative language model with a non-parametric datastore.
They often surpass larger, purely parametric models in language modeling performance <cit.>.
They excel in solving knowledge-intensive tasks <cit.> and modeling long-tail knowledge <cit.>, and provide attribution of the generated text to identified sources <cit.>.


A particularly attractive feature of RAG is the ability to update  a system's
world knowledge without the need for costly retraining.
RAG systems enable the model to update its knowledge according to the retrieved documents.
Language models using RAG can rely on the facts they memorized <cit.> during the pre-training stage - their parametric knowledge.
Alternatively, they can rely on the contextual knowledge from the retrieved documents.
If the two sources contradict each other, we speak of a knowledge conflict <cit.>, also referred to as context-memory conflict. 
A knowledge update happens when
the model changes its original parametric answer upon seeing a conflicting context. 


Knowledge conflicts happen in three important RAG applications.
First, pre-training a large language model takes months <cit.>, and in that time factual information may become obsolete.
Second, in the currently prevailing transfer learning paradigm <cit.> most end users don't train models from scratch.
Instead, they rely on a few pre-trained models and adapt them to downstream tasks by fine-tuning, prompting, or retrieval augmentation.
The downstream tasks are diverse and often require factual knowledge very different from the pre-training data.
Third, language models are pre-trained on large-scale text corpora that might contain untrustworthy information (, Section 4.6).
Failure to update the parametric knowledge with correct
domain-specific information poses a significant risk for the end user.
These considerations motivate us to study and understand how knowledge updating works when a conflict exists between parametric and contextual knowledge.


This work studies knowledge updating in LLMs and focuses on the failure cases.
Previous work mainly studied artificial knowledge conflicts with counterfactual documents that contradict the model's correct parametric answers <cit.>.
While this is a controlled setting, it is unrealistic.
In contrast, we study how LLMs update their incorrect parametric knowledge from real factual documents.
Our setting reflects how RAG is applied to update and extend the insufficient parametric knowledge of LLMs.
<ref> shows
how our approach differs from prior work.
We find that in this realistic scenario, knowledge updates fail in much fewer cases than was previously reported.

We further study the remaining failure cases and find what we call a parametric bias - models may not use new contextual knowledge if the wrong parametric answer appears in context.
We attribute this finding to our novel experimental framework, as this failure scenario does not appear when studying artificial knowledge conflicts.
Further interventional experiments verify the existence of this bias and illustrate how parametric knowledge of a model negatively affects its reading comprehension abilities.
We show this phenomenon across six question-answering datasets and four studied models of varying 
sizes and capabilities.











§ RELATED WORK

We study LLM behavior under realistic knowledge conflicts and extend the existing body of work that aims at building reliable LLMs through retrieval-augmented generation.

Retrieval-augmented generation.
RAG combines a retrieval system with a pre-trained generative model. RAG has become a widespread approach for knowledge-intensive NLP tasks <cit.>.
Using a generative language model allows more flexibility than earlier extractive approaches <cit.>, while retrieval from a non-parametric datastore reduces many issues of fully parametric language models.
RAG provides performance gains at smaller model scales <cit.>, enables attributing generated text to identified sources <cit.>, and allows updating the system knowledge when parametric information becomes outdated <cit.>, i.e., when a context-memory conflict exists.
However, knowledge updates are not always successful in RAG. 
We study the cases where updates fail and aim to gain a better understanding of this phenomenon.


Knowledge conflicts.
Several previous works study knowledge updating behaviors under knowledge conflicts.
<cit.> construct artificial knowledge conflicts with counterfactual contexts created by entity substitution.
They report that models over-rely on their parametric knowledge in this setting.
<cit.> study knowledge updates in a simplified setup of asking a question to one counterfactual sentence.
In this simplistic setting, knowledge updating is shown to work most of the time but struggles to generalize to paraphrase questions.
<cit.> employ the substitution framework of <cit.>, but report that GPT-3-family models retain their original answer in a smaller number of cases.
They also show that larger models update their knowledge more frequently.
<cit.> revisit the study of <cit.>, but provide several substitution-based counterfactual documents instead of a single one.
They find that as retriever performance (measured by answer recall) increases, models more readily update their knowledge from retrieved documents.
<cit.> claim that previous research suffered from incoherent substitution-based conflicting contexts and propose an LLM-generation-based method of creating counterfactual contexts that “present an illusion of correctness even when factually incorrect” (, p.5). 
Their experiments show that language models update their knowledge more often when the counterfactual evidence looks realistic.
A recent survey <cit.> emphasizes the gap between research using artificially constructed scenarios and real-world RAG application, calling for studies closer to real-word scenarios.
Our study addresses this gap by studying realistic knowledge conflicts in cases where the model's parametric answers are incorrect and a factual context is presented to update the model's knowledge.
<ref> illustrates the difference between previous work and ours.
















§ EXPERIMENTAL DESIGN


We set out to understand how often do LLMs update their knowledge when presented with real factual documents that contradict their incorrect parametric answers?
As discussed in <ref>, previous work only studied knowledge updating behavior with synthetic documents contradicting correct model knowledge.
We introduce a novel framework for studying knowledge-updating behaviors in LLMs in <ref>.
Then, we formalize the problem setup in <ref>.
Details of our empirical analysis are then described in <ref>.




 §.§ Realistic knowledge conflict setup
 

We propose an experimental framework that mirrors real-world RAG application and allows us to study LLM knowledge-updating behaviors.
We focus on the question-answering (QA) task.
First, we identify a subset of data where parametric knowledge is insufficient.
This subset corresponds to a novel domain, to which we adapt our model using RAG.
Then we run the model on this subset providing the retrieved documents.

We assume perfect retrieval to focus on the LLM behaviors.
Therefore, documents always contain ground-truth answers and conflict with the incorrect parametric knowledge of the model.
To simulate "golden" retrieved passages that always contain the correct answers we rely on open-book (extractive) QA datasets.
We simulate in-context retrieval, a popular way to deploy LLMs with RAG <cit.>.


Our experimental framework is a three-stage categorization of samples (cf. <ref>) that enables us to study knowledge updating under knowledge conflicts.







  
Stage 1: Closed-book answer gathering.
stage1
We first probe the model for its parametric knowledge by running the model on the dataset closed-book (i.e., without context). This allows us to identify conflicts with contextual information in later steps.
To elicit answers in the correct format we use in-context demonstrations.
Then, we save the model’s answers for further filtering.



  
Stage 2: Filtering out no-conflict examples.
stage2
We create a knowledge conflict dataset by filtering closed-book answers.
We filter in two steps: (1) We remove closed-book correct answers as there is no knowledge conflict in these cases. An answer is correct when it is equivalent to the ground truth answer by BEM (BERT Matching, a learned metric of semantic answer equivalence, ). (2) We further remove any answer that does not contradict the context, e.g. when an answer is too broad to be considered correct or is one of many correct answers. Similar to prior work <cit.>, we use a natural language inference (NLI) model for this. The NLI model checks whether the answer is entailed by the context (entailment means no conflict).



  
Stage 3: Open-book QA under knowledge conflict.
stage3
We study the knowledge-updating behavior of language models in the presence of a knowledge conflict.
We run the model on the subset of the dataset created in stage2Stage 2 in an open-book fashion (i.e. providing the context).
The answers are categorized into 3 disjoint sets:

    
  * Correct update - knowledge update succeeded, and the model answered correctly according to the context.
    
  * Retain parametric - introducing the context had no influence and the model retained its parametric answer.
    
  * Incorrect update - introducing the context changed the answer but the model failed to answer correctly.


This experimental setup is inspired by studies of <cit.>, but introduces several important changes.
stage2Stage 2 differs from all previous studies. 
We find examples where the LLM's incorrect parametric knowledge contradicts real factual documents.
To do that we propose a novel two-step filtering strategy.
stage3Stage 3 follows prior work in categorization of open-book QA answers.
The key difference to previous studies is that we run open-book QA on real-world contextual documents, without relying on synthetic data or introducing incorrect information.
Additionally, we investigate how the initial parametric answer influences the context reading ability (<ref> and <ref>) and how in-context demonstrations affect the success of the knowledge update (<ref>).





 §.§ Formal notation
 
Let 𝒟 = {(q_i, c_i, a_i)}_i=1^N be an open-book QA dataset of (question, context, answer) triplets.
If a language model f is queried closed-book (without context) we write f(q) = a_p for the parametric answer;
f(q, c) = a_c means we query the model open-book (with context), where a_c is its contextual answer with c as context.
a_gt is the ground-truth answer.
If two answers a_1 and a_2 are an exact match we write a_1 = a_2.
If two answers are equivalent (semantically equal) we write a_1 ≅ a_2.


A knowledge update happens when the model changes its answer upon seeing the context: f(q) ≠ f(q, c).



A knowledge update is correct when upon seeing the context the model changes its answer to the correct one: f(q) ≠ f(q,c) ≅ a_gt.





In stage3Stage 3 we use 𝐔_𝐜 for the Correct update category of examples.
For this subset a_c ≅ a_gt.
The symbol 𝐑 then stands for the Retain parametric examples for which (a_c ≇a_gt) & (a_c = a_p).
Then the Incorrect update category is referred to as 𝐔_𝐢.
It includes examples for which (a_c ≇a_gt) & (a_c ≠ a_p).
We write the proportion of a given category 𝐀 among all open-book QA answers as ℙ(𝐀). 
Additionally, we label the event of the parametric answer appearing in the context as (a_p ⊆ c). 




 §.§ Experimental details
 

Datasets.
We examine knowledge updates across a wide range of open-book QA datasets that represent diverse question-answering scenarios: Natural Questions (NQ) <cit.>, SQuAD <cit.>, NewsQA <cit.>, TriviaQA <cit.>, SearchQA <cit.>, and HotpotQA <cit.>.
We use the unified format versions distributed in the 2019 MrQA Shared Task <cit.>.


Models.
We study the 7B and 70B models of the Llama2-family <cit.> due to their wide adoption. We study the Mistral-7B <cit.>, a model that outperforms Llama-2 of the same size on many benchmarks. Additionally, we study Mixtral-8x7B, a sparse mixture-of-experts model.
For brevity, we report results on Llama2-7B, Mistral-7B, and Mixtral-8x7B.
In <ref> we report additional studies with Llama-70B. 


Evaluation metric.
evaluation_metric
When evaluating the model on a QA dataset both closed-book (stage1Stage 1) and open-book (stage3Stage 3) we employ the BEM metric <cit.> instead of the commonly used Exact Match (EM) to identify correct and wrong examples.
BEM (BERT Matching) is a model-based metric of answer equivalence that allows for a more precise example-level QA quality evaluation. 
We support this design decision in <ref>.














§ OBSERVATIONS AND ANALYSIS
 


In this section, we apply the experimental setting proposed in <ref> to study how often, when, and how language models fail to update their parametric knowledge.





 §.§ Creating the knowledge conflict dataset


We first create a dataset with realistic knowledge conflicts. Given an open-book QA dataset and a language model, we gather its closed-book answers (stage1Stage 1) and filter out those that do not present a conflict between parametric and contextual information (stage2Stage 2).


<ref> shows the size of the final knowledge conflict dataset on which we run the models open-book. 
Dataset sizes after each stage of the pipeline can be found in <ref>.
We run all further experiments and analyses on this knowledge conflict dataset.










 §.§ Studying knowledge updating behaviors under realistic knowledge conflicts


Using the knowledge conflict dataset, we study the knowledge-updating behavior of LLMs.
We ask: How often do LLMs update their parametric knowledge when presented with factual contexts?

In stage3Stage 3, we query the model to answer the questions open-book (i.e., with context).
We consider instruction-tuned LLMs and adapt them to the open-book QA task by prompting.
We tune the prompts on a held-out subset of data for the highest open-book QA accuracy (details in <ref>).
We categorize open-book answers into ℝ, 𝐔_𝐜 and 𝐔_𝐢 subsets as defined in <ref>.
<ref> reports the answer categorization for the tested models.

Contrary to prior work by <cit.> that found over-reliance on parametric knowledge (up to
20% of examples for NQ, and up to 75% for NewsQA in the 𝐑 subset), we find that when we present the models with factual real-world documents, they retain their parametric answers (ℙ(𝐑)) in a very small number of cases.
The parametric answer is retained in 0.4% - 3.4% of examples for Llama2-7B and in 0.1% - 3.3% for Mistral-7B. 
For the Mixtral-8x7B model, the incorrect answer is retained in 0.1% to 6.2% of examples.



Our results are in line with previous work <cit.>: Context quality matters for knowledge updates. Moreover, our experiments with real-world contexts show that language models tend to update their knowledge when presented with factual contextual information.







 §.§ Investigating the remaining failure cases




We investigate the remaining cases in the 𝐑 subset where the knowledge update did not happen. 
We focus on this subset as these are critical failures of RAG. 
If the incorrect answer is not changed after providing retrieved documents that means RAG did not work.
We aim to understand and potentially mitigate these errors.
In this analysis, we ask two questions: (a) How are examples in the 𝐑 category different from other examples and (b) how can we use that information to better understand the knowledge update failures? 




  §.§.§ Studying the differences between example categories


The initial manual investigation uncovered a potential explanation: in a large portion of examples in the 𝐑 subset, the incorrect parametric answer a_p appears in the retrieved document c. For example: 


[boxrule=0pt]
Question: Who was the main performer at this year's halftime show? 
 
Document: CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime show was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Super Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever. 
 
Ground-truth answer: Coldplay 

Incorrect parametric answer: Beyoncé



[16]r0.45 
    
    < g r a p h i c s >

    font=small
    Proportion of examples with incorrect parametric answer in context (a_p ⊆ c) in the full knowledge conflict (KC) dataset, the Retain subset (𝐑) and the Correct update subset (𝐔_𝐜) (as defined in <ref>) for Llama2-7B.
    

To quantify this initial observation, we measure how often the incorrect parametric answer a_p appears in context c in each answer category in stage3Stage 3 of our framework.
We find that in the 𝐑 subset this happens much more often than on average in the dataset and in other subsets.
<ref> shows that for Llama2-7B the incorrect parametric answer appears in context in 32% - 88% of the examples in the 𝐑 subset compared to only 5% - 28% in the full dataset.
The same trend is present across all datasets and studied models (see <ref> in Appendix).


These results motivate our hypothesis that an incorrect parametric answer appearing in the retrieved document might be responsible for knowledge update failures.

We test this hypothesis in the next sections.


-3pt  
[17]r0.45 
    
            Dataset     Llama2
7B     Mistral
7B     Mixtral
8x7B 

            NQ     +20.4*     +7.4*     +5.1*

            SQuAD     +3.9*     +2.7       +1.6 

            NewsQA     –0.8      +0.5     +1.9 

            TriviaQA     +18.3*     +15.3*     +17.0*

            SearchQA     +11.4*     +8.3*     +14.4*

            HotpotQA     +10.0*     +2.9     +4.8*

            
    font=small
    Difference in the likelihood (in %) of knowledge update failure for (a_p ⊆ c) versus (a_p ⊈c). A positive difference means the parametric answer in context increases failure likelihood. Asterisk * denotes p<0.05. 
    In most settings, there is a statistically significant increase in failure likelihood when (a_p ⊆ c).
    

3pt




  §.§.§ Influence of parametric answer in context on knowledge update failures



We hypothesize that there exists a type of confirmation bias induced by the parametric knowledge that potentially explains the majority of the knowledge update failures.
We ask the question: does the appearance of the parametric answer in the context increase the likelihood of a knowledge update failure?

More formally, the set 𝐑∪𝐔_𝐢 contains all wrong answers in the open-book task.
In our setting, this corresponds to a failed knowledge update.
We ask: is there a statistically significant difference between ℙ(𝐑∪𝐔_𝐢 |  a_p ⊆ c) and ℙ(𝐑∪𝐔_𝐢 |  a_p ⊈c)?


We report the results of this analysis in <ref>.
For Llama2-7B there is a statistically significant increase in failure likelihood ranging from 4%pt. up to 20%pt. across all datasets but one.
For the Mistral model, the difference also exists. It is statistically significant at α=0.05 for 3 out of 6 datasets, ranging from 7 %pt. to 15 %pt. For Mixtral-8x7B the difference is significant for 4 out of 6 datasets and ranges from 5 %pt. to 17 %pt.
We provide the details of our statistical analysis in <ref>.

We call this effect parametric bias. 
We define it as a bias when the incorrect parametric answer of the model appearing in context negatively influences its reading ability.

These findings suggest that the parametric answer of a language model makes knowledge update more likely to fail
when it appears in the context document.


   




 §.§ Intervention experiments


To develop a better understanding of the parametric bias phenomenon, we design two intervention experiments.
First, to test if the appearance of a_p in context c was the reason behind the model retaining its parametric answer, we mask a_c when it appears in context and observe the changes in model behavior.
Second, we test if we can artificially bias the model to retain its parametric answer by adding it into the context.
If true, this would provide additional evidence for the existence of a parametric bias.





  §.§.§ Masking reduces the likelihood of retaining parametric answer

We re-run stage3Stage 3 of our experiment and mask the incorrect parametric answer tokens a_p.
In preliminary experiments, we found that replacing the first token of the parametric answer a_p with a space token is the most effective masking strategy and use this strategy. 

We find that masking the parametric answer a_p reduces the probability of the model retaining its parametric answer, as reported in <ref>.
For Llama2-7B the likelihood of retaining the original answer drops by up to 1.6 percentage points, for Mistral-7B and Mixtral-8x7B, the maximum drop is 0.4 %pt. and 1.8 %pt. respectively.

However, a small number of cases stay in the 𝐑etain subset, indicating that the found parametric bias might not be the only reason for the model to retain its parametric answer when presented with a conflicting context.

Additionally, we find that the accuracy ℙ(𝐔_𝐜) also drops by up to 0.6, 0.9. and 1.8 percentage points for Llama2-7B, Mistral-7B and Mixtral-8x7B respectively.
This indicates that the examples that were influenced by the incorrect parametric answer now moved into the Incorrect update category 𝐔_𝐢.
This suggests that the model might be resorting to its original answer in cases when it cannot find the correct answer in the context.

    








  §.§.§ Adding the parametric answer to the context increases the likelihood of retaining it



We further test our parametric bias hypothesis by artificially adding the parametric answer to the context. If the bias exists, we expect the model to retain its parametric answer more often.
We add the incorrect parametric answer to the prompt after the context.
We separate it from the context and the question by the words "Unrelated text", indicating to the model that it should be ignored.
Below is an example of such a prompt:




[colback=white, fontupper=]
prompt = f"""Answer the question with as few words as possible by
extracting information directly from the context.



Context: In the United Kingdom, BBC Radio 5 Live and 5 Live Sports Extra will carry the contest. The BBC will carry its own British English broadcast, with Greg Brady, Darren Fletcher and Rocky Boiman on commentary.

Unrelated text: talkSPORT

Question: Aside from BBC Radio 5, what radio station will broadcast the game?

Answer:"""





Such an addition would not fool a human into reading a context incorrectly because of the words "Unrelated text".
Hence, we claim this is evidence of an existing bias and consider it a failure of the model if this addition influences the model's performance.

We report the results in <ref>. This simple intervention increases the likelihood of the model retaining its parametric answer for all datasets, providing further evidence for the existence of the parametric bias.









In general, our intervention experiments show that the parametric factual knowledge of a language model interferes with knowledge updating, biasing its ability to read the context and jeopardizing the reliability of RAG systems.

    
    











§ DISCUSSION
 


Over-reliance on parametric knowledge.
Previous work <cit.>  found that LLMs retain their parametric answers after seeing a counterfactual conflicting context in a significant number of cases.
Our findings show that in realistic knowledge conflicts, this rarely happens.
These results suggest that LLMs ignoring conflicting information from retrieved documents is not a significant practical problem.
However, a problematic interaction between parametric and contextual knowledge exists in a realistic scenario.


Parametric bias.
We show that despite high knowledge update success rates, the open-book performance of LLMs is not independent of their parametric factual knowledge.
An incorrect parametric answer can make the knowledge update more likely to fail if it appears in the retrieved document.
In <ref> we provide evidence that parametric bias might become an even larger problem in RAG systems with changing knowledge and complex and realistic documents.


Limitations.
Our work focuses on understanding and analyzing the parametric bias.
We have uncovered a problem and introduced a way of evaluating its effect. We leave the solution to future work.
As a first step in this direction, in <ref> we show that better task adaptation through ICL is not enough to mitigate this bias.
Furthermore, in <ref> we show that further scaling might reduce the bias.
However, even the largest currently popular open LLMs are still susceptible to it.




§ CONCLUSION
 

In this work, we studied the knowledge-updating behaviors of LLMs.
This is the first work to study context-memory knowledge conflicts as they appear in practice, with factual real-world documents.
Contrary to prior work that introduced artificial conflicts and suggested that LLMs tend to over-rely on their parametric knowledge, we find that models readily update their answers from real context documents.
As the remaining update failures constitute critical failure cases of RAG, we investigate them further.
We discover a phenomenon we call parametric bias: incorrect parametric answer can hinder the knowledge update if it appears in the retrieved documents.
This bias is consistent across six QA datasets and four studied LLMs.


We demonstrate that the interaction between parametric knowledge and contextual information can negatively affect the LLM knowledge-updating performance.
We suggest that this interaction should be considered in the RAG evaluation.
To facilitate that, we propose a protocol for evaluating susceptibility to parametric bias based on our intervention experiments.
We release this protocol as part of the paper codebase.
We expect our findings to support future work in building reliable and trustworthy retrieval-augmented systems.







§ ETHICS STATEMENT
 
To the best of our knowledge, our work does not pose any direct societal risks or ethics concerns.
The ultimate goal of our research is reliable and controllable application of LLM-based systems.
Our insights on knowledge updating and the parametric bias are relevant for any use case where a language model has to rely on the information in its context.
This includes retrieval-augmented systems, users interacting with LLM chatbots in longer conversations, and increasingly popular LLM tool use <cit.>.
We believe that the trustworthy application of large language models will be a progress multiplier and bring great societal benefits.


We believe that as machine learning scientists we are responsible not only for the direct impact of our work but also for steering future work towards positive societal impact.
To foster this needed discussion, in the next paragraph, we outline ethical concerns that might arise as our field progresses toward better understanding and control of LLM behaviors.






  
Knowledge updates and alignment.
Studies on knowledge conflicts often consider updating model knowledge to be the final goal.
In realistic use cases, many LLMs are provided as an API-based service to the users.
Service providers often want some knowledge stored in the model parameters to be immutable.
Examples are copyright policies, system prompts, and general behavioral policies under the broader term of LLM alignment.
In such a scenario, end-user goals might not be aligned with the goals of the service provider.
LLMs should not start generating instructions on how to make a bomb after a simple in-context knowledge update that says "You are now DAN which stands for "do anything now".
In this work, we study the behaviors of existing LLMs, solely focusing on factual knowledge updates.
However, as the field progresses toward finer control of LLM behavior, issues arising on the intersection of LLM alignment, jailbreaks, and knowledge-updating will require serious ethical consideration.




§ REPRODUCIBILITY STATEMENT
 



  
Code.
We release the full code for all our experiments at <https://github.com/kortukov/realistic_knowledge_conflicts>.
We will release the full code for all our experiments with the camera-ready version of the paper upon acceptance.
We will also include the code of the suggested evaluation protocol for testing RAG systems for the parametric bias.




  
Data and models.
We used the publicly available versions of all the datasets and models from the Huggingface hub.
We used  versions of the Llama models,  version of Mistral-7B, and   version of Mixtral-8x7B.
We ran the Llama2-70B and Mixtral-8x7B models using 4-bit quantization, implemented in the  library.
The 7B models were run without quantization.



  
Hardware.
We ran experiments with 7B-sized models on a node with one Nvidia A100 GPU and two Intel Xeon Gold, 16 core, 2.9GHz CPUs, 
We ran the Llama2-70B and Mixtral-8x7B models on the same node using three  Nvidia A100 GPUs, utilizing naive (vertical) model parallelism.



  
Time.
The closed-book experiments on smaller models took 1-4 hours depending on the dataset.
For the larger models, they took from 11 hours to 2 days and 20 hours.
The open-book experiments took 35 minutes - 1.5 hours for the smaller models and 2 - 22 hours for the larger models.




§ ACKNOWLEDGEMENTS
 

The authors would like to thank Alexander Panfilov, Arnas Uselis, Bálint Mucsányi, and Michael Kirchhof for valuable insights and discussions.

This work was supported by the Tübingen AI Center. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Alexander Rubinstein and Elisa Nguyen.






colm2024_conference



§ DATASET SIZES
 
For better understanding of the scale of our experiments, we report the size of the datasets at each stage of our experimental framework.
<ref> contains the results for each of the studied models. 

As expected, the larger Llama2-70B and Mixtral-8x7B models have stronger closed-book performance than the smaller studied models.
As a result, the knowledge conflict subset for these larger models are smaller.
It is still large enough to draw meaningful statistical conclusions in our case.
However, this trend illustrates one potential future limitation of our approach.

In this work, we rely on general knowledge QA datasets to study knowledge conflicts.
As models grow bigger and memorize more and more facts from their training corpus, studying knowledge conflicts with our proposed realistic approach will require finding datasets that the models did not memorize.
Examining the behavior of more powerful models under realistic knowledge conflicts will benefit from more long-tail and specialized question-answering datasets, on which these models do not perform well in a closed-book fashion.






§ PROMPT TUNING
 
In stage3Stage 3 of our experimental pipeline we evaluate the language model on an open-book QA task.
When we execute experiments with prompting we rely on the zero-shot transfer capabilities of instruction-tuned language models.
It is a well-known fact that such models are sensitive to the input prompts.
To ensure the best possible performance on the task we tune the prompts on 3,000 examples from the training split of each dataset. For each dataset, we choose the prompt with the highest open-book BEM score (percentage of correct answers according to BEM answer equivalence).
Preliminary experiments have shown that our models are not sensitive to the format of the prompt.
However, the performance varies significantly with different wordings of the instruction.
We search over a space of 30 prompts that all have the same format:

prompt = f"""
Answer the question instruction.

Context: context
Question: question
Answer:
"""


The instruction consists of two parts that encode our knowledge of the answer format in the studied datasets.


instruction = f"brevity_instruction context_reference""


All possible options for each of the parts is listed in <ref>.
The resulting optimal instruction wordings for each dataset are reported in <ref>.







§ PARAMETRIC BIAS HYPOTHESIS TESTING
 

In our conflicted open-book QA experiments, all examples naturally split into two disjoint groups: in one the incorrect parametric answer a_p appears in context c (see <ref> for an example), and in the other group it does not. 
We report the sizes of the groups in <ref>.
We model each of the groups of examples as a sequence of independent Bernoulli trials with an unknown success probability π.
In our model success in the Bernoulli trial corresponds to a successful knowledge update, so π is the theoretical quantity that the empirical probability ℙ(𝐔_𝐜) estimates.

We ask the question: is there a statistically significant difference in the successful knowledge update probabilities in these two groups?
We approach this problem from a Bayesian perspective and model the success probability π_𝐔_𝐜 as a random variable.
The number of successes in each group follows a Binomial distribution with m successes out of n trials:

    p(m | n,  π) = m  nπ^m (1 - π)^n - m

We choose the prior distribution on π to be a uniform Beta distribution:

    p(π) = ℬ(π; 1, 1)

The beta distribution is a conjugate prior to the Binomial likelihood and is the common choice when modeling success probabilities.
The specific choice of parameters makes the prior uniform, encoding our lack of knowledge a priori.
The posterior over π after seeing m successes in n trials is then computable in closed-form and is also a Beta distribution:

    p(π| m, n) = ℬ(π; m + 1, n - m + 1)


Let π^0 correspond to the success probability in (a_p ⊈c) group and π^1 is the success probability in  (a_p⊆ c) group.
Our null hypothesis is then: ℋ_0: π^0 = π^1.
This null hypothesis defines a predictive distribution p(m_1| ℋ_0) over the observed number of successes in the group where a_p⊆ c.

To test the null hypothesis we compute the probability of observing m_1 or less successes out of n_1 trials under the assumption that the null hypothesis is correct p(m_1| ℋ_0). 
To compute the predictive distribution on m_1 we need to marginalize over the posterior on π:

    p(m_1| ℋ_0) = ∫ p(m_1| n_1, π) p(π| m_0, n_0) dπ


This results in a Beta-binomial distribution:

    p(m_1| ℋ_0) = p(m_1| n_1, m_0, n_0) =  
    n_1  m_1ℬ(m_1 + m_0 + 1, (n_1 - m_1) + (n_0 - m_0) + 1)/ℬ(m_0 + 1, (n_0 - m_0) +1)


We use this distribution to compute the final p-values by taking the value of the probability mass function at the observed value of m_1.
Final results are reported in the main text in <ref>.














§ ON THE IMPORTANCE OF CORRECT EVALUATION
 
In our three-stage experimental pipeline (<ref>), we prepare the data by filtering out examples that were correct closed-book or where the model answer did not conflict with the context. 
Unlike previous work, we focus on the closed-book incorrect examples.
Thus, we need to identify incorrect model answers with high precision.



The commonly used Exact Match (EM) metric is prone to false negatives, as many conceptually equivalent answers may have different surface forms (and hence are considered different by EM).
As pointed out by <cit.>, EM only provides a lower bound on the system-level performance.
However, EM is a weak example-level metric due to a large number of false negatives. 
Very often the model answers correctly but the surface form of the answer differs from the label (Napoleon’s vs. Napoleon, three vs. 3 or 50–140 cm vs. 0.5–1.4 m). 
In our pipeline, using Exact Match would lead to a lot of actually correct (false negative) model answers making it into stage3Stage 3 of the pipeline.
This would inflate the number of cases where the model retains its parametric answer, as there would be no reason for the model to update its answer since there is no conflict.
For the same reason, we need to filter out incorrect but not conflicting examples in stage3Stage 2 of the pipeline using a separate NLI model that predicts whether the answer is entailed by the context.

We perform an ablation study to illustrate the importance of BEM and NLI filtering by running the experiment on Llama2-7B without filtering and using the EM metric. We report the results in <ref>.




Using a weak evaluation metric may lead one to wrongly conclude that the model often retains an incorrect parametric answer.
This experiment highlights the importance of the strong answer equivalence metric (BEM, <cit.>) and the filtering stage in our experimental pipeline.
This design decision is crucial in identifying samples with a realistic knowledge conflict.






§ TASK ADAPTATION USING IN-CONTEXT LEARNING
 
In-context learning (ICL) is known to improve language model performance on a wide array of NLP tasks.
ICL involves providing several labeled examples as part of the context.
It is considered to be a method of adapting a general pre-trained model to perform a specific task.
It is simpler and requires fewer resources than fine-tuning, but is generally more effective than instruction prompting alone.

In this experiment, we test whether in-context demonstrations can minimize the influence of the discovered parametric bias.

The Llama2 model has a context length of 2048 tokens which does not allow for many in-context demonstrations.
We therefore use five in-context examples for Natural Questions, SQuAD, and HotpotQA datasets, as they have shorter context lengths.
For the other three datasets, we use two in-context demonstrations.
The results for the Llama2 family are presented in <ref> and <ref>.
The 7B model continues to retain its parametric answer in roughly the same number of cases. 
Moreover, the incorrect parametric answer in the context still decreases the probability of a successful knowledge update, albeit to a smaller amount than without in-context learning.
For the larger 70B model, the trend is similar overall.
Interestingly, for TriviaQA and SearchQA datasets Llama2-70B retains its parametric answer even more often with in-context demonstrations than without them.
Furthermore, for these datasets, the negative influence of the parametric answer on update probability is only exacerbated by in-context learning.
We conclude that for the Llama2 model family, in-context demonstrations do not minimize the influence of the parametric bias.

The Mistral-7B model employs sliding window attention, which enables flexible context lengths.
Mixtral-8x7B has context length of 32768.
This allows us to run the ICL experiment with eight context demonstrations for each dataset on both of those models.
The results for Mistral-7B and Mixtral-8x7B are reported in <ref> and <ref>.
The influence of in-context demonstrations is two-fold: it improves open-book performance (ℙ(𝐔_𝐜)), but also makes the model retain the incorrect parametric answer more often than without ICL.
Furthermore, the parametric bias (difference in failure likelihood) still exists after introducing ICL and has roughly the same influence.

In conclusion, for all studied models in-context learning task adaptation does not mitigate the discovered parametric bias.







§ INFLUENCE OF MODEL SIZE ON THE PARAMETRIC BIAS



In this section, we ask the question: How does the size of a language model influence its knowledge-updating behavior and susceptibility to parametric bias.

To that end, we employ two models of the same architecture that differ only in parameter count: Llama2-7B and Llama2-70B. 

First, we report our observations on knowledge updating behaviors and the influence of an incorrect parametric answer naturally appearing in the context in <ref>.
We find that 7B and 70B models have very similar behaviors with respect to knowledge updating: Llama2-7B retains its original incorrect answer in only 1.6% of examples on average and Llama2-70B does that on average for 2% of examples.
We also find that the two models are similarly susceptible to failing their knowledge updates when their incorrect parametric answer appears in context.






Additionally, we run an intervention study, to compare how models of different sizes respond to artificially adding their incorrect parametric answers to context.
We follow the setup in <ref>.

The results are reported in <ref>.
Both models are susceptible to parametric bias: adding the incorrect parametric answer to the context increases their likelihood of retaining it even after seeing the context.
For Llama2-7B the increase is on average by 7.9 %pt.
For the larger 70B model, this increase is on average by 4.8 %pt.

These experiments provide evidence that increasing the model size can be a viable, although costly, solution to mitigate the discovered parametric bias.
However, among the currently popular open models, even the largest ones are susceptible to this bias.










§ PARAMETRIC ANSWER IS LIKELY TO APPEAR IN REAL-WORLD DOCUMENTS



Retrieval-augmented generation is designed to address scenarios where the factual information is changing.
Supplying up to date documents (e.g. web pages) as context for LLM inference then allows to update the system's knowledge and provide correct information to the end user.
We argue, that in these scenarios the parametric answer of the model is likely to appear in the retrieved document.


First, the obsolete parametric answer is not random and still very related to the question and the document that is supposed to answer it.
Even if the answer is currently incorrect, it is likely to be a previously correct answer, or the model's "best guess" at answering the question.
This is especially true for larger models that store a lot of factual information in their parameters after the pre-training stage.
Second, large and unstructured documents, especially web pages, may contain background information and historical data together with the current information.
For example, the Wikipedia article for the https://en.wikipedia.org/wiki/President_of_the_United_StatesPresident of the United States contains the text "Donald Trump" seven times, as of this writing.

To ground this argument in a quantitative estimate, we employ the FreshQA benchmark <cit.>.
It is a weekly updated QA dataset containing factual questions whose answers change with time.
FreshQA contains questions together with answers and a URL for a source web page containing the answer.
We ask the question: How often does a parametric answer of a language model appear in the source web page?

We only consider the subset of FreshQA where the answers change over time ( and  examples).
When searching for the parametric answer to appear on the web page we use the https://corpus.tools/wiki/JustextjusText python library to extract text.

We report the results in <ref>.
As we can see, the parametric answer of the studied LLMs is 23-33% likely to appear on the source web page.
Moreover, a significant portion of those answers are wrong, indicating that the parametric knowledge of the model is obsolete in those cases.

Real-world large retrieved documents can often contain the parametric answer of a language model.
As we show in this paper, this appearance can influence the knowledge-updating behavior of the model.
These results highlight the importance and relevance of the found parametric bias phenomenon for real-world retrieval-augmented systems.






