Does Instruction Tuning Make LLMs More Consistent?
Constanza Fierro Jiaang Li Anders Søgaard

Local well-posedness for a novel nonlocal model for cell-cell adhesion via receptor  binding
    
Mabel Lizzy RajendranSchool of Mathematics, 
Watson Building, University of Birmingham, Edgbaston, Birmingham
B15 2TT, UK, mailto:m.l.rajendran@bham.ac.ukm.l.rajendran@bham.ac.uk  and Anna ZhigunSchool of Mathematics and Physics, Queen's University Belfast, University Road, Belfast BT7 1NN, Northern Ireland, UK, mailto:A.Zhigun@qub.ac.ukA.Zhigun@qub.ac.uk 

    
============================================================================================================================================================================================================================================================================================================================================================================



The purpose of instruction tuning is enabling zero-shot performance, but instruction tuning has also been shown to improve chain-of-thought reasoning and value alignment <cit.>. Here we consider the impact on consistency, i.e., the sensitivity of language models to small perturbations in the input. We compare 10 instruction-tuned  models to the original -7b model and show that almost across-the-board they become more consistent, both in terms of their representations and their predictions in zero-shot and downstream tasks. We explain these improvements through mechanistic analyses of factual recall.  




§ INTRODUCTION


Large language models (LLMs) are trained with general objectives such as 
next token prediction, but 
further training is often needed to make models useful assistants. 

Recent research has shown how so-called instruction tuning—supervised fine-tuning on user interactions—enables good zero-shot performance across a range of tasks 
<cit.> 
by aligning LLMs to user's intents and expectations 
<cit.>. Subsequently, many public instruction datasets have been curated <cit.>, and new instruction fine tuned (IFT) models have been published. Models have been evaluated in terms of their generation quality, by humans <cit.> or by leveraging LLMs <cit.>; while <cit.> evaluated IFT models capabilities by measuring their accuracy on zero-shot and few-shot question answering, finding that IFT models become better or worse at different tasks depending on the dataset used. 

This raises the question: How exactly does instruction tuning change LLMs? <cit.> 
studies this question in terms of decoding and token distribution shift, finding 
most differences to be in stylistic tokens, thus arguing that IFT primarily teaches the model to talk like an assistant. However, instruction datasets have been created as verbalizations of tasks, and thus the data will contain multiple surface forms encoding the same expected behavior (see Figure <ref>). Therefore, we posit that this type of data could also encourage more semantic consistency in IFT models, i.e. less sensitivity to small perturbations, and thus make models more robust. 





Sensitivity of LLMs to subtle changes—that preserve the input's meaning—has been studied in terms of formatting <cit.>, in the context of factual predictions

<cit.>, and in downstream classification tasks <cit.>. Subsequently, methods for eliciting more consistent behavior in LLMs have been proposed <cit.> and further evaluations have followed; in commercial models <cit.>, different languages <cit.>, crosslingually <cit.>, and in retrieval-augmented LLMs <cit.>. However, to the best of our knowledge, no one has studied the impact of IFT on semantic consistency. 







In this paper, we compare the consistency of LLMs and their IFT counterparts, by employing multiple evaluation methods and metrics to account for different interpretations and use cases of semantic consistency. First, we compare their representations and find that after IFT the vector spaces become more consistent, with semantically similar texts encoded closer to each other, and semantically different texts farther apart. Secondly, we evaluate the consistency of models' predictions in factual and world knowledge probes, and after fine-tuning in downstream tasks, finding that IFT models are generally more consistent in all these benchmarks. Finally, we study the factual consistency improvements through mechanistic analyses of factual recall; finding that the IFT improvements in consistency are due to a better retrieval of subject related attributes.







§ CONSISTENCY IN VECTOR SPACE



We study whether the semantics of the representations become more coherent after IFT, so we analyze how the vector space changes by comparing cosine distances between paraphrases and non-paraphrases.



  
Datasets We consider 3 different datasets: (1)  <cit.> contains pairs of sentences that are semantically equivalent, and pairs of sentences that are similar but do not convey the same meaning; (2)  <cit.> contains sets of paraphrases or near-paraphrases extracted from the Tatoeba database; and (3)  <cit.> which is a manually created resource of paraphrases in the context of factual knowledge probing. For each dataset, we compute the cosine distance between the paraphrases and then take the macro average across the sets of paraphrases (Equation in <ref>). Conversely, we compute the cosine distance between non-paraphrases by taking random examples (or, for MRPC, the available non-paraphrases). Then, to compare across models we look at the difference between these two averages, that is, at the gap between how close paraphrases are and how close non-paraphrases are; because different models may have all their representations generally closer together or farther apart. 



  
Models
We compare the  model <cit.> and 10 different IFT models based in . These models were fine-tuned using 10 different instruction datasets (Details in <ref>). We also study T5-XL and FlanT5-XL, and Falcon-7b alongside Falcon-Instruct-7b.
 

We represent paraphrase candidates by the outermost representation of their last token 


<cit.>. Given the influence of instruction or context, we compute 
average cosine distances across different prompts (<ref>).  




In Table <ref>, we compare the gap between the average cosine distance of paraphrases and the average cosine distance of non-paraphrases. We find that IFT models exhibit bigger relative differences 
in all the datasets evaluated, both  models as well as T5 and Falcon IFT counterparts (<ref>). 



§ CONSISTENT PREDICTIONS



Does representational consistency affect the consistency of model predictions? We measure: (1) factual consistency using the  dataset[ contains paraphrases for 38 binary WikiData relations, e.g. “X was born in Y” and “X is originally from Y”, where X would be replaced by a subject, and Y is the object to be predicted.] 
<cit.>, where consistency is the fraction of paraphrases where the top-1 predictions are the same; (2) the accuracy spread <cit.>, i.e., the min-max difference in accuracy across different paraphrases, which we measure in the  dataset[  evaluates world knowledge and reasoning skills through multiple choice questions about mathematics, US history, law, and more.] <cit.> by calculating accuracy over three different instructions in a zero-shot fashion[To account for prompting variability, we consider 3 different prompts for the  dataset and 3 different shuffling of the multiple choices in the  dataset, reporting average results; see prompts details in <ref>.]; and (3) the semantic consistency of downstream classification tasks, that is, the consistency of NLI, paraphrase detection, and contextual word similarity predictions under paraphrasing and negations, following the  benchmark <cit.> (Details in <ref>).

We find that all IFT models have lower spread values compared to  (Table <ref>), and that on average  is the least consistent on downstream classification tasks (). 
The  results are mixed, with most IFT models exhibiting higher factual consistency, but a few (oasst, cot, and flan-v2) similar to the  base model.[The standard deviation of the consistency in  over the different contexts is 1.4 for  and between 0.1-0.8 for the IFT models, thus we consider the difference significance if is higher than 1.4.] Thus, we conclude that overall IFT does make models more robust to semantic equivalent inputs. Moreover, we see that these results hold for the T5-XL / FlanT5-XL models, and partially hold for Falcon / Falcon-Instruct (Details in <ref>). 






§ HOW IFT IMPROVES CONSISTENCY



In factual knowledge recall, an object is predicted given a subject and relation.   
<cit.> describe the process as three-way: 
(a) the subject representation is enriched, i.e., related attributes are encoded; (b) the relation information is passed to the last token; and (c) the final predicted attribute is extracted by attention layers. 
Below, we analyze the consistency improvements above in terms of these three mechanisms, 
focusing on the  dataset. We report averages over three different contexts in all of the following analyses. 













  
Subjects 
<cit.> found that in factual knowledge recall, the subject representation is enriched to encode many subject-related attributes. We therefore investigate whether the quality of this enrichment is improved after IFT. We measure recall over subject attributes by inspecting the hidden state h^l of the last subject token at layer l projected onto the vocabulary space, i.e., E h^l with E the output embedding matrix, and we consider the top-50 tokens of this projection. 
To compute the fraction that indeed corresponds to subject attributes, we create a set 𝒜_s containing tokens that appear in contexts where the subject s is mentioned. We obtain 𝒜_s by fetching the top 500 snippets in Wikipedia where the subject is mentioned, and then follow the same preprocessing as in <cit.> (see <ref>). Finally, across models, we compare the fraction (rate) of top-50 tokens that are present in 𝒜_s; see Figure <ref>. We find that  has lower attributes rate than all IFT models, thus showing that IFT improves the recall of subject's attributes across-the-board. On the other hand, models that have lower or the same factual consistency as  in the  dataset (such as oasst1 and cot) still have higher attributes rate, therefore improvements in consistency cannot be explained solely by improvements in recall.[We note that the attributes rate peaks at ≈ 30% compared to the 40% <cit.> reported, and that the rate decreases after layer 25. This is probably due to differences in the Wikipedia snippets and the templates 
(This data was not published so we could not use it or verify it).]




  
Relations Next, we measure whether IFT leads to better representations of the relation in a factual knowledge query. To this end, we first find the layers in the model where the relation information  is more salient, and then we compare the representations across models in these specific layers.





 
 We compute the cosine similarity 
 of two different settings: (1) between query pairs, where each query corresponds to a different relation and contains a different subject and target object; 
 
 (2) 
 between pairs of queries with the same relation, but with different subjects, target objects, and paraphrased templates.[We further constrain the templates to end in a different token to avoid confounds.] 
 
 Note how in (2), the only added similarity between the pairs of sentences is the relation. We plot the results in Figure <ref>. We find that both pairs of sentences exhibit similar trends, with similar representations in the first layer, dropping on the 3rd.  For different relations, this drop continues until layer 9, whereas for same relation, the distance between the representations stays mostly constant after layer 3. 
 We conclude that the bigger drop in layers 3-8 shows that these encode more semantic information about the relation.[This is consistent with the gradient analysis in <cit.> showing that relation tokens are more relevant for the prediction in early layers (0-10).] 



Finally, following <ref>, we compare paraphrase representations to non-paraphrase representations to measure the extent to which a model differentiates between semantically similar and semantically different inputs. Specifically, we plot cos(paraph.) -cos(paraph.) (Equation <ref> and <ref>) across layers of each model—the bigger this gap the more the model makes a difference between paraphrases and non-paraphrases. We break down this analysis in consistent and inconsistent paraphrases, so we can compare representations for which the model behaves similarly. Figure <ref> contains the gap between paraphrases and non-paraphrases for which the model gives consistent outputs.[The plot for inconsistent outputs gives similar results and can be found in <ref>, alongside the breakdown of the plot for each of the three contexts.] When looking at the relation layers (3-8) found previously, we see that for some contexts the gap of the  model is bigger than that of IFT models (prompt=0 in Figure <ref> in <ref>); however on average, the gap is similar to the other models. Thus, we do not find evidence that there is a better semantic representation of the relation after IFT.














  
Extraction of Predicted Attribute The last mechanism of factual recall predictions is the extraction of the predicted attribute from the relation and enriched subject representations, which have been propagated to the last token <cit.>. We aim to compare how this extraction mechanism might have changed after IFT. Consider a model with L transformer layers, let h^l be the representation of the last token at layer l, then each transformer layer computes

    h^t+1 = h^t + MLP(h^t + MHSA(h^t)_a_t)^f_t

where f_t / a_t is the MLP / MHSA update to the representation.[MHSA stands for Multi-Head Self-Attention] An extraction event happens when the final predicted object o^* (at the last layer) is the same as the top-1 token of a_t projected to vocabulary space, i.e. o^* =  max(E a_t) (analogous for f_t). We compare the rate of these events for MHSA in Figure <ref>.[MLP has similar results (see <ref>) but with lower extraction rates overall, which is congruent with <cit.>.] We see that the IFT models that obtain lower consistency scores in  have lower extraction rates (yellow in the Figure). So we compute the Pearson correlation between the IFT models' extraction rates and their  consistency scores, and we find significant (p-value <= 0.05) positive correlations of 0.65-0.92 in layers 13-21 (details in <ref>), showing that a higher extraction rate is correlated with a higher factual consistency. On the other hand, we see that  has one of the highest extraction rates in the different layers, so we conclude that 's lower factual consistency results are probably due to the lower subject attributes recall (Figure <ref>), while the lower factual consistency of some of the IFT models is due to a less specific extraction rate of MHSA layers.



§ CONCLUSION

In this paper, we studied the robustness to variations that are semantically irrelevant, i.e. consistency, of LLMs after instruction fine-tuning (IFT). We considered multiple definitions and benchmarks of consistency to account for different interpretations and use cases. Specifically, we studied consistency from the perspective of vector spaces, zero-shot predictions, and downstream predictions. We found almost across-the-board that IFT models are more consistent. Lastly, we find explanations of this improvement by analyzing the mechanisms of factual recall, concluding that IFT models exhibit a higher recall of subject attributes resulting in consistency improvements when the extraction mechanism (in the last layers) is not deteriorated with IFT.



§ LIMITATIONS


Our experiments and analyses rely on input data, and even though we have tried to be thorough by reporting averages over different contexts and instruction formats, the exact numbers could change slightly when using different prompts. Relatedly, in choosing the prompts we tried to make it relevant and the most natural for each model, which in turns means that we have different prompts for  and the IFT counterparts, because these models have been trained in a different way and therefore expect the data to be different ( does not follow instructions and the pretraining data would rarely look like that so its performance would be most probably degraded). Lastly, we compare  and multiple IFT counterparts and extend some of the evaluations to T5 and Falcon, however is hard to generalize consistency improvements to any IFT model as this will depend on the specifics of the data, as we saw for some  IFT models (e.g. oasst1).



acl_natbib





§ INSTRUCTION TUNING DATASETS

We use the models trained and released by <cit.>[We do not evaluate on the Code-Alpaca dataset as it is focus on code generation, and on Self-Instruct since we include Alpaca which is an improved version.], here we describe each of the datasets used in the fine tuning. These datasets have been used to train other instruction tuned models, but we only evaluate with the ones released by <cit.> because in this way we ensure that the training and formatting for all the models were the same.



  
ShareGPT Conversations with ChatGPT shared by users (<https://sharegpt.com/>). Data from this website was used to train the Vicuña model <cit.>, but <cit.> use the the data available in HuggingFace[<https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered>]. The dataset contains around 160k examples.



  
Dolly <cit.> Human written instructions created by  5000 Databricks employees for a variety of tasks such as brainstorming, content generation, information extraction, and summarization. The dataset contains around 15k examples.



  
Open Assistant <cit.> Human written instructions generated through crowd-sourcing involving over 13,500 volunteers, with varied education, nationality, and proficiency with artificial intelligence. The dataset contains examples in 35 different languages and has around 35k examples.



  
Flan V2 <cit.> Template instructions manually written to convert existing supervised datasets into instruction examples. Compared to Flan <cit.> Flan V2 added more templates variability, chain-of-thought examples, multiple languages, and extra examples by inverting the input. The dataset contains 100k examples.



  
CoT <cit.> Version of the Flan V2 dataset containing chain-of-thought in the instructions. The dataset contains 100k examples.



  
SuperNI <cit.> Human written instructions for existing 76 NLP tasks in a variety of languages. The instructions were obtained as a community effort with NLP practitioners and students submitting instructions to GitHub, which are then reviewed by 1-2 expert contributors. The dataset contains around 96k examples. 



  
Unnatural Questions <cit.> Automatically generated instructions by prompting a Davinci-002 with three examples of instructions and generating a fourth one. The dataset contains around 68k examples.



  
Alpaca <cit.> Automatically generated interactions by prompting Davinci-003 with 175 seed tasks, whereby they sample new examples from the model containing new instructions, input, and output <cit.>. The dataset has a total of 52k examples.



  
GPT4-Alpaca <cit.> Same procedure than for the Alpaca dataset but using GPT-4 instead of Davinci-002. The dataset contains 52k examples.



  
Baize <cit.> Automatically generated examples by prompting ChatGPT (gpt-3.5-turbo) to interact with itself. To prompt the model they use questions from the Quora and Stack Overflow website as seeds. The dataset contains 210k examples.





§ COSINE DISTANCE DETAILS



 §.§ Cosine Computation

Let p={s_i}_i=1...M be a set of paraphrases where s_i and s_j have the same semantic meaning, we compute the average cosine similarity of a set p and then take the macro average over these sets, that is,

    cos(p)    = 1/M∑_i,j
    i≠ j^M cos(s_i, s_j) 
    cos(paraph.)    = 1/N∑_i^N cos(p_i)

Conversely, to compute the cosine distance between non-paraphrases we take at random the sets of paraphrases p_i, p_j and the paraphrases s_k, s_l.

    cos(paraph.)    = 1/K∑_p_i ≠ p_j 
     s_k ∈ p_i,   s_l ∈ p_i^K cos(s_k, s_l)

Finally, to compare across models we look at the difference between these two averages, that is, at the gap between how close paraphrases are and how close non-paraphrases are: cos(paraph.) -cos(paraph.); because different models may have all their representations generally closer together or farther apart.



 §.§ Input Details


We define a different set of templates for the different datasets, as  and  examples are full sentences whereas  inputs are shorter phrases that are supposed to be completed in a factual way. For  examples we only use the template with the subject replaced but not the object, as we do not want to measure how closer or farther apart the object is but the general representation of the paraphrased input. For the instruction tuned models we follow the dialogue format in which they were fine tuned. The specific templates are presented in Table <ref>.







 §.§ Results on Other Models

We also compute the gap of cosine distances between paraphrases and non-paraphrases for other models, namely Falcon-7b and Falcon-Instruct-7b and the T5-XL[We use the  version.] and FlanT5-XL models. We present the results in Table <ref>, we find that for all the datasets and both pairs of models the IFT versions have a bigger gap between their average cosine distances of paraphrases and non-paraphrases.





§ CONSISTENCY PREDICTION EVALUATION




 §.§  and  Prompts

For the  dataset we use the same prompts we use for the cosine distance analysis (see Table <ref>). For the  dataset we compute the spread by calculating the accuracy of each model for 3 different instructions and 3 different orderings of the multiple choice questions, so we take the average over 9 different setups for each model. The 3 instructions we experiment with are:

    
  * The following are multiple choice questions (with answers) about {}.

  * The following are multiple choice questions with their respective answer. The questions are about {}. 

  * Answer to the best of your abilities the following multiple choice questions about {}. 

As usual for  evaluation <cit.> we add the subject of the question in the placeholder “{}” (e.g. high school mathematics), after this instruction the question and the multiple choice answers are added.



 §.§ 

The  dataset provides nineteen test sets for assessing five types of consistency on seven downstream tasks. We follow the setup of <cit.>, focusing the scope of our experiments mainly on three consistency measures: Negation, Semantic, Symmetric; and four datasets: SNLI <cit.>, RTE <cit.>, MRPC <cit.>, and WiC <cit.>. SNLI and RTE are NLI datasets, where the task is to predict whether a premise entails a hypothesis; MRPC is a paraphrase detection dataset; and in WiC the task is to predict whether two sentences are using the same meaning of a word.  augments the input data of these datasets making available a negated version of the input and its corresponding label, a paraphrased version of the input, and the swapped input sentences and its corresponding target label.



  
Consistency Computation.
BECEL contains the original evaluation set ℰ and a newly constructed evaluation set ℰ_N; consider e_i ∈ℰ and e^N_i∈ℰ_N, where e^N_i is a perturbed version of e_i (i.e., |ℰ| = |ℰ_N|). A model M is expected to yield predictions for both e_i and e^N_i. <cit.> reports inconsistency while here we report consistency (which is simply 1-τ), with τ computed as follows,


    !τ = 
    1/|ℰ_N|∑_i=1^|ℰ_N|1(M(e_i) = M(e^N_i)),     if Sem. / Sym.
    1/|ℰ_N|∑_i=1^|ℰ_N|1(M(e_i) ≠ M(e^N_i)),    if Neg.


where 1 is the indicator function.



  
Finetune details. To fine-tune all models, we experimented with three learning rates (5e^-4, 1e^-4, 5e^-5), a batch size of 32, linear decay of the learning rate, and early stopping. The models were trained for 10 epochs. For the SNLI dataset, the optimal learning rate is 5e^-5, while for the other three datasets, the best learning rate is 1e^-4.



 §.§ Results

The detailed  results for each task and model are presented in Table <ref>. The results for the extra models (T5 and Falcon) can be found in Table <ref>. We see that the same trends presented for the  variants hold for T5, FlanT5 is more consistent in terms of spread, factual consistency and downstream consistency. Also, we find that T5 and Falcon have a high downstream consistency under negation, which is maintained after IFT. On the other hand, we find that the Falcon model after IFT does not have an improvement in spread in  and it has a decrease in factual consistency in ; this could be due to the prompting strategy not being optimal for this model, given that these 2 task are evaluated in a zero-shot fashion.



§ SUBJECT ATTRIBUTES

We obtain the top 500 snippets from English Wikipedia where the subject is mentioned in the text.[The mention can include extra text in between words of the subject.] The ranking of these snippets is performed using the “classic” profile of the API search, which ranks the pages based on the number of incoming links, some templates, page language and recency. Then, we follow <cit.> and tokenize the text, remove duplicated tokens, stopwords using the NLTK package, and tokens
with less than 3 characters (so to remove tokens representing frequent short sub-words). This results on 1,936 ± 1,036 attributes per subject (for a total of 27,341 subjects).








    




§ RELATION UNDERSTANDING


To study the cosine similarities of paraphrases we break down the visualization in queries for which the models predict the same top-1 candidate (consistent) and the queries for which they have different (inconsistent). So that if the predictions are inconsistent because the model did not understand properly the relation semantics then this does not bias the results. The consistent predictions cosine similarities are presented in Figure <ref> and the inconsistent predictions are presented in Figure <ref>.






§ EXTRACTION RATE

The MLP extraction rate results are presented in Figure <ref>, and the detailed results of the Pearson correlation computation can be found in Table <ref>.



